How this site is generated

This perl script kicks off the all of the work. It is essentially a wrapper around a set of commands which do things like produce galleries of images (see the Photos of landscapes and sunsets for an example of this); process the raw HTML I write and produce the version with added eye-candy; grab log files from the real web servers; and publish the processed web site.

This is a program which designed to maintain the content of a remote web site from a local copy. Only the changes which have occured on the local system are actually transmitted, making it ideal for a dialup line.

See SiteCopy home page and the FreshMeat app entry for more details.

This is a program designed for off-line HTML generation for Unix. It provides such things as pre-processing, meta-HTML, embedded perl, m4 and much more. I abused it a lot to produce these web pages.

See the WML home page for more details.

This perl script does the real work of taking the HTML I write and munging it into the form you see before you.

SourceSites is built around making my life easier to maintain the web site. SourceSites goes through various stages to do this. The firstly is goes through and finds all of the description files I've scattered around the system. These files contain information on every file and is used to hold information like the author of the document, document title, document header, section the document is in and so on. Files can be recognised (or ignored) based upon the full pathname, partial file name or regular expression. It is also posssible to run commands here so that description files and index files can be automatically generated. An example of this is the Essays, Rants and Raves section which has an automatically generated index page. The description file and the index file are generated by a script which looks for tags within each of the files within the area. In this way new files can easily be added into the system - it just has to be copied into place and have the correct tags put into place.

After the finding of the description file, the script scans the source area and processed area building up a list of the entries in both. Once done, the two lists are scanned and at that point it knows what the delete from the processed area and what needs to be processed.

How each entry to be processed is handled depends on what that entry is. Directories are created in the processed area. Files not matched by a given rule within SourceSites are merely copied. HTML files are attacked by wml. However this is wml given a large number of defines and some pre-processing to begin with. Thus this file was processed using the following command line:

% /usr/local/bin/perl -e 'print "#include <head.html>\n";
> while(<>) {
>   print "$_";
> }
> print "#include <foot.html>\n";'
   <Source>/about/generation.html
 | /usr/bin/sed 's/\*\//\&#42;\&#47;/g'
 | /usr/bin/sed 's/\/\*/\&#47;\&#42;/g'
 | /usr/bin/sed 's/\\n/\\@NEWLINE\\@/g'
 | /usr/local/bin/wml
     -v
     --norcfile
     --settime
     --pass="1239"
     --epilog=weblint
     -DROOTFILE=about/generation.html
     -DWEB_SITE=www.bpfh.net
     -DSOURCE_DIR=<Source>
     -DFINAL_PLACE=<Processed>/about/generation.html
     -DREMOTE_DIR=<Processed>
     -I<Source>/about
     -I<Includes>
     -I<Includes>/www.bpfh.net
     -I<Source>
     -DLOCAL_CTIME='932049965'
     -DLOCAL_HEAD='How this site is generated'
     -DLOCAL_TITLE='About BPFH - Site generation'
     -DLOCAL_AUTHOR='simes'
     -DLOCAL_MTIME='932049965'
     -DLOCAL_SECTION='About BPFH'
     -DPROG_DEF_LIST="LOCAL_CTIME:LOCAL_HEAD:LOCAL_TITLE
                      :LOCAL_AUTHOR:LOCAL_MTIME:LOCAL_SECTION"
     -o <Processed>/about/generation.html  ;
 /usr/local/bin/perl -e '
> my $prn=0;
> my $in=$ARGV[0];
> my $out="$ARGV[0].tmp";
> open(FILE,$in) || die "Failed to open $in for reading - $!\n";
> open(OUT,">$out") || die "Failed to open $out for writing - $!\n";
> while(<FILE>) {
>   ((!$prn)&&(!/^$/o))&&($prn=1);
>   ($prn)&&print OUT $_;
> }
> close(FILE);
> close(OUT);
> if (!rename($out,$in)) {
>   unlink($out);
>   die "Failed to rename(\"$out\",\"$in\") - $!\n";
> }
> exit 0;' <Processed>/about/generation.html
** WML:Verbose: Processing time (seconds):
** WML:Verbose: main |  ipp   mhc   epl  gm4  div asub hfix hstr slic |  TOTAL
** WML:Verbose: ---- | ---- ----- ----- ---- ---- ---- ---- ---- ---- | ------
** WML:Verbose: 6.01 | 1.86  1.50  3.25   --   --   --   --   -- 2.12 |  14.74
%

It should be noted that the final output is put through weblint to make sure that the HTML actually conforms to some sort of standard. My normal aim is to produce HTML which has no warns when weblint is run over it.

Yes, it does take a long time to produce the final HTML file from the source I write. The above time (14.74 seconds) is a little longer than normal thanks to the length of this document. However it is is nothing when compared to the time it takes to produce the maps of the site from the hash of files in the source area.

However, we are producing static HTML here which is just seen as a plain file by the web server. Thus the above time does not factor into the time taken to serve the document out in any way shape or form.

This perl script does the work of taking the images I give it and generating the thumbnails and surrounding HTML. I've done quite a bit of work on it over the past couple of years. The result of this is that I can put an image into its correct directory, put a single line into an index file within that directory and then wait. The thumbnail, HTML for the main image and the index page are all automatically generated for me.

I'm quite happy with the thumbnail generation - even though its automatically done for me it does quite well in shrinking the image. Looking over all of the images within photo collection, the thumbnail images range from 14% down to 1.5% of the size of the full-sized image. Taking the largest and smallest percentages, along with the largest thumbnail, we see:

It should be noted that the size of the thumbnail often depends on the complexity of the full size image.

In case you're wondering, the thumbnails are generated using the netpbm tools and (in the case of JPEGs) the JPEG commands which come with the JPEG library.

Where: