Phases – MakeStaticSite

MakeStaticSite’s workflow comprises a number of phases, each phase having one or more tasks, as indicated in the following visualisation, arranged in temporal ordering. The labelling more or less reflects the descriptions given previously.

Setup

The opening lines of a MakeStaticSite configuration file, showing the comments header describing what it is and when created.

Configuration dialogue

The first step is to run the interactive setup.sh script, where the user specifies site options, to generate a configuration file.

0. Initialisation

Configure

The system reads and processes configuration options

read runtime options
read configuration file
prepare output layout

1. Prep the CMS

CMS-specific streamlining

Optionally use WP-CLI to

ensure nice permalinks
remove query strings and remote shortlinks
replace with static search

(Currently WordPress-only, but modules could be developed for other CMS.)

2. Generate static site

Generate wget mirror (first run)

Run wget with the supplied configuration options

optionally incorporate further input files
optionally set up a zip files
store the mirror in its own directory with timestamp

3. Augment

Retrieve additional assets

Run wget with the supplied configuration options to fill in gaps

optionally crawl static files for further URLs
Filter the URLs to retrieve only assets (multimedia files, etc.)
Re-run wget on the additional URLs

4. Refine

Further processing of output

Carry out further processing of output in preparation for deployment

Convert remaining absolute paths to relative paths
Standardise on deployment domain
Standardise internal anchor suffixes

5. Add Extras

Copy from Extras Folder

Allow the inclusion of further files, copied from a designated directory. The files which might include downloads or non-static files.

6. Optimise

Further post-processing for deployment

Further post-wget processing

regenerate canonical URLs in document headers
convert feeds to be XML files and modify references accordingly
Apply HTML Tidy to make more standards-compliant and use pretty-print (dependent on options)

An accessibility validator could be added here.

7. Use snippets

Use and process snippets

Optionally augment the static mirror with snippets, substituting chunks of HTML in particular places. For example, to hide links to CMS logins.

8. Create offline zip

Create a custom zip archive

The mirror acts as the basis for a zip archive, ready for uploading with the site or other distribution channels. Anchors point to .html files, to maintain navigation.

9. Deploy

Finalise and deploy locally or remote

Optionally copy over additional

Ensure anchors conform to canonical URLs
Finally deploy on local or remote server. Remote options include rsync and Netlify (can deploy to both in same run)

10. Conclude

Conclude without exit

The script ends by stopping the clock, but there is no exit statement, which allows further scripts to run afterwards.

Sample Terminal Output

The following is pretty much a complete copy of the terminal output (STDOUT and STDERR) generated when building and deploying makestaticsite.sh (constants.sh has output_level=quiet). More granular information is written to the log file ( log_level=normal). To facilitate the illustration, we split the output into the respective phases.

Phase 0: Initialisation

paul@linux-pt:~/scripts/makestaticsite$ ./makestaticsite.sh -i mssweb
Welcome to MakeStaticSite version 0.22
Starting at phase 0: Initialisation.
Ending at phase 9: Deploy.
Reading custom configuration data from config/mssweb.cfg ... Done.

Phase 1: Prep the CMS

Starting the static site generation ...
Will capture snapshot from https://makestaticsite.sh using wget.
WP-CLI is installed and will be run locally
Found WordPress site at /var/webs/mss.ptlocal
Checking WordPress settings to remove query strings and shortlinks
Updating Perform options ... Done.
It looks like there is already a search page.  Assume that it contains the shortcode [static_search], so we won't update it.

Phase 2: Generate static site

Reading list of additional URLs to crawl from config/input_mss_localweb.txt.
Running Wget with options: --mirror --convert-links --adjust-extension --page-requisites -U Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) -X/wp-json -R wp-admin,xmlrpc*,index.html?*,*login*,*logout* --no-check-certificate --directory-prefix mssweb20230214_122515 --input-file=/home/paul/scripts/makestaticsite/tmp/wget_inputs_main.txt -a /home/paul/scripts/makestaticsite/log/makestaticsite.log
Creating a mirror of https://makestaticsite.sh in /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb ... Done.
WARNING: Wget ERROR code 8, i.e. the Web server gave an error on retrieving at least one file, probably HTTP 404 (file not found - possibly specified in the input file).  Less likely is a 500 (internal server error), which in the case of a CMS might be due to a plugin or module.
It should be safe to proceed, but you may like to rerun and/or review the output by referring to the log file, which has more detail. 
Continuing (to run unattended).

Phase 3: Augment

Searching for additional URLs to retrieve with Wget (working in /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb) ... Done.

Phase 4: Refine

Carrying out post-Wget processing in /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb ... Done.
Replacing remaining occurrences of mss.localweb with makestaticsite.sh ... Done.
Converting feed files and references from index.html to index.xml ... Done.
Updating anchors for makestaticsite.sh from http: to https: ... Done.

Phase 5: Add Extras

Copying additional files from /home/paul/scripts/makestaticsite/extras/mss.localweb/ to the static mirror (for distribution) ... Done.

Phase 6: Optimise

Updating canonical URLs in document headers ... Done.
Running HTML Tidy on html files with options -m -q -indent --indent-spaces 2 --gnu-emacs yes --tidy-mark no ... Done.
Created a sitemap file at /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb/sitemap.xml

Phase 7: Use snippets

n/a – not used

Phase 8: Create offline zip

Creating a ZIP archive ... 
ZIP archive created.

Phase 9: Deploy

Updating internal anchors to conform with canonical URLs ... Done.
Deploying on Netlify ... 
Site already linked to "soft-figolla-77cff1"
Admin url: https://app.netlify.com/sites/soft-figolla-77cff1

To unlink this site, run: netlify unlink
Deploy path: /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb
Deploying to main site URL...
- Hashing files...
✔ Finished hashing 
- CDN diffing files...
✔ CDN requesting 3 files
- Uploading 3 files
✔ Finished uploading 3 assets
- Waiting for deploy to go live...
✔ Deploy is live!

Logs:              https://app.netlify.com/sites/soft-figolla-77cff1/deploys/63eb7dc5dbb9cc20463446eb
Unique Deploy URL: https://63eb7dc5dbb9cc20463446eb--soft-figolla-77cff1.netlify.app
Website URL:       https://makestaticsite.sh
Deploying on a remote server using rsync over ssh ... 
Sync to remote server
From: /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb/
To: ***

Phase 10: Conclude

Completed in 37 seconds.
All done!
A static mirror of https://makestaticsite.sh has been created in /home/paul/scripts/makestaticsite/mirror/mssweb20230214_122515/mss.localweb
The site has been deployed on ptworld.net, for web access at http[s]://makestaticsite.sh.

Thank you for using MakeStaticSite, free software released under the GNU Affero General Public License version 3.  The latest version is available from https://makestaticsite.sh/download/makestaticsite_latest.tar.gz.

The deployment message needs some explanation. It has actually been deployed to two services, first Netlify, where its hosted under makestaticsite.sh, and then another server, ptworld.net, which can act as a mirror or, in the rare case of hosting or DNS issues, a fallback (with the relevant changes to domain name settings).