Penny Black Philately

by John Vincent

Posted on October 1, 2023

Stamps

Gatsby Stamps is a Stamp Catalog application.

Use Gatsby Stamps to view the stamps of selected Nations.

Live Deployment

Gatsby Stamps at Digital Ocean

Technical

This implementation is a port from philately.johnvincent.io, which was built using React

The intention is to build the application using Gatsby and React.

Gatsby statically generated pages are ideal for performance and SEO.

stamps.johnvincent.io is built using Gatsby, React, HTML5, Sass and CSS3
Gatsby is a React-based, GraphQL powered, static site generator. By weaving together the best parts of React, webpack, react-router, and GraphQL, Gatsby builds blazingly fast websites.
stamps.johnvincent.io is a Progressive Web App (PWA)
stamps.johnvincent.io is fully responsive, adapting for mobile, table and desktop viewports.
All routing is handled by Gatsby
All client and server communications are performed using https.
stamps.johnvincent.io fully implements Google Analytics
stamps.johnvincent.io fully supports Google Webmaster Tools
stamps.johnvincent.io is deployed to an Ubuntu droplet at Digital Ocean and kept running using PM2
stamps.johnvincent.io resources are served from Nginx Server
The Ubuntu droplet at Digital Ocean can only be accessed with SSH from a particular client. All other access is disabled.

Performance

Deploy Stamps

stamps.johnvincent.io is deployed to a Digital Ocean Droplet.

Please see Configure and Deploy to Ubuntu Droplet at Digital Ocean for deployment details.

Maintenance

The following describe tasks required for the maintenance of stamps.johnvincent.io at Digital Ocean.

Update SSL Certificates to Ubuntu at Digital Ocean

Maintaining Droplets at Digital Ocean

Website Review

Website Validation

Website Validation Reference

Stamps Website Validation

Client Technologies

Production Deployment Technologies

Application Calculations

To make an estimate as to how many pages/urls Gatsby would need to generate, some facts need to be gathered:

Since 1840, when the first stamp was issued in England, over 780 countries have issued stamps. About 250-300 of these countries still exist.
Probably about 500,000 different stamps have been produced worldwide.

For very rough estimation purposes, lets assume:

500,000 stamps
300 countries
3 stamps per set

which roughly calulates to:

stamps per country: 1700
sets per country: 570

If each country, on average started issuing stamps in 1850, then:

Number of decades issuing stamps: 18
Number of years issuing stamps: 173
Number of sets per year: 3.5
Number of stamps per set: 3

The usual architecture of a catalog application would require the following navigation:

List of countries
Country, select a decade
Decade, select a year
Year, select a set
Set, select a stamp
Stamp, provide full details

Such an architecture would require the following pages be generated:

(300)*(18)*(10)*(3.5)*(3) = 567,000

There would doubtless also be many other pages to generate.

Generating such a large number of pages:

will take a very considerable amount of time and resources.
will slow development to a crawl.
will require the pages to be very simple to minimize memory usage.
may never generate due to out of memory errors.
will be impossible to test.

While this architecture would probably work, given sufficient resources, I chose a more practical approach.

Application Architecture

I chose to work with 28 countries. Creating the data for more countries would require considerable time and may be implemented in the future.

For this application, I created all of the pages with gatsby-node.js. The createPages function provides excellent control over the generation of the pages.

Catalogs Section for SEO

For SEO purposes, I created the Catalogs section which is accessible from a link in the footer. This section has the following navigation:

List of countries, select a country
Country, select a decade
Decade, select a year
Year

This architecture requires the following pages be generated:

(28)*(18)*(10) = 5040

which is very manageable.

When all countries are implemented the following pages would be generated:

(300)*(18)*(10) = 54,000

which is still very manageable.

These pages are mostly just links, although the Year pages have stamp images and details. They are not meant for a human user.

Catalog Section for Users

The country catalogs which are displayed on the home page all link to the country catalog section, for example https://www.stamps.johnvincent.io/catalog/Australia

This page is generated for each country, thus a total of 28 pages. All of these pages are generated from the same template MainTemplate.jsx.

Snippet from gatsby-node.js

catalogs.forEach(({ data }) => {
	const { country, ckey } = data;
	const url = `/catalog/${ckey}/`;
	createPage({
		path: url,
		component: require.resolve('./src/templates/MainTemplate.jsx'),
		context: {
			pageMetaData: { permalink: url, meta_title: `Stamps of ${country}` },
			config,
			info: { type: 'catalog_main', country, ckey },
			slug: ckey
		}
	});
});

MainTemplate.jsx

import React from 'react';

import { graphql } from 'gatsby';

import Layout from '../containers/Layout';
import { SEO } from '..';
import Main from '../components/catalog/Main';

export default function MainTemplate(
	{ data: { catalogsAllJson: catalog }, pageContext: { pageMetaData, config, info }
	}
) {
	return (
		<Layout config={config}>
			<SEO config={config} pageMetaData={pageMetaData} />
			<Main config={config} info={info} catalog={catalog} />
		</Layout>
	);
}

export const data = graphql`
query ($slug: String!) {
  catalogsAllJson(ckey: { eq: $slug }) {
		ckey
		country
		data {
			years {
				sets {
					idx
					title
					year
					stamps {
						id
						description
						currency
						image
					}
				}
				year
			}
			country
			code
			idx
			ckey
			decades {
				from
				to
				list
			}
			stats {
				range {
					high
					low
				}
				counts {
					sets
					stamps
				}
			}
			ckey
		}
  }
}
`;

The key is to pass from gatsby-node.js

slug: ckey

in the page context to the template. GraphQL takes the value of $slug from the page context.

Main.jsx is a React component which provides for the dynamic nature of the application.

Catalog Search

Search capabilities are not static and never can be. It is always dynamic. This requires a specific architecture.

MainTemplate.jsx provides the Search component which is referenced in the Header component.

The Search component allows the user to enter a query string, the application then links to the country search page, for example https://www.stamps.johnvincent.io/search/Belgium?query=leopold

Notice the use of a query string.

This page is generated for each country, thus a total of 28 pages. All of these pages are generated from the same template SearchTemplate.jsx.

Snippet from gatsby-node.js

catalogs.forEach(({ data }) => {
	const { country, ckey } = data;
	const url = `/search/${ckey}/`;
	createPage({
		path: url,
		component: require.resolve('./src/templates/SearchTemplate.jsx'),
		context: {
			pageMetaData: { permalink: url, meta_title: `Stamps of ${country}` },
			config,
			info: { type: 'catalog_search', country, ckey },
			slug: ckey
		}
	});
});

SearchTemplate.jsx

import React, { useState, useEffect } from 'react';

import { graphql } from 'gatsby';

import Layout from '../containers/Layout';
import { SEO, Header } from '..';
import Main from '../components/catalog/Main';

import search from '../components/utilities/search';

export default function SearchTemplate(props) {
	const [hydrated, setHydrated] = useState(false);
	useEffect(() => {
		setHydrated(true);
	}, []);
	if (!hydrated) {
		return null;		// Returns null on first render, so the client and server match
	}

	const { location, data: { catalogsAllJson }, pageContext: { pageMetaData, config, info }} = props;

	let noData = true;
	let catalog = {};
	const params = new URLSearchParams(location.search);
	const query = params.get('query');
	if (query && query.length > 0) {
		catalog = search(query, catalogsAllJson);
		const { data } = catalog;
		const { decades, years } = data;
		noData = !decades || decades.length < 1 || !years || years.length < 1;
	}

	if (noData) {
		return (
			<Layout config={config}>
				<SEO config={config} pageMetaData={pageMetaData} />
				<Header info={{ ...info, query }} />
			</Layout>
		);
	}

	return (
		<Layout config={config}>
			<SEO config={config} pageMetaData={pageMetaData} />
			<Main config={config} info={{ ...info, query }} catalog={catalog} />
		</Layout>
	);
}

export const data = graphql`
query ($slug: String!) {
  catalogsAllJson(ckey: { eq: $slug }) {
		ckey
		country
		data {
			years {
				sets {
					idx
					title
					year
					stamps {
						id
						description
						currency
						image
					}
				}
				year
			}
			country
			code
			idx
			ckey
			decades {
				from
				to
				list
			}
			stats {
				range {
					high
					low
				}
				counts {
					sets
					stamps
				}
			}
			ckey
		}
  }
}
`;

Again, pass from gatsby-node.js

slug: ckey

in the page context to the template. GraphQL takes the value of $slug from the page context.

Notice

export default function SearchTemplate(props) {
	const [hydrated, setHydrated] = useState(false);
	useEffect(() => {
		setHydrated(true);
	}, []);
	if (!hydrated) {
		return null;		// Returns null on first render, so the client and server match
	}
...

It is a requirement for the server and the client to initially produce identical output.

Hydration is the process of using client-side JavaScript to add application state and interactivity to server-rendered HTML. Gatsby uses hydration to transform the static HTML created at build time into a React application.

This code allows for the server and client to initially generate identical code, and then for the client to act as a React application, or as it were, it transforms static to dynamic.

Gatsby Data

All external data is added to data/catalogs and loaded into Gatsby/GraphQL using the plugin gatsby-source-filesystem.

In practice, for the data to be useful to the application via GraphQL, the data must be in a very particular format. GraphQL has it's own requirements. Violate these and will get the following error on loading the json with plugin.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

This may be remedied with:

npm run start --node-flags --max-old-space-size=4096--no-warnings

however this error, without stating as such, is actually indicating a problem with the architecture of the data.

Loading Json hashmaps into GraphQL quickly caused out of memory errors.

However, the application also has very specific requirements of the data. It is extremely difficult to architect the data so it will quickly load into GraphQL in a format that is useful to the application and for the application pages to be generated within a reasonable timeframe.

In practice, the source data requires a complex transformation application to provide the data in a useful format.

Gatsby Images

This application is very image intensive. To maximize the value of Gatsby all images should use gatsby-plugin-image. In practice, generating the vast number of images using the plugin takes large amounts of memory and time. It is not practical for this application to load it's images using gatsby-plugin-image.

Debugger Problem

The Visual Studio Code debugger failed with

Cannot find module '../build/Release/sharp-darwin-x64.node'

The problem is in gatsby-plugin-sharp. There appears to be no solution other than to upgrade to Gatsby V5.

See Migrating Gatsby from V4 to V5/

This would force upgrades to Node V18 and to React 18, which causes upgrades to many packages that are not compatible. For example:

npm outdated

npm install gatsby@latest --legacy-peer-deps

npm install react@latest react-dom@latest --legacy-peer-deps

which caused more errors with "@lmdb/lmdb-darwin-arm64": "2.6.0-alpha6"

I concluded that migrating to Gatsby V5 was not possible at this time.

End

Configure and Deploy to Ubuntu Droplet at Digital OceanConfiguring DMARC for Google Workspace