Postcode-SIMD reference tables

Phrases in the titles of 263 papers that use SIMD, 2005–2026, from SIMD literature word clouds, Michele Tinti, CC BY 4.0.

Since its first edition in 2004, the Scottish Index of Multiple Deprivation has been used in more than 250 published studies, found by repeated literature searches; the list is close to complete but not guaranteed to be. This repository fetches every public source needed to attach SIMD to health and administrative records, each pinned by its SHA256, and builds tables that link by postcode. Every row also carries its 2001, 2011 and 2022 data zones, so SIMD can be joined by data zone as well.

Column provenance: https://mtinti.github.io/postcode-simd/. For every column of both tables: where its value comes from, what was done to it, how the build checks it, and why.

Pinned sources archived on Zenodo: https://doi.org/10.5281/zenodo.23119742. Every open-licence download the build reads, byte for byte, with its licence evidence and credit; the NRS postcode products are cited there, not redistributed. The build falls back to these copies if a publisher moves a file, and verifies them by SHA256 like any other.

A CLI pipeline that rebuilds two postcode-to-SIMD tables from hash-pinned public files. Routine maintenance is a refresh of one of the two NRS postcode products. Adding a SIMD edition is a separate, less frequent change to the source registry and output schemas.

Table	Postcode source	One row per	Key	Answers
postcode_simd.parquet, the main table	Scottish Statistics Postcode Lookup (SSPL) 2026/2	whole postcode, latest life, both user types: 230,103 rows × 399 columns	pc_norm	SIMD on the latest postcode; either one edition or edition by event year
postcode_simd_history.parquet, the history table	Scottish Postcode Directory (SPD) 2026/2	postcode life, both user types: 247,773 rows × 442 columns	pc_norm, introduced_on	date-valid postcode records, split parts, or explicitly chosen SPD latest linkage

Six SIMD editions (2004–2020v2) are attached to both. The two NRS products allocate data zones differently: the SSPL takes the zone containing the centroid of the postcode's 2022 output area, the SPD the zone containing the postcode itself. Every build reports observed data-zone and band differences; release changes and source corrections can also contribute. Neither table has been validated against an official PHS postcode-level lookup. Choose one product consistently for an analysis. Each new release replaces its table completely.

Start here

For a human review, read How it is built, then the build's results/BUILD_REPORT.md. The report shows the actual checks, changes since the previous snapshot, and an example postcode. Updating the sources is the maintenance runbook. The data dictionaries describe the columns of the main and history tables.

Install and build

Python 3.12 or newer; runtime versions are pinned in pyproject.toml.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Verify files already supplied under manual_data/, then rebuild.
simd-ingest build

# Or download the explicitly pinned release, then rebuild.
simd-ingest build --source-mode download

python -m simd_ingest.cli is equivalent to simd-ingest. Settings live in config/workflow.yaml; use --config <path> for a separate review output directory.

simd-ingest fetch --source-mode download  # download/verify only
simd-ingest audit --source-mode download  # read-only audit of data/sources and results
python -m simd_ingest.trace "AB12 3GQ"     # follow one main-table record back to its source rows
python -m simd_ingest.trace "AB12 3GQA" --table history
python -m pytest tests -q

Offline mode never accesses the network. Audit never downloads or repairs files, regardless of source mode. Download mode rechecks cached hashes; it does not discover or accept “latest” automatically.

What a build leaves

• results/postcode_simd.parquet: the main table, keyed by pc_norm.
• results/postcode_simd_history.parquet: the history table, keyed by (pc_norm, introduced_on). Besides SIMD it carries the Scottish Government Urban Rural Classification of all nine published versions, 2003-2004 to 2022, for every postcode life: urbanrural<version>_6fold, _8fold and _status. Each life's own grid reference is placed in that version's polygons, which reproduces the 2022 codes NRS publishes on every current life. A code is null, and the status says why, for a point outside every polygon, a point on an edge between two classes, and a post-office box. Read with pandas, a nullable code column arrives as a float; cast it to Int8.
• Both tables carry the Scottish Government's SIMD domain ranks for every edition, simd<edition>_<domain>_domain_rank: income, employment, health, education, access, crime and housing, copied exactly as published. They are unweighted ranks, not PHS population-weighted bands, and they are not whole numbers: many end in .5, so read them as decimals. 2004 published no crime domain, so there is no 2004 crime column.
• Both tables also carry each domain's quintile, decile and vigintile, simd<edition>_<domain>_domain_<band>, as the Scottish Government published them on statistics.gov.scot: copied, never derived, so a few tied zones sit in adjacent bands exactly as published. The ranks still come from the shapefiles; every build checks that the published ranks are the same ranking before any band is used. For 2020v2 housing, two Government publications give 628 zones slightly different ranks; those zones carry simd2020v2_housing_domain_rank_source_status = rank_sources_disagree, which describes the rank, not the band, and is no reason to exclude them.
• Both tables also carry a computed population-weighted Scotland quintile and decile of each domain rank, simd<edition>_<domain>_domain_pw_scotland_<band>. Nobody publishes these: they are cut from every data zone of the edition by a population-midpoint rule that each build checks reproduces PHS's published overall Scotland bands exactly; equal ranks share a band. They sit beside the Government's published, unweighted domain bands, a different basis: report which you used. simd<edition>_population is the data-zone population they were weighted by, repeated on every postcode in the zone, so never sum it or weight by it across rows. Release is metadata/an attribute of each table, not part of its key.
• results/postcode_simd.csv.gz and results/postcode_simd_history.csv.gz: the same rows as a gzipped CSV, the form in which a table is shared. They omit the grid reference and coordinate columns, which the Parquet keeps. results/CSV_README.txt carries the attribution.
• results/BUILD_REPORT.md: the human review entry point, including the agreement between the two tables.
• results/manifest.json: all checks, observations, input pins and both tables' fingerprints.
• results/runs/<run-id>/: retained configuration/source/schema/decision snapshots, runtime and code hashes, and that run's report. Failed builds retain their error and checks too.

The build writes both candidates, reopens them and checks saved values and metadata before replacing the current tables. Validation failure leaves the previous publication untouched. Run one writer per results directory; files are replaced atomically individually, not as a multi-file transaction. After an interrupted publication, run audit; if it fails, rebuild from pinned sources. Keep results/runs, source archives and the corresponding code in your backups. Old table copies are not retained.

There is no scheduler, server, intermediate-table cache or orchestration database. The old orchestration plans remain historical records.

Use the tables

The main table has one row per whole postcode. To inspect its latest source record (including a deleted latest life):

SELECT pc_norm, is_current, simd2020v2_pw_scotland_quintile
FROM 'results/postcode_simd.parquet'
WHERE pc_norm = 'G718BQ';

The history table keeps every life and the NRS split parts. A question about a date, or about which split part applies, goes there:

SELECT pc_norm, pc_base, introduced_on, deleted_on, simd2020v2_pw_scotland_quintile
FROM 'results/postcode_simd_history.parquet'
WHERE pc_base = 'G718BQ' AND introduced_on <= DATE '2015-06-01'
  AND (deleted_on IS NULL OR deleted_on > DATE '2015-06-01');

pc_norm removes ASCII spaces; in the history table it keeps the NRS split suffix and pc_base removes a validated suffix from a flagged small-user record. Original postcode text is preserved. PHS population-weighted fields (pw) and Government unweighted fields (uw) are distinct.

For cohort linkage in SQL there are two sets of queries, one per postcode product and neither the default: docs/sql/spd/ reads the history table and docs/sql/sspl/ reads the main table. Each set has link_by_era.sql, which chooses the SIMD edition from the year of the health data by PHS Table 4, and link_latest.sql, which uses one edition throughout. Every query is standalone, written as numbered steps that name the guidance or project choice. All five queries share 84 core columns: statuses, product provenance, both record keys, the edition's data and intermediate zones and PHS geography codes, all 14 stored measures, the seven Scottish Government domain ranks with their published quintiles, deciles and vigintiles, the 2020v2 housing source status, and the computed population-weighted quintile and decile of each domain. Product-specific own-record context follows (132 total columns for SSPL, 151 for SPD era/latest, 154 for SPD as-of); select shared columns by name when combining results. The SPD set selects the latest life and the A part itself; the SSPL set does not, because NRS did. Both attach a large user's SIMD through its linked small-user postcode and give PO boxes none. Applying that rule to SSPL is a project interpretation of PHS Appendix A, not verified parity with PHS's own lookup. NRS recommends SSPL for statistical production and SPD for operational/administrative use; the project keeps this choice explicit. The SPD set adds link_as_of.sql for a reliable address date: it selects the postcode life valid on that date, not historical administrative snapshots. The three SPD queries also return the Urban Rural Classification version that suits the year, chosen by reference year as a project choice, with a status saying why when it is empty. It needs only the address date: the year of that date chooses the edition, and analysis_year is an optional override for holding one edition across a study. The step-by-step commentary is in LINKAGE_BY_ERA.md.

To review the logic before trusting it, read walkthrough_as_of.sql: the dated query written short, every step with its reason, returning the rank, both publishers' bands and the urban-rural class. A test requires it to agree with the generated query case for case.

To load the shared CSVs into SQL Server, import_csv.sql stages and types each file and check_loaded_digest.sql verifies the structure, the key, the row count and the row digest the build recorded. tests/ground_truth.csv holds 360 boundary cases, a data zone at the edge of every quintile and decile for every edition and both publishers, each with a postcode, for testing a loaded table or a new query; regenerate it with python -m simd_ingest.ground_truth --seed 2026.

The Python helpers read either table: current lookups against the main table, current or as-of lookups against the history table, with own-record geography and an optional split consensus/conflict policy on history. Date-valid and split-report questions against the main table are refused; the SQL sets choose an edition by year without needing postcode history. Exclusions remain consumer choices, not deletions from the tables. Adding a source edition does not automatically change the analyst's edition-by-year policy.

Optional Docker runner

docker compose build
docker compose run --rm build
docker compose run --rm test

This packages the same CLI, with sources and results mounted from the host. See Docker for offline and audit commands.

Code to read

cli.py handles arguments; pipeline.py contains the single build sequence. core/sspl.py and core/spd.py read the two postcode products, phs.py, govscot.py, join.py and output.py contain the shared data rules, and core/agreement.py compares the two tables. core/text_output.py writes the shared CSV and the digest a database load is checked against. sources.yaml declares sources and editions, sspl_schema.yaml and spd_schema.yaml declare the postcode file headers, output_schema.yaml and output_schema_history.yaml declare the saved tables, export_contract.yaml says what leaves the build, and decisions.yaml records policy. Tests include synthetic postcode refreshes of both products, a new SIMD edition/vintage, corrupted inputs and saved files; downloaded-data tests also pin both tables' rows-only fingerprints.