Dataverse Fixity Validation

Fixity (checksum) validation and remediation tools for Dataverse, developed for Borealis, the Canadian Dataverse Repository, in support of the Borealis Preservation Plan.

About fixity validation

Borealis conducts routine fixity checks to ensure file integrity over time. A fixity check compares a file’s current checksum against the checksum recorded when the file was deposited or last updated to ensure the file has remained unchanged in storage.

Fixity checking verifies bit-level integrity. It does not assess the quality or completeness of research data, nor does it ensure that file formats will remain usable or readable over the long term. Responsibility for these activities is shared through broader preservation and curation practices outside the scope of this tool.

For each file uploaded to Borealis, the fixity check process involves:

1. Dataverse automatically generates and stores an MD5 checksum, or fingerprint, for each file.
2. Borealis runs routine fixity validation using the Dataverse Native API. The validation process checks stored files against their recorded checksum values. A fixity check is successful when the checksum of the stored file matches the value recorded during ingest or a subsequent authorized update.
3. The results of the fixity checks are written to a separate MySQL database for tracking and reporting purposes.
4. Files that fail validation are reviewed by staff. Where remediation is required, a trusted replacement copy may be requested and processed using the remediation workflow.

In the Borealis production environment, a full validation run is scheduled monthly.

Tools in this repository

This repository contains two complementary tools:

• checksum_validate — Detector: performs routine fixity validation, tracks validation status in a separate MySQL database, and reports validation failures.
• checksum_update — Remediator: supports administrator-led remediation of failed files, including replacing a stored file with a trusted copy or removing an affected file when necessary.

The general workflow is:

┌─────────────────────────────┐      monthly (~3 days, prod app3)
│      checksum_validate      │  ── walks dataverse tree
│       (the DETECTOR)        │  ── POST /api/admin/validate/dataset/files/{id}
│                             │  ── tracks state in its own MySQL DB
│                             │  ── emails CSV + HTML report
└──────────────┬──────────────┘
               │  failures reviewed by staff; backup copies requested
               ▼
        files.csv  (file_id, nextcloud URL, date, provenance note, [status])
               │
┌──────────────┴──────────────┐      on demand (Jenkins: DV_Checksum-File_Update)
│       checksum_update       │  ── downloads replacement from Nextcloud
│      (the REMEDIATOR)       │  ── overwrites S3 object (aws s3 cp)
│                             │  ── updates Dataverse Postgres directly
│                             │     (checksum/filesize/contenttype, prov note,
│                             │      or full file+metadata delete)
└─────────────────────────────┘

Detector: checksum_validate/

Walks all dataverses/datasets under dataverse_alias (pyDataverse), asks Dataverse to validate each dataset's files, re-tests failures individually (/api/admin/validateDataFileHashValue/{id}), and records per-file status in MySQL (validation_status + errors — see create_tables.sql). Files checked within the last min_frequency days (default 14) are skipped, so a 3-day run resumes cheaply if interrupted. Sends a condensed e-mail report with two CSV attachments (error report + per-dataset summary).

python3 checksum_validation.py -c config/config.ini [-l DEBUG] [-f] [-i RUN_ID] [-d DATASETS_FILE]
#  -f  force: ignore min_frequency           -i  run id used in log/report filenames
#  -d  cache file for the tree-walk dataset list (exists -> loaded, else written)

• Must run on an app node (/api/admin/* is localhost-only).
• Config: copy config/config.ini.sample to config/config.ini (gitignored) and fill in your values; validated against config/config_spec.ini. Sections: DATAVERSE_APP (base_url, api_token, dataverse_alias), VALIDATION_APP (num_threads, timeout, min_frequency, log_dir), MYSQL_DB, REPORTER (smtp, sender, recipients).
• Resume support: re-running skips datasets whose files were all checked within min_frequency; -d/--datasets-file PATH caches the tree-walk enumeration (file exists → loaded, skipping the walk; missing → walk runs and writes it). Requests retry with backoff for ~10 min on connection errors/5xx, so a Payara restart mid-run pauses the run instead of killing it.

Reading the report

The e-mail (subject Dataverse TDR Report #<run-id> (<date>)) carries a condensed summary plus two CSV attachments. Gotchas in the summary lines:

• "Number of datasets skipped" (top) = datasets the check phase did not forward to validation for any reason; "Total number of sets/files skipped" (bottom) = only those skipped by the min_frequency rule. The two can legitimately differ.
• "Datasets dropped (API errors)" = datasets that could not be validated this run (also listed in the error CSV with status dropped); they will be retried next run since their timestamps were not updated.
• A scheduled monthly run normally validates everything (the monthly gap exceeds min_frequency); large skip counts are expected only on resume/re-runs.

Attachments: ..._errors_<date>.csv (dataset_id, pid, datafileId, storageIdentifier, status, errorMessage — one row per invalid file and per dropped dataset; a single all-NONE row means a clean run) and ..._dataset_summary_<date>.csv (dataset_id, pid, total, valid, invalid per dataset).

Remediator: checksum_update/

Jenkins-invoked (run_dv_checksum_update.sh <build#> <csv path>); builds a venv and runs update_checksum.py against an uploaded CSV. Per row, by status column: (empty) replace the S3 object, recompute MD5/filesize/contenttype, update Postgres, re-ingest tabular files; D delete the physical S3 file (keep metadata, set provenance note); A delete the file and all its metadata (~16 tables). CSV format and column semantics: see checksum_update/README.md.

Requires: Dataverse superuser API token, direct Postgres credentials, aws CLI configured for the S3 endpoint, network access to the trusted file-share host (replacement_url_pattern in config). Copy checksum_update/config.ini.sample to config.ini and fill it in. This tool mutates production data — test any change against a non-production copy of your installation first.

Deploying at another Dataverse installation

Both tools were built for Borealis but have no Borealis-only logic; to adapt them, know the assumptions:

1. Detector setup: create the tracking database from create_tables.sql (MySQL/MariaDB), pip install -r requirements.txt into a venv, copy checksum_validate/config/config.ini.sample → config.ini, run on an application node (the /api/admin/ validation endpoints are localhost-only).

2. Remediator assumptions — verify before pointing it at your installation:

   • it writes checksumtype='MD5' and computes MD5 when replacing files — if your installation uses SHA-1/256/512 (:FileFixityChecksumAlgorithm), adapt update_checksum() first;
   • it manipulates the Dataverse PostgreSQL schema directly (datafile, filemetadata, datatable cascade, dvobject, ...) — schema changes between Dataverse versions can break it; test against a non-production copy of YOUR Dataverse version;
   • it assumes an S3 file store driven via the aws CLI (originally an OpenStack Swift s3api endpoint — hence copy-aside backups instead of bucket versioning); file-store installations need replace_file/delete_file adapted;
   • set replacement_url_pattern to your own trusted file-share host.

3. Python 3.8+ (tested on 3.12 / Ubuntu 24.04); the remediator needs the libmagic system library (Ubuntu: libmagic1) and the aws CLI; no services are bundled — scheduling (cron/CI) is up to you. Example monthly detector cron (runs take days; re-runs after an interruption are cheap because of the min_frequency skip):

   0 3 1 * * cd /opt/checksum_validate && venv/bin/python3 checksum_validation.py -i "$(date +\%Y\%m)" >/dev/null 2>&1
   

Licensed under Apache-2.0 (see LICENSE), provided as-is with no support guarantees. Bug reports via issues are appreciated. Because these tools operate directly on the Dataverse database and storage, external pull requests are not actively solicited and are unlikely to be merged — please fork and adapt for your own installation instead.