From a7b5918f587689abbb5cdceb6ca879568c870f46 Mon Sep 17 00:00:00 2001 From: David Roe Date: Tue, 21 Jul 2026 13:06:26 -0400 Subject: [PATCH 1/6] Prepare 1.0.0 release: README, CHANGELOG, packaging, release workflow Rewrite README.md as the PyPI landing page: a dict-in/dict-out pitch, a verified quickstart (config.ini + PostgresDatabase(create=True) + create_table + insert_many + search), a feature list covering the 1.0 surface (joins, $col/$size, bulk loading with reload/revert, stats/counts caching, schema management with a versioned metadata format and migrations, staged writes, introspection, notifications, diffing), absolute-URL documentation links, and supported versions (Python >= 3.8, PostgreSQL 13-18, psycopg 3). Drop the "built upon psycopg2" falsehood. Add CHANGELOG.md (Keep-a-Changelog) with a 1.0.0 section: breaking changes each with a migration note, additions, curated fixes, and infrastructure. Bump pyproject to 1.0.0; declare the license as the SPDX expression GPL-2.0-or-later with license-files (setuptools >= 77 rejects a License trove classifier alongside an SPDX expression); add Development Status, audience, Python 3.8-3.13 and topic classifiers; add Homepage/Repository/Changelog URLs. Add .github/workflows/release.yml: a Trusted-Publishing workflow that builds, twine-checks, and uploads to PyPI on a published GitHub release, with the one-time PyPI setup documented inline. Co-Authored-By: Claude Fable 5 --- .github/workflows/release.yml | 87 ++++++++++++++++ CHANGELOG.md | 131 +++++++++++++++++++++++++ README.md | 180 +++++++++++++++++++++++++++++----- pyproject.toml | 22 ++++- 4 files changed, 391 insertions(+), 29 deletions(-) create mode 100644 .github/workflows/release.yml create mode 100644 CHANGELOG.md diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..ac08cb0 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,87 @@ +# Publish psycodict to PyPI when a GitHub release is published, using PyPI +# Trusted Publishing (OpenID Connect). No API token is stored in the +# repository: the publish job proves its identity to PyPI with a short-lived +# OIDC token minted by GitHub for the `pypi` environment. +# +# ONE-TIME PyPI SETUP (maintainer, before the first release): +# +# 1. On https://pypi.org, register a "pending publisher" so the first upload +# can create the project via trusted publishing. Go to your account -> +# "Publishing", and add a new pending publisher with exactly these values: +# PyPI Project Name: psycodict +# Owner: roed314 +# Repository name: psycodict +# Workflow name: release.yml +# Environment name: pypi +# (If the project already exists on PyPI, add the trusted publisher under +# the project's Manage -> Publishing page instead.) +# +# 2. In this repository, go to Settings -> Environments and create an +# environment named "pypi" (matching the `environment:` below). Optionally +# add required reviewers or a tag restriction there as an extra guard on +# who and what can publish. +# +# RELEASE FLOW: +# +# Bump `version` in pyproject.toml and update CHANGELOG.md, merge to master, +# then create a GitHub Release with the tag `v1.0.0` (matching the version). +# Publishing that release triggers this workflow: the sdist and wheel are +# built, metadata-checked, and uploaded to PyPI. Nothing is published on +# pushes, pull requests, or draft releases. + +name: Release + +on: + release: + types: [published] + +permissions: + contents: read + +jobs: + build: + name: Build distributions + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + cache: pip + + - name: Build sdist and wheel + run: | + python -m pip install build twine + python -m build + + # The same gate CI enforces, so a release can never upload metadata that + # PyPI would reject at the end of the process. + - name: Check metadata + run: twine check --strict dist/* + + - uses: actions/upload-artifact@v7 + with: + name: dist + path: dist/ + + publish: + name: Publish to PyPI + needs: build + runs-on: ubuntu-latest + # Must match the environment named in the PyPI pending-publisher setup above. + environment: pypi + permissions: + # id-token: write lets the job mint the OIDC token that trusted + # publishing exchanges for a one-time PyPI upload token. + id-token: write + contents: read + steps: + - uses: actions/download-artifact@v7 + with: + name: dist + path: dist/ + + # No username/password/token inputs: trusted publishing authenticates + # via the OIDC identity of this `pypi` environment. + - uses: pypa/gh-action-pypi-publish@release/v1 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..33e51d3 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,131 @@ +# Changelog + +All notable changes to psycodict are documented here. The format is based on +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project +follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## 1.0.0 + +The first release with a stable, documented API. It consolidates a large body +of standalone bug fixes, a driver port, a rebuilt query and data-management +layer, and full test coverage. Because it tightens several long-standing +defaults, it contains breaking changes; each is listed below with a one-line +migration note. + +### Breaking changes + +- **The search/extras table split is gone.** Tables are now a single table + instead of a `search` table paired with an `extras` table. *Migration:* rerun + `create_table` without the extras argument; columns that used to live in the + extras table are ordinary columns now. (#59) +- **Ported from psycopg2 to psycopg 3.** psycodict no longer depends on + psycopg2, and it re-exports `SQL`, `Identifier`, `Placeholder`, `Literal`, + `Composable` and `Composed` from `psycodict` so callers need not import a + driver directly. *Migration:* install with the `pgbinary` or `pgsource` extra + (`pip install "psycodict[pgbinary]"`), and import the SQL composition classes + from `psycodict` rather than `psycopg2.sql`. (#88) +- **The old `join_search` method is removed.** *Migration:* pass `join=` to + `search` (or `count` / `lucky`) instead; see the joins section of + QueryLanguage.md. (#89) +- **Non-public methods are underscore-prefixed.** The following are now private: + `cursor` → `_cursor`, `log_db_change` → `_log_db_change`, + `is_alive` → `_is_alive`, `is_read_only` → `_is_read_only`, + `can_read_write_knowls` → `_can_read_write_knowls`, + `can_read_write_userdb` → `_can_read_write_userdb`, + `register_object` → `_register_object`, `logger` → `_logger`, + `has_id` → `_has_id`. *Migration:* add the leading underscore at call sites, + or stop calling them — none were part of the intended public API. (#92) +- **`count` and `count_distinct` no longer cache by default** (`record=False`). + Search-page counts are still recorded. *Migration:* pass `record=True` to + cache a count you will ask for repeatedly. (#93) +- **Numeric precision is derived from significant digits**, and an all-zero + decimal round-trips to an exact integer zero rather than a slightly different + float. *Migration:* none needed unless you relied on the previous, less + precise rounding. (#94) +- **`reindex=False` is now an error in `rewrite` and `update_from_file`** + instead of being silently ignored; `rewrite` no longer accepts the parameter + at all, and its trailing parameters are keyword-only. *Migration:* drop + `reindex=False`, or reindex explicitly after the operation. (#96) +- **`include_nones` defaults to `True`**, and the choice is stored per table. + *Migration:* pass `include_nones=False` at `create_table` to keep the old + behavior of omitting `None`-valued keys from result dictionaries. (#103) +- **Metadata format 1.** The layout of the `meta_*` tables is now versioned + (format 0 is the unstamped 0.x baseline; format 1, aligned with this major + release, adds `meta_indexes.whereclause`), stamped in the new `meta_format` + table and checked on connect. A format-0 database keeps working — every + connection warns and operates at the old format, with the format-1 features + unavailable — and read-only replicas need no action. See + MetadataFormats.md for the compatibility policy. *Migration:* connect once + with `PostgresDatabase(upgrade=True)` (or run `db.upgrade_metadata()`) to + migrate the `meta_*` tables in place and silence the warning. **Caveat:** + do not run a pre-1.0 psycodict against a database a 1.0+ psycodict has + created or migrated — it predates `meta_format`, so it cannot be told to + refuse, and its index/reload paths would silently drop partial-index + predicates. This only matters during a mixed-version rollout. (#90, #110) + +### Added + +- **Joined queries.** `join=` on `search`, `count` and `lucky` supports inner, + left, right and full joins, with `"table.column"` qualification in queries, + projections, sorts, `$col` and `$raw`, and dotted paths in joined + projections. (#89) +- **New query operators.** `$col` compares two columns, `$raw` documents the + raw-SQL escape hatch, and `$size` constrains array/jsonb cardinality; path + specifiers now work in plain projections and sorts, not just queries. + (#89, #107, #108) +- **Partial indexes**, declared through the meta-index machinery (part of + metadata format 1). (#110) +- **`staged()` transactional uploads** — a context manager that groups writes + with write exclusion and drift detection. (#100) +- **`db.refresh_tables()`** re-reads the schema so structural changes are picked + up without restarting the process. (#99) +- **Metadata format protocol.** The `meta_format` stamp — `(version, + min_compat)` — written at creation and checked on connect; older-but- + compatible databases warn and degrade gracefully instead of being refused, + newer databases are accepted when their stamped `min_compat` admits this + psycodict, migrations run stepwise via `db.upgrade_metadata()` / + `PostgresDatabase(upgrade=True)`, and `db.meta_format` exposes the format a + connection operates at. MetadataFormats.md documents the policy and the + checklist for future format changes. (#90, #110) +- **Query and lock introspection.** `show_queries` and `show_blocked` report + running and blocked queries via `pg_blocking_pids`, and reloads now warn about + outstanding locks. (#91) +- **Slow-query log analysis** in the new `psycodict.slowlog` module. (#102) +- **`db.compare` / `db.show_differences`** detect drift between two databases. + (#101) +- **Schema-change notifications** and a small publish/subscribe layer built on + PostgreSQL LISTEN/NOTIFY. (#111) +- **Documentation.** `DataManagement.md` (write side) and `Searching.md` (read + API), plus docstring housekeeping across the package. (#104, #105, #106) + +### Fixed + +Roughly two dozen fixes landed while splitting the search/extras tables and +hardening standalone use; the highlights: + +- Counts-table totals are now maintained correctly on writes rather than drifting + out of sync. (#61–#87) +- `text[]` values no longer corrupt on `COPY` because of quoting. (#61–#87) +- A Python `None` in a jsonb value maps to SQL `NULL` with consistent semantics, + including nested jsonb decoding. (#61–#87) +- `$maxgte` and related operators work without requiring custom SQL functions. + (#61–#87) +- Statistics and counts no longer accumulate duplicate rows, and `refresh_stats` + converges. (#97) +- Sage-free statistics, fresh-database bootstrap (`create=True`), and jsonb + `$in` / `$nin` on composite values all work in standalone (non-LMFDB) use. + (#56, #57) +- A leftover `_tmp` index or constraint is caught by a preflight check before a + reload or rewrite. (#95) +- `create_table_like` preserves per-column `STORAGE` / `COMPRESSION` settings and + analyzes the copy. (#98) + +### Infrastructure + +- A test suite of 500+ tests and a continuous-integration workflow covering the + supported Python and PostgreSQL versions, plus downstream regression jobs that + run LMFDB and seminars against the new code. (#58) + +## 0.1.x and earlier + +Pre-1.0 releases did not keep a changelog. diff --git a/README.md b/README.md index 945fb40..325c9f4 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,178 @@ -# psycodict: dictionary-based python interface to PostgreSQL databases +# psycodict -This project was split off from the [L-functions and modular forms database](https://www.lmfdb.org) -so that other projects could use the SQL interface that we created for that project. +A dictionary-based Python interface to PostgreSQL, extracted from the +[L-functions and Modular Forms Database](https://www.lmfdb.org) (LMFDB) so that +other projects can use the SQL interface built for it. Queries are Python +dictionaries and results are Python dictionaries: you describe *what* you want +as a `dict`, psycodict turns it into SQL, runs it, and hands the rows back as +`dict`s. On top of that query language it provides the machinery a large, +mostly read-only research database needs — bulk loading from files, cached +statistics and counts, schema management, and tools for keeping copies of a +database in sync. -Built upon [psycopg2](https://pypi.org/project/psycopg2/), the core of the interface is the ability to create -SELECT queries using a dictionary. In addition, the package provides a number of other features that were useful for the LMFDB: +## Install - * Data management tools wrapping PostgreSQL's mechanisms for loading from and saving to files (see [DataManagement.md](DataManagement.md)) - * Statistics tables for storing statistics and counts (this is particularly useful in the LMFDB's context since the data changes rarely) +psycodict runs on [psycopg 3](https://www.psycopg.org/psycopg3/). psycopg is an +optional dependency so that you can choose between the binary and pure-Python +builds; install psycodict with one of the two extras: -The query language is specified in [QueryLanguage.md](QueryLanguage.md) and the read API (`search`, `lucky`, `count`, `random`, …) in [Searching.md](Searching.md). -What the version number promises — the public API, database metadata compatibility, and the deprecation policy — is laid out in [Versioning.md](Versioning.md). +``` +pip install "psycodict[pgbinary]" # pulls in psycopg[binary]; no system libpq needed +pip install "psycodict[pgsource]" # pulls in psycopg; builds against your system libpq +``` + +## Quickstart + +psycodict reads its connection settings from a `config.ini` in the working +directory (one is created with default values on first run if none exists). +Point it at your server: -# Install +```ini +[postgresql] +host = localhost +port = 5432 +user = postgres +password = +dbname = mydb +``` + +Then create a table, insert some rows, and search — a query is a dict, and each +result is a dict: + +```python +from psycodict.database import PostgresDatabase + +# create=True bootstraps the meta tables the first time you connect to a fresh database +db = PostgresDatabase(create=True) + +db.create_table( + "demo_primes", + [("n", "integer"), ("label", "text"), ("factors", "jsonb"), ("is_prime", "boolean")], + label_col="label", + sort=["n"], +) + +db.demo_primes.insert_many([ + {"n": 6, "label": "6", "factors": {"2": 1, "3": 1}, "is_prime": False}, + {"n": 7, "label": "7", "factors": {"7": 1}, "is_prime": True}, + {"n": 10, "label": "10", "factors": {"2": 1, "5": 1}, "is_prime": False}, + {"n": 12, "label": "12", "factors": {"2": 2, "3": 1}, "is_prime": False}, +]) + +for row in db.demo_primes.search( + {"is_prime": False, "n": {"$lte": 10}}, + projection=["n", "factors"], +): + print(row) +``` ``` -pip3 install -U "psycodict[pgbinary] @ git+https://github.com/roed314/psycodict.git" +{'n': 6, 'factors': {'2': 1, '3': 1}} +{'n': 10, 'factors': {'2': 1, '5': 1}} ``` -or + +`search` returns every matching row; `lucky` returns the first match (or a +single column of it), and `count` counts them: + +```python +db.demo_primes.count({"is_prime": False}) # 3 +db.demo_primes.lucky({"n": 7}, projection="factors") # {'7': 1} ``` -pip3 install -U "psycodict[pgsource] @ git+https://github.com/roed314/psycodict.git" + +## Features + +- **Dictionary query language.** Equality and null tests, ranges (`$lte`, + `$gte`, `$lt`, `$gt`), membership and containment (`$in`, `$nin`, + `$contains`), array and jsonb path access (`"ainvs.2"`), disjunction with + `$or`, cardinality with `$size`, comparisons between columns with `$col`, and + a raw-SQL escape hatch (`$raw`). Multi-table **joins** via `join=` on + `search`, `count` and `lucky` (inner, left, right or full), with + `"table.column"` qualification usable in queries, projections, sorts, `$col` + and `$raw`. +- **Bulk data management.** Load and dump whole tables to and from files + (`copy_from` / `copy_to`), reload a table from a file and `revert` to the + previous contents if something is wrong, and group writes into `staged()` + transactional uploads with write exclusion and drift detection. +- **Cached statistics and counts.** Counts and statistics tables let search + pages report totals and distributions without re-scanning a dataset that + rarely changes. +- **Schema management.** Meta tables record every table, column, index and + constraint; their layout carries a versioned format stamp checked on connect + (an older-format database keeps working, with a warning, until migrated in + place with `PostgresDatabase(upgrade=True)`); partial indexes are supported, + and `db.refresh_tables()` picks up schema changes without a restart. +- **Introspection.** Inspect running and blocked queries (`show_queries`, + `show_blocked`) and analyze the slow-query log (`psycodict.slowlog`). +- **Change notifications.** LISTEN/NOTIFY-based schema-change notifications and + a small publish/subscribe layer. +- **Database diffing.** `db.compare` and `db.show_differences` detect drift + between two databases. + +## Documentation + +- [QueryLanguage.md](https://github.com/roed314/psycodict/blob/master/QueryLanguage.md) — how Python dictionaries become SQL `WHERE` clauses. +- [Searching.md](https://github.com/roed314/psycodict/blob/master/Searching.md) — the read-side API: `search`, `lucky`, `lookup`, `count`, projections, sorts and joins. +- [DataManagement.md](https://github.com/roed314/psycodict/blob/master/DataManagement.md) — the write side: creating tables, loading data, reloading and reverting, statistics. +- [MetadataFormats.md](https://github.com/roed314/psycodict/blob/master/MetadataFormats.md) — how the layout of the meta tables is versioned, cross-format compatibility, and the checklist for changing it. + +See the [CHANGELOG](https://github.com/roed314/psycodict/blob/master/CHANGELOG.md) for the release history. + +## Supported versions + +- Python 3.8 or newer. +- PostgreSQL 13 through 18. +- psycopg 3 (installed through the `pgbinary` or `pgsource` extra above). + +## Setting up PostgreSQL + +If you do not already have a server, install +[PostgreSQL](https://www.postgresql.org/) and create a +[user](https://www.postgresql.org/docs/current/sql-createuser.html) and a +[database](https://www.postgresql.org/docs/current/sql-createdatabase.html). +For example, in `psql`: + +```sql +CREATE USER myuser WITH PASSWORD 'a good password'; +CREATE DATABASE mydb OWNER myuser; ``` -# Getting started +Making `myuser` the database owner matters on PostgreSQL 15 and later: +`GRANT ALL PRIVILEGES ON DATABASE` no longer implies the right to create +objects in the `public` schema, where psycodict puts its (unqualified) tables. +If the user is not the owner, grant that privilege explicitly — connected to +`mydb` — instead: -You will first need to install [postgres](https://www.postgresql.org/) and create a [user](https://www.postgresql.org/docs/current/sql-createuser.html) and a [database](https://www.postgresql.org/docs/current/sql-createdatabase.html). For example, you might execute the following commands in psql: +```sql +GRANT USAGE, CREATE ON SCHEMA public TO myuser; +``` - CREATE DATABASE database_name; - CREATE USER username; - ALTER USER psetpartners WITH password 'good password'; - GRANT ALL PRIVILEGES ON DATABASE database_name TO username; +`PostgresDatabase(create=True)` bootstraps the meta tables on first connection, +so the database only needs to exist and let `myuser` create tables in it — you +do not have to create any schema by hand. -# Running the tests +## Running the tests Install the test dependencies and point the standard PostgreSQL environment variables at a database you do not mind being written to: ``` -pip3 install -e ".[pgbinary,test]" +pip install -e ".[pgbinary,test]" createdb psycodict_test PGDATABASE=psycodict_test pytest ``` The connection is configured through `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD` and `PGDATABASE`, defaulting to `postgres@localhost:5432` and a -database named `psycodict_test`. The database only needs to be empty: the -meta tables are bootstrapped on first connection, and every test creates its -own randomly named tables and drops them afterwards. +database named `psycodict_test`. The database only needs to be empty: the meta +tables are bootstrapped on first connection, and every test creates its own +randomly named tables and drops them afterwards. If no server is reachable the tests that need one are skipped, so -`pytest tests/test_encoding.py tests/test_utils.py tests/test_config.py` + +``` +pytest tests/test_encoding.py tests/test_utils.py tests/test_config.py +``` + works with no database at all. Two further sets of tests are opt-in: @@ -69,3 +192,10 @@ turns "no database, skip" into a hard failure — which is what continuous integration wants, so that a misconfigured job cannot report success by quietly skipping everything. +## Provenance + +psycodict was split out of the [LMFDB](https://www.lmfdb.org), where it grew as +the project's `db` interface. It is written and maintained by David Roe and +Edgar Costa, with contributions from the wider LMFDB community, and is +distributed under the GNU General Public License, version 2 or later (see +[LICENSE](LICENSE)). diff --git a/pyproject.toml b/pyproject.toml index 711c09f..147a3d8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,10 +16,25 @@ requires-python = ">=3.9" authors = [{name = "David Roe", email = "roed.math@gmail.com"}, {name = "Edgar Costa", email = "edgarc@mit.edu"}] license = "GPL-2.0-or-later" keywords = ["postgres", "database", "interface"] +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Intended Audience :: Science/Research", + "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Topic :: Database", + "Topic :: Scientific/Engineering :: Mathematics", +] [project.urls] -homepage = "https://github.com/roed314/psycodict" -repository = "https://github.com/roed314/psycodict" +Homepage = "https://github.com/roed314/psycodict" +Repository = "https://github.com/roed314/psycodict" +Changelog = "https://github.com/roed314/psycodict/blob/master/CHANGELOG.md" [project.optional-dependencies] pgsource = ["psycopg>=3.1"] @@ -43,7 +58,7 @@ filterwarnings = ["error::DeprecationWarning:psycodict.*"] target-version = "py39" [tool.ruff.lint.per-file-ignores] -# The psycopg2 availability check in __init__.py deliberately runs before the +# The psycopg availability check in __init__.py deliberately runs before the # imports that depend on it. "psycodict/__init__.py" = ["E402"] # Pre-existing findings, silenced so that adopting a linter does not smuggle @@ -53,4 +68,3 @@ target-version = "py39" # base.py:1024 F841 `ordered` assigned from a call kept for its side # effect (_order_columns mutates its argument) "psycodict/base.py" = ["E741", "F841"] - From d67e184502eb5b0a000c12273b17a6d1fa21570b Mon Sep 17 00:00:00 2001 From: David Roe Date: Wed, 22 Jul 2026 01:53:41 -0400 Subject: [PATCH 2/6] Document the configuration discovery from roed314/psycodict#119 README quickstart: config.ini is now discovered ($PSYCODICT_CONFIG, then an existing ./config.ini, then ~/.psycodict/config.ini) rather than created in the working directory. CHANGELOG: breaking-changes entry for the readargs flip, the discovery and the log/secrets relocation, plus the postgresql_dbname defaults fix. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 16 ++++++++++++++++ README.md | 7 ++++--- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33e51d3..f951b06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,19 @@ migration note. created or migrated — it predates `meta_format`, so it cannot be told to refuse, and its index/reload paths would silently drop partial-index predicates. This only matters during a mixed-version rollout. (#90, #110) +- **`Configuration` behaves like a library.** It no longer parses the host + program's command line by default (`readargs` was auto-enabled in scripts + and an unrecognized option raised `SystemExit`); with no explicit location + the configuration file is discovered — `$PSYCODICT_CONFIG`, then + `./config.ini` if it already exists, then `~/.psycodict/config.ini` — and a + missing file is created under `~/.psycodict`, never in the working + directory; the default `slow_queries.log` lands in a `logs` directory next + to the configuration file; and the default secrets file sits next to the + configuration file. Existing `./config.ini` setups keep working unchanged. + *Migration:* pass `readargs=True` in a script whose command line psycodict + should parse; set `PSYCODICT_CONFIG` (or pass `config_file`) to pin a + location. Subclasses supplying their own parser (LMFDB, seminars) are + unaffected. (#119) ### Added @@ -103,6 +116,9 @@ migration note. Roughly two dozen fixes landed while splitting the search/extras tables and hardening standalone use; the highlights: +- `postgresql_dbname` was the one option whose default ignored the `defaults` + dictionary passed to `Configuration`. (#119) + - Counts-table totals are now maintained correctly on writes rather than drifting out of sync. (#61–#87) - `text[]` values no longer corrupt on `COPY` because of quoting. (#61–#87) diff --git a/README.md b/README.md index 325c9f4..53922b8 100644 --- a/README.md +++ b/README.md @@ -23,9 +23,10 @@ pip install "psycodict[pgsource]" # pulls in psycopg; builds against your sys ## Quickstart -psycodict reads its connection settings from a `config.ini` in the working -directory (one is created with default values on first run if none exists). -Point it at your server: +psycodict reads its connection settings from a `config.ini`: it uses +`$PSYCODICT_CONFIG` if set, then `config.ini` in the working directory if one +exists, and otherwise `~/.psycodict/config.ini` (created with default values +on first run). Point it at your server: ```ini [postgresql] From 19180704e085553320e094fc8aa89651fdf0f7da Mon Sep 17 00:00:00 2001 From: David Roe Date: Wed, 22 Jul 2026 02:37:53 -0400 Subject: [PATCH 3/6] CHANGELOG: note the ~/.psycodict/logs fallback from psycodict#119 review Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f951b06..ad12dab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -69,7 +69,8 @@ migration note. `./config.ini` if it already exists, then `~/.psycodict/config.ini` — and a missing file is created under `~/.psycodict`, never in the working directory; the default `slow_queries.log` lands in a `logs` directory next - to the configuration file; and the default secrets file sits next to the + to the configuration file (falling back to `~/.psycodict/logs` when that + location is not writable); and the default secrets file sits next to the configuration file. Existing `./config.ini` setups keep working unchanged. *Migration:* pass `readargs=True` in a script whose command line psycodict should parse; set `PSYCODICT_CONFIG` (or pass `config_file`) to pin a From 8ee02a5319d9db7db4bc64882ec4360727e9f902 Mon Sep 17 00:00:00 2001 From: David Roe Date: Wed, 22 Jul 2026 03:25:11 -0400 Subject: [PATCH 4/6] Reconcile the release capstone with the merged 1.0 wave Rebased onto main, which absorbed everything this PR used to anticipate; the reconciliation: - Version: 1.0.0 is now set in psycodict.__init__.__version__, the single source of truth #121 established -- pyproject keeps main's dynamic version, SPDX license string and >=3.9 floor, and this PR's classifiers drop the License:: entry (setuptools >= 77 rejects it next to an SPDX string) and the 3.8 entry. - README: the rewrite now carries the Versioning.md pointer (#123) in its Documentation list, which switches to relative links -- on the Sphinx site (#122) those resolve to the copied guide pages, and they still render on GitHub; CHANGELOG and LICENSE stay absolute since neither is on the site. Supported Python is 3.9+; blob/master URLs become blob/main. - CHANGELOG: entries for the late wave -- sum/random honoring saving (#118) and the 3.9 floor (#121) under breaking changes; the docs site (#122), Versioning.md/CONTRIBUTING/SECURITY and __version__ (#121) under added; reload metafile+resort (#114), update_from_file stats publication (#115), copy_dumps delimiter escaping (#116) under fixed; config.ini untracking (#120), this release workflow and CITATION.cff (#124) under infrastructure. - release.yml: the flow notes now say bump __init__.__version__ and merge to main. - Fixes the docs build on main, broken by the #119 x #122 crossing (#119 merged after #122's last CI run): the Configuration docstring's nested defaults list needed blank lines to be valid RST under autodoc, and the README's relative LICENSE link had no target on the site. Suite 857 passed / 36 skipped; ruff clean; sphinx -W clean; python -m build + twine check --strict pass with Version 1.0.0, License-Expression GPL-2.0-or-later, Requires-Python >=3.9 and no License classifier in the wheel metadata. Co-Authored-By: Claude Fable 5 --- .github/workflows/release.yml | 5 +++-- CHANGELOG.md | 36 +++++++++++++++++++++++++++++++---- README.md | 15 ++++++++------- psycodict/__init__.py | 2 +- psycodict/config.py | 2 ++ pyproject.toml | 4 +--- 6 files changed, 47 insertions(+), 17 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ac08cb0..bfd664a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -23,8 +23,9 @@ # # RELEASE FLOW: # -# Bump `version` in pyproject.toml and update CHANGELOG.md, merge to master, -# then create a GitHub Release with the tag `v1.0.0` (matching the version). +# Bump `__version__` in psycodict/__init__.py and update CHANGELOG.md, merge +# to main, then create a GitHub Release with the tag `v1.0.0` (matching the +# version). # Publishing that release triggers this workflow: the sdist and wheel are # built, metadata-checked, and uploaded to PyPI. Nothing is published on # pushes, pull requests, or draft releases. diff --git a/CHANGELOG.md b/CHANGELOG.md index ad12dab..482e89f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,6 +76,13 @@ migration note. should parse; set `PSYCODICT_CONFIG` (or pass `config_file`) to pin a location. Subclasses supplying their own parser (LMFDB, seminars) are unaffected. (#119) +- **`sum` and `random` honor the `saving` flag.** With the default + `saving = False` they no longer write to the counts/stats cache tables (they + were the only two paths that did). *Migration:* enable `saving` on the stats + table — as data-management deployments already do — to persist computed + statistics. (#118) +- **Python 3.9 or newer is required** (3.8 is past end of life). *Migration:* + upgrade the interpreter; no code changes. (#121) ### Added @@ -110,16 +117,21 @@ migration note. - **Schema-change notifications** and a small publish/subscribe layer built on PostgreSQL LISTEN/NOTIFY. (#111) - **Documentation.** `DataManagement.md` (write side) and `Searching.md` (read - API), plus docstring housekeeping across the package. (#104, #105, #106) + API), plus docstring housekeeping across the package (#104, #105, #106); + `Versioning.md` states what the version number promises (#123), and + `CONTRIBUTING.md` and `SECURITY.md` set out the contribution workflow and + the security policy. +- **A documentation site.** Sphinx/MyST build of the guides plus an autodoc + API reference, built warning-free in CI and publishable on Read the Docs; + the canonical copies of the guides stay at the repository root. (#122) +- **`psycodict.__version__`** — the package version as an attribute, and the + single source of truth for packaging. (#121) ### Fixed Roughly two dozen fixes landed while splitting the search/extras tables and hardening standalone use; the highlights: -- `postgresql_dbname` was the one option whose default ignored the `defaults` - dictionary passed to `Configuration`. (#119) - - Counts-table totals are now maintained correctly on writes rather than drifting out of sync. (#61–#87) - `text[]` values no longer corrupt on `COPY` because of quoting. (#61–#87) @@ -136,12 +148,28 @@ hardening standalone use; the highlights: reload or rewrite. (#95) - `create_table_like` preserves per-column `STORAGE` / `COMPRESSION` settings and analyzes the copy. (#98) +- `reload` with a metafile and resorting no longer fails with a `TypeError`. + (#114) +- Non-inplace `update_from_file(restat=True)` publishes the refreshed + statistics it computes (they used to be left orphaned in `_tmp` tables) and + rebuilds the counts indexes on the table it swaps in. (#115) +- `copy_dumps` escapes the COPY delimiter inside text, json and array values, + so its output round-trips through `COPY FROM` and matches `COPY TO` byte for + byte. (#116) +- `postgresql_dbname` was the one option whose default ignored the `defaults` + dictionary passed to `Configuration`. (#119) ### Infrastructure - A test suite of 500+ tests and a continuous-integration workflow covering the supported Python and PostgreSQL versions, plus downstream regression jobs that run LMFDB and seminars against the new code. (#58) +- `config.ini` is no longer tracked in the repository; copy + `config.ini.example` (or rely on the configuration discovery above). (#120) +- A trusted-publishing release workflow: publishing a GitHub release builds + the distributions, checks their metadata and uploads to PyPI via OpenID + Connect — no long-lived token is stored anywhere. (#113) +- `CITATION.cff`, so GitHub renders a citation for the package. (#124) ## 0.1.x and earlier diff --git a/README.md b/README.md index 53922b8..5534320 100644 --- a/README.md +++ b/README.md @@ -111,16 +111,17 @@ db.demo_primes.lucky({"n": 7}, projection="factors") # {'7': 1} ## Documentation -- [QueryLanguage.md](https://github.com/roed314/psycodict/blob/master/QueryLanguage.md) — how Python dictionaries become SQL `WHERE` clauses. -- [Searching.md](https://github.com/roed314/psycodict/blob/master/Searching.md) — the read-side API: `search`, `lucky`, `lookup`, `count`, projections, sorts and joins. -- [DataManagement.md](https://github.com/roed314/psycodict/blob/master/DataManagement.md) — the write side: creating tables, loading data, reloading and reverting, statistics. -- [MetadataFormats.md](https://github.com/roed314/psycodict/blob/master/MetadataFormats.md) — how the layout of the meta tables is versioned, cross-format compatibility, and the checklist for changing it. +- [QueryLanguage.md](QueryLanguage.md) — how Python dictionaries become SQL `WHERE` clauses. +- [Searching.md](Searching.md) — the read-side API: `search`, `lucky`, `lookup`, `count`, projections, sorts and joins. +- [DataManagement.md](DataManagement.md) — the write side: creating tables, loading data, reloading and reverting, statistics. +- [MetadataFormats.md](MetadataFormats.md) — how the layout of the meta tables is versioned, cross-format compatibility, and the checklist for changing it. +- [Versioning.md](Versioning.md) — what the version number promises: the public API, metadata compatibility, and the deprecation policy. -See the [CHANGELOG](https://github.com/roed314/psycodict/blob/master/CHANGELOG.md) for the release history. +See the [CHANGELOG](https://github.com/roed314/psycodict/blob/main/CHANGELOG.md) for the release history. ## Supported versions -- Python 3.8 or newer. +- Python 3.9 or newer. - PostgreSQL 13 through 18. - psycopg 3 (installed through the `pgbinary` or `pgsource` extra above). @@ -199,4 +200,4 @@ psycodict was split out of the [LMFDB](https://www.lmfdb.org), where it grew as the project's `db` interface. It is written and maintained by David Roe and Edgar Costa, with contributions from the wider LMFDB community, and is distributed under the GNU General Public License, version 2 or later (see -[LICENSE](LICENSE)). +[LICENSE](https://github.com/roed314/psycodict/blob/main/LICENSE)). diff --git a/psycodict/__init__.py b/psycodict/__init__.py index cf7d297..87d3f52 100644 --- a/psycodict/__init__.py +++ b/psycodict/__init__.py @@ -27,7 +27,7 @@ # Single source of truth for the package version: pyproject.toml reads it via # ``[tool.setuptools.dynamic]``, and it works from an uninstalled checkout too. -__version__ = "0.1.13" +__version__ = "1.0.0" try: import psycopg diff --git a/psycodict/config.py b/psycodict/config.py index 7cda7f2..f13704a 100644 --- a/psycodict/config.py +++ b/psycodict/config.py @@ -76,6 +76,7 @@ class Configuration(): - ``parser`` -- an argparse.ArgumentParser instance. If not provided, a default will be created. - ``defaults`` -- a dictionary with default values for the created argument parser. Only used if a parser is not specified. The keys used are: + - ``config_file`` -- the filename for the configuration file. If not given, it is discovered: the ``PSYCODICT_CONFIG`` environment variable, then ``config.ini`` in the current directory if it exists, @@ -95,6 +96,7 @@ class Configuration(): - ``postgresql_user`` -- the username when connecting to the database - ``postgresql_password`` -- the password for connecting to the database - ``postgresql_dbname`` -- the name of the database to connect to + - ``writeargstofile`` -- a boolean, if config file doesn't exist, it determines if command line arguments are written to the config file instead of the default arguments - ``readargs`` -- a boolean (default False), determining whether command line arguments are read. Leave this off in libraries and applications diff --git a/pyproject.toml b/pyproject.toml index 147a3d8..ef899ec 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,9 +19,7 @@ keywords = ["postgres", "database", "interface"] classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Science/Research", - "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", @@ -34,7 +32,7 @@ classifiers = [ [project.urls] Homepage = "https://github.com/roed314/psycodict" Repository = "https://github.com/roed314/psycodict" -Changelog = "https://github.com/roed314/psycodict/blob/master/CHANGELOG.md" +Changelog = "https://github.com/roed314/psycodict/blob/main/CHANGELOG.md" [project.optional-dependencies] pgsource = ["psycopg>=3.1"] From ebf309ed29a0822a8ff7b1ea72262cbdef1f68ea Mon Sep 17 00:00:00 2001 From: David Roe Date: Wed, 22 Jul 2026 03:31:23 -0400 Subject: [PATCH 5/6] Start at 1.0.0rc1 and refuse a release tag that mismatches the version The uploaded version is whatever __version__ says at the tagged commit -- the tag name controls nothing -- and PyPI never accepts a version twice, so a rehearsal tag against final-version source would have burned 1.0.0 on the rehearsal. The build job now refuses any tag that is not v<__version__>. With that guard in place, ship the first release as a candidate: __version__ starts at 1.0.0rc1, so the v1.0.0rc1 rehearsal exercises the entire trusted-publishing path (including the pending publisher creating the project) on a version pip ignores by default; bumping to 1.0.0 and tagging v1.0.0 is then a one-line follow-up. The release-flow notes describe the sequence. Co-Authored-By: Claude Fable 5 --- .github/workflows/release.yml | 35 ++++++++++++++++++++++++++++++----- psycodict/__init__.py | 2 +- 2 files changed, 31 insertions(+), 6 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index bfd664a..08a4db8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -24,11 +24,20 @@ # RELEASE FLOW: # # Bump `__version__` in psycodict/__init__.py and update CHANGELOG.md, merge -# to main, then create a GitHub Release with the tag `v1.0.0` (matching the -# version). -# Publishing that release triggers this workflow: the sdist and wheel are -# built, metadata-checked, and uploaded to PyPI. Nothing is published on -# pushes, pull requests, or draft releases. +# to main, then create a GitHub Release whose tag is `v` -- the +# build refuses a tag that does not match `__version__`, so a rehearsal tag +# can never upload the real version by accident. Publishing that release +# triggers this workflow: the sdist and wheel are built, metadata-checked, +# and uploaded to PyPI. Nothing is published on pushes, pull requests, or +# draft releases. +# +# First release: rehearse with a release candidate. With `__version__` at +# "1.0.0rc1", publish a GitHub Release tagged `v1.0.0rc1` (marked as a +# pre-release); pip ignores release candidates unless asked (`pip install +# --pre psycodict`), so this exercises the whole trusted-publishing path -- +# including the pending publisher creating the project -- without touching +# the version users will install. Then bump `__version__` to "1.0.0" and +# publish `v1.0.0`. name: Release @@ -46,6 +55,22 @@ jobs: steps: - uses: actions/checkout@v7 + # The uploaded version is whatever __version__ says at the tagged + # commit, so require the tag to agree -- otherwise a rehearsal tag + # against a final version (or vice versa) would silently publish the + # wrong thing, and PyPI never accepts a version twice. + - name: Check that the tag matches the package version + run: | + version=$(sed -n 's/^__version__ = "\(.*\)"$/\1/p' psycodict/__init__.py) + if [ -z "$version" ]; then + echo "Could not read __version__ from psycodict/__init__.py" >&2 + exit 1 + fi + if [ "$GITHUB_REF_NAME" != "v$version" ]; then + echo "Tag $GITHUB_REF_NAME does not match psycodict.__version__ = $version (expected tag v$version)" >&2 + exit 1 + fi + - uses: actions/setup-python@v6 with: python-version: "3.12" diff --git a/psycodict/__init__.py b/psycodict/__init__.py index 87d3f52..7b0ee0a 100644 --- a/psycodict/__init__.py +++ b/psycodict/__init__.py @@ -27,7 +27,7 @@ # Single source of truth for the package version: pyproject.toml reads it via # ``[tool.setuptools.dynamic]``, and it works from an uninstalled checkout too. -__version__ = "1.0.0" +__version__ = "1.0.0rc1" try: import psycopg From 6c5e30ed5cdeaa38ddbbe17ef636e418febed984 Mon Sep 17 00:00:00 2001 From: David Roe Date: Wed, 22 Jul 2026 03:49:00 -0400 Subject: [PATCH 6/6] Fix the README for its PyPI audience Four review findings: - The Documentation list's relative guide links render as relative hrefs on PyPI, resolving against pypi.org and 404ing. They now point at the live Read the Docs pages (verified 200), with a pointer to the Markdown originals at the repository root and the generated API reference; the pyproject urls gain a Documentation entry. - The features tour advertised `revert`, which does not exist; the rollback method is `reload_revert`. - The quickstart's example config connected as the postgres superuser with a blank password while the setup section below creates myuser with a password as the owner of mydb; someone following both sections in order would fail to authenticate. The quickstart now uses the setup section's credentials and says so. - The classifiers stopped at Python 3.13 while CI tests through 3.14. Co-Authored-By: Claude Fable 5 --- README.md | 27 ++++++++++++++++----------- pyproject.toml | 2 ++ 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 5534320..9953f58 100644 --- a/README.md +++ b/README.md @@ -26,14 +26,15 @@ pip install "psycodict[pgsource]" # pulls in psycopg; builds against your sys psycodict reads its connection settings from a `config.ini`: it uses `$PSYCODICT_CONFIG` if set, then `config.ini` in the working directory if one exists, and otherwise `~/.psycodict/config.ini` (created with default values -on first run). Point it at your server: +on first run). Point it at your server — here the user and database from +[Setting up PostgreSQL](#setting-up-postgresql) below: ```ini [postgresql] host = localhost port = 5432 -user = postgres -password = +user = myuser +password = a good password dbname = mydb ``` @@ -91,9 +92,9 @@ db.demo_primes.lucky({"n": 7}, projection="factors") # {'7': 1} `"table.column"` qualification usable in queries, projections, sorts, `$col` and `$raw`. - **Bulk data management.** Load and dump whole tables to and from files - (`copy_from` / `copy_to`), reload a table from a file and `revert` to the - previous contents if something is wrong, and group writes into `staged()` - transactional uploads with write exclusion and drift detection. + (`copy_from` / `copy_to`), reload a table from a file and `reload_revert` + back to the previous contents if something is wrong, and group writes into + `staged()` transactional uploads with write exclusion and drift detection. - **Cached statistics and counts.** Counts and statistics tables let search pages report totals and distributions without re-scanning a dataset that rarely changes. @@ -111,11 +112,15 @@ db.demo_primes.lucky({"n": 7}, projection="factors") # {'7': 1} ## Documentation -- [QueryLanguage.md](QueryLanguage.md) — how Python dictionaries become SQL `WHERE` clauses. -- [Searching.md](Searching.md) — the read-side API: `search`, `lucky`, `lookup`, `count`, projections, sorts and joins. -- [DataManagement.md](DataManagement.md) — the write side: creating tables, loading data, reloading and reverting, statistics. -- [MetadataFormats.md](MetadataFormats.md) — how the layout of the meta tables is versioned, cross-format compatibility, and the checklist for changing it. -- [Versioning.md](Versioning.md) — what the version number promises: the public API, metadata compatibility, and the deprecation policy. +- [QueryLanguage](https://psycodict.readthedocs.io/en/latest/QueryLanguage.html) — how Python dictionaries become SQL `WHERE` clauses. +- [Searching](https://psycodict.readthedocs.io/en/latest/Searching.html) — the read-side API: `search`, `lucky`, `lookup`, `count`, projections, sorts and joins. +- [DataManagement](https://psycodict.readthedocs.io/en/latest/DataManagement.html) — the write side: creating tables, loading data, reloading and reverting, statistics. +- [MetadataFormats](https://psycodict.readthedocs.io/en/latest/MetadataFormats.html) — how the layout of the meta tables is versioned, cross-format compatibility, and the checklist for changing it. +- [Versioning](https://psycodict.readthedocs.io/en/latest/Versioning.html) — what the version number promises: the public API, metadata compatibility, and the deprecation policy. + +The same documents live at the repository root as Markdown, next to an +[API reference](https://psycodict.readthedocs.io/en/latest/api/index.html) +generated from the docstrings. See the [CHANGELOG](https://github.com/roed314/psycodict/blob/main/CHANGELOG.md) for the release history. diff --git a/pyproject.toml b/pyproject.toml index ef899ec..4f43569 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,12 +25,14 @@ classifiers = [ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Topic :: Database", "Topic :: Scientific/Engineering :: Mathematics", ] [project.urls] Homepage = "https://github.com/roed314/psycodict" +Documentation = "https://psycodict.readthedocs.io/" Repository = "https://github.com/roed314/psycodict" Changelog = "https://github.com/roed314/psycodict/blob/main/CHANGELOG.md"