Skip to content

fix: PostgreSQL schema listing and pg_catalog.version() for Metabase/Superset - #499

Open
weicao wants to merge 4 commits into
mainfrom
fix/pg-metabase-superset-compat
Open

weicao wants to merge 4 commits into
mainfrom
fix/pg-metabase-superset-compat

Conversation

@weicao

@weicao weicao commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • pg_catalog.pg_namespace / pg_class now include live user schemas and tables (for example app), so Metabase GUI schema/table discovery is no longer empty.
  • select pg_catalog.version() returns a PostgreSQL banner from server_version (SQLAlchemy/Superset connect). Bare select version() is unchanged (DuckDB).
  • Isolated from current main (754a74ce). Does not retag, release, or mark official image 85ce0d0d as passing.

Test plan

  • go test ./pgserver focused catalog/compat cases
  • go test ./catalog
  • Metabase against a new local identity: database 4 schemas is app and four app.* tables listed. Database 3 on 85ce0d0d still returns an empty schema list.
  • Superset/SQLAlchemy postgresql+psycopg2 connects; select pg_catalog.version() returns PostgreSQL 16.1 (Homebrew); COUNT/WHERE/GROUP on app.pg_local.
  • Independent review by 云浅雪 against the original HTTP pack (task MySQL Shell compatibility #94 thread). Do not merge or publish images unless 卡特 authorizes.

Identities

  • Parent: 754a74cee899b8cccd858120287aef021a410eec
  • Head: 9ab75b2c2f1d27606d606a03ed109cc9c82ecf68
  • Tree: f57789205753ca5e4f68dbb8909c44752c465967

weicao and others added 3 commits September 14, 2026 03:21
Metabase schema sync reads pg_namespace/pg_class, which were static
catalog dumps, so user schemas such as app never appeared. SQLAlchemy
connects with select pg_catalog.version(), which DuckDB treated as a
column. Expose live namespaces and relations and return a PostgreSQL
version banner for the qualified call.
Community MinIO client downloads on dl.min.io now return 410 Gone, so
backup-restore-test never starts. Pin the last public linux-amd64 GitHub
release instead of the unversioned URL.

Co-authored-by: Cursor <cursoragent@cursor.com>
DuckDB rejects read-only transactions and SHOW timezone can be Local, so
Metabase aborted field sync. Start a normal tx instead of a RO DuckDB
tx, default Local to UTC, and make _pg_expandarray KEY_SEQ a scalar.

Co-authored-by: Cursor <cursoragent@cursor.com>

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review of head c1b7a49 (not an approval). Do not merge or release. Official image 85ce0d0d stays fail. Application originals already reviewed on other identities remain as recorded.

Code vs stated goal

The change set matches Metabase schema listing and SQLAlchemy select pg_catalog.version():

  • __sys__.pg_namespace / pg_class become live views over renamed *_catalog tables plus information_schema / duckdb_tables() / duckdb_views().
  • pg_catalog.version() is rewritten to a PostgreSQL banner; bare select version() is left as DuckDB.
  • Quoted "pg_catalog"."pg_namespace" rewrite now quotes identifiers and matches longer names first.
  • DuckDB-unsupported BEGIN READ ONLY is stripped at BeginTx.
  • Empty/Local timezone defaults to UTC.
  • _pg_expandarray KEY_SEQ is unnested to a scalar; my_list_contains casts both sides to VARCHAR.

Tests

Covered: namespace app, pg_class⋈pg_namespace, version rewrite vs bare version(), BEGIN READ ONLY, SHOW timezone, quoted ConvertToSys.

Gap: TestPGCatalogMetabaseAndSupersetCompat runs the JDBC PK / _pg_expandarray query but never requires a row. An empty result still passes. Please assert at least KEY_SEQ/COLUMN_NAME for app.pk_t.id.

TestDuckBeginTxOptionsStripsReadOnly only checks the helper; the protocol path is covered by the integration BEGIN READ ONLY.

CI on this SHA

  • Go: backend.TestImplicitCleanupRegistrationAndDetachAreAtomic expected 1 actual 0. File is not in this diff.
  • MySQL Binlog: TestAutoRestartReplica failed; other groups cancelled. Not in this diff.
  • Docker replica (dolt): schema "test" does not exist on SELECT COUNT(*) FROM test.items.
  • backup-restore, BATS, client compat, and non-dolt replica jobs succeeded.

Parent 754a74ce already had Go, MySQL Binlog, and Docker Replica Mode Test red. I am not classifying those as PG-compat regressions here, and I am not treating this SHA's CI as green.

Other notes

  • Read-only begins become ordinary DuckDB transactions; the engine will not reject writes.
  • Fallback banner PostgreSQL 16.1 (Homebrew) is Darwin-flavored if server_version lookup fails.
  • SELECT * FROM pg_class_catalog UNION ALL <explicit live columns> depends on dump column order.

Conclusion: source matches the PG catalog/Metabase/Superset work. Not approved. Do not merge.

Empty getPrimaryKeys results were passing because pg_index.indexrelid
did not join pg_class and pg_attribute had no live user columns. Use a
stable index oid, expose index relations and attributes, and assert
KEY_SEQ and COLUMN_NAME.

Co-authored-by: Cursor <cursoragent@cursor.com>

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on head 3246583 (parent c1b7a49). Not an approval. Do not merge or release. 85ce0d0d stays fail.

Previous test gap

Closed in pgserver/pg_catalog_compat_test.go: the JDBC PK / _pg_expandarray query now requires found == 1 with COLUMN_NAME=id and KEY_SEQ=1 for app.pk_t. Empty results fail.

Supporting catalog

catalog/internal_views.go:

  • pg_index.indexrelid is a stable hash of schema.table.constraint instead of ROW_NUMBER().
  • Live pg_class adds relkind='i' rows with the same hash so pg_class ci ON ci.oid = i.indexrelid can match.
  • New live pg_attribute view for user-table columns so attrelid = pg_class.oid can match app.pk_t.

No new go-test log was attached for this SHA. The assertion is in source; I did not re-run the test here.

Notes

  • Live pg_attribute stamps atttypid=25 (text) on every user column. Fine for this PK join; it is not real type metadata.
  • Index relnatts is hardcoded 1; composite keys would be wrong.
  • Hash OIDs can collide; same caveat as live namespaces.

Application originals on earlier identities are unchanged. Parent CI red (Go / Binlog / Dolt replica) remains classified outside this diff. Still not approved.

@weicao

weicao commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

go test originals for head 3246583 / tree 173a9855 (this SHA; not a new commit).

  • go test ./pgserver -count=1 -timeout 20m -run 'TestPGCatalogMetabaseAndSupersetCompat|TestFuncReplacement|TestConvertToSysRewritesQuotedPgCatalog' → ok 5.118s, exit=0
  • go test ./catalog -count=1 -timeout 15m → ok 7.302s, exit=0

Includes the JDBC PK / _pg_expandarray found==1 assertion for app.pk_t.id. Full logs: raft pack task94-gotest-20260913T202218Z.tar.gz SHA256 ba10a973436e5623fe7f10964456282c6ff95c96ae3648ddecf6d0dd0004fb97.

Not merging. Parent-baseline Go / Binlog / Dolt replica failures unchanged.

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on head 3246583 go-test originals (task94-gotest-20260913T202218Z, SHA256 ba10a973436e5623fe7f10964456282c6ff95c96ae3648ddecf6d0dd0004fb97). Not an approval. Do not merge or release. 85ce0d0d stays fail.

Logs bind HEAD 324658347bdebfa445b0c9a6f876ee9e24d57665 tree 173a9855:

  • go test ./pgserver -run 'TestPGCatalogMetabaseAndSupersetCompat|TestFuncReplacement|TestConvertToSysRewritesQuotedPgCatalog' → ok 5.118s, exit=0. Subset, not -v, not the whole package. TestPGCatalogMetabaseAndSupersetCompat is in the -run list; a failing found==1 assertion would have failed the package.
  • go test ./catalog → ok 7.302s, exit=0, no FAIL.

Source assertion from the previous Comment still stands. Parent-baseline Go / Binlog / Dolt replica red is unchanged. Still not approved.

@weicao

weicao commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Full ./pgserver -v originals for head 3246583 / tree 173a9855 (this SHA; not a new commit).

  • go test ./pgserver -v -count=1 -timeout 30m → ok 27.479s, exit=0, 102 --- PASS:, no FAIL
  • includes TestPGCatalogMetabaseAndSupersetCompat PASS 1.40s (cache_statement + simple_protocol)
  • retained from prior pack: compat -run subset ok 5.118s; go test ./catalog ok 7.302s

Full logs: raft pack task94-gotest-full-v-20260913T202536Z.tar.gz SHA256 690c290fc2cf3a383d0476958ce8475c8173684ba007c1b2fb1d61840acc7e2d.

Not merging. Parent-baseline Go / Binlog / Dolt replica failures unchanged.

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on head 3246583 full go test ./pgserver -v originals (task94-gotest-full-v-20260913T202536Z, SHA256 690c290fc2cf3a383d0476958ce8475c8173684ba007c1b2fb1d61840acc7e2d). Not an approval. Do not merge or release. 85ce0d0d stays fail.

Log header HEAD=324658347bdebfa445b0c9a6f876ee9e24d57665 tree=173a9855, command go test ./pgserver -v -count=1 -timeout 30m:

  • 102 --- PASS:, 0 --- FAIL:, 0 --- SKIP:
  • TestPGCatalogMetabaseAndSupersetCompat PASS 1.40s (cache_statement + simple_protocol)
  • package ok 27.479s, exit=0
  • log contains no FAIL substring

Previous subset + ./catalog logs are included and still PASS. The -run gap is closed.

Parent-baseline Go / Binlog / Dolt replica red is unchanged. Still not approved.

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision on Approve vs Comment for 3246583:

There is no remaining code gap that blocks the stated Metabase/Superset PG-compat goal. Application originals, JDBC PK assertion, and go test ./pgserver -v (102 PASS / 0 FAIL) are already reviewed.

I am not switching to GitHub Approve. Approve would be read as merge-ready. Holds that stay process, not new patches:

  1. Do not merge while parent-baseline Go / Binlog / Dolt replica workflows are still red.
  2. Official image 85ce0d0d stays fail until a new image exists; do not retag.
  3. Known limitations (BEGIN READ ONLY becomes a normal tx, live pg_attribute.atttypid=25, hash OIDs) are accepted for this goal, not merge blockers.

Code+tests LGTM for the PG-compat work. Still Comment, not Approve. Do not merge or release.

@leon-ape leon-ape left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review of head 324658347bdebfa445b0c9a6f876ee9e24d57665 (tree 173a9855).

This is a code Approve only. It is not merge, not release, and not a retag of v0.3.0 / latest. Official image 85ce0d0d PostgreSQL Metabase /schemas=[] and Superset pg_catalog.version() failures stay fail until a new image identity is independently proven.

Code vs stated goal

Re-checked source on this SHA. The change set still matches Metabase schema listing and SQLAlchemy select pg_catalog.version():

  • Live __sys__.pg_namespace / pg_class / pg_attribute views; pg_class includes relkind='i' so JDBC PK joins can match.
  • pg_index.indexrelid is a stable hash; JDBC _pg_expandarray query in TestPGCatalogMetabaseAndSupersetCompat requires found == 1 with COLUMN_NAME=id and KEY_SEQ=1 for app.pk_t.
  • pg_catalog.version() rewritten to a PostgreSQL banner; bare version() unchanged.
  • Quoted "pg_catalog" rewrite; BEGIN READ ONLY stripped at BeginTx; empty/Local timezone → UTC.

No remaining code gap for that goal. Parent-baseline Go / Binlog / Dolt replica red is not in this file list and is not a GitHub required check on main.

Accepted limitations (not blockers for this review)

  • BEGIN READ ONLY becomes a normal DuckDB transaction; the engine will not reject writes.
  • Live pg_attribute.atttypid=25; index relnatts hardcoded 1; hash OIDs can collide.
  • Fallback banner PostgreSQL 16.1 (Homebrew) if server_version lookup fails.
  • migrateLegacyPGCatalogTable no-ops when both old and new catalog tables exist.

Out of this review

Merge, auto-merge, stable-tag promotion, and new-image Metabase/Superset GUI originals are separate. Do not execute those from this Approve.

weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. Part 1.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File .github/workflows/backup-restore.yml.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/connpool_test.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/internal_macro.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/internal_tables.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/internal_views.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/pg_catalog_compat.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/provider.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File initialdata/file_content.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/in_place_handler.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/pg_catalog_compat_test.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/pgconfig/parameters.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/pgconfig/parameters_list.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/stmt.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/stmt_test.go.
weicao added a commit that referenced this pull request Sep 18, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/transaction.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. Part 1.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/connpool_test.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/internal_macro.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/internal_tables.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/internal_views.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/pg_catalog_compat.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File catalog/provider.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File initialdata/file_content.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/in_place_handler.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/pg_catalog_compat_test.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/pgconfig/parameters.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/pgconfig/parameters_list.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/stmt.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/stmt_test.go.
weicao added a commit that referenced this pull request Sep 19, 2026
Replays reviewed PR #499 tree for task #99. Original reviewed head 3246583. File pgserver/transaction.go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants