Skip to content

Commit 822625b

Browse files
waleedlatif1waleedlatif1
committed
chore(knowledge): register the directory-sync cron, and collapse the two migrations into one
The directory-sync scheduler is now in the Helm cron map beside member-sync, and in the self-hosting background-jobs table. The Helm templates iterate the map, so the values entry is the whole change. The unique address index and the directory-group tables ship as one migration. The tables and the duplicate pre-check run inside the runner's batch transaction; the embedded COMMIT then lets the index on the hot `user` table build concurrently. A failure after that COMMIT replays the whole file against tables that are already committed, so every earlier statement is idempotent. Replaying it on a real database exposed one defect on the way: drizzle's derived foreign-key name for the membership table is 71 characters, Postgres silently truncates identifiers at 63, and the replay guard looked the full name up in `pg_constraint` and never found it. Both foreign keys now carry explicit short names in the schema. The pre-COMMIT section replays cleanly twice in a rolled-back transaction with both guards resolving.
1 parent 7ecdc94 commit 822625b

9 files changed

Lines changed: 293 additions & 21933 deletions

File tree

‎apps/docs/content/docs/platform/self-hosting/background-jobs.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Point cron at an **internal** address where possible (the in-cluster Service, or
4949
| Workspace file search dispatch | `/api/cron/workspace-file-search-dispatch` | `*/1 * * * *` | Dispatches indexing work for workspace file search |
5050
| Connector sync | `/api/knowledge/connectors/sync` | `*/5 * * * *` | Knowledge base connector syncs |
5151
| Connector member sync | `/api/knowledge/connectors/member-sync` | `*/5 * * * *` | Per-member access sync for permission-aware connectors |
52+
| Connector directory sync | `/api/knowledge/connectors/directory-sync` | `*/5 * * * *` | Refreshes the directory groups administrator-mode connectors mirror, so a membership change takes effect without waiting for a content sync |
5253
| Workspace events poll | `/api/workspace-events/poll` | `*/15 * * * *` | Workspace event triggers |
5354
| Table row TTL cleanup | `/api/cron/cleanup-table-row-ttl` | `*/15 * * * *` | Deletes table rows whose TTL column has expired |
5455
| Data drains | `/api/cron/run-data-drains` | `0 * * * *` | Enterprise data drains |

‎helm/sim/values.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1483,6 +1483,15 @@ cronjobs:
14831483
successfulJobsHistoryLimit: 3
14841484
failedJobsHistoryLimit: 1
14851485

1486+
connectorDirectorySync:
1487+
enabled: true
1488+
name: connector-directory-sync
1489+
schedule: "*/5 * * * *"
1490+
path: "/api/knowledge/connectors/directory-sync"
1491+
concurrencyPolicy: Forbid
1492+
successfulJobsHistoryLimit: 3
1493+
failedJobsHistoryLimit: 1
1494+
14861495
runDataDrains:
14871496
enabled: true
14881497
name: run-data-drains

‎packages/db/migrations/0321_permission_mirrored_knowledge.sql‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
-- Administrator-mode knowledge connectors: mirrored source permissions.
2+
--
3+
-- Two new tables hold the external directory groups an administrator crawl mirrors onto document
4+
-- ACLs, and who belongs to them. Groups are scoped by workspace, provider and tenant so two
5+
-- connectors over one directory resolve it once; membership is keyed by case-folded email, because
6+
-- a directory reports addresses and most members of a granted group have no Sim account yet.
7+
--
8+
-- `user` gains a case-insensitive unique index. Every identity binding by email compares the folded
9+
-- address, so two accounts differing only in case were one identity to it, each inheriting the
10+
-- other's grants. The pre-check below runs inside the runner's batch transaction and fails the deploy
11+
-- naming the duplicate count, rolling back having changed nothing: a concurrent unique build that
12+
-- hit a duplicate would instead leave an INVALID index that `IF NOT EXISTS` skips on every later run,
13+
-- so the constraint would appear to exist while enforcing nothing.
14+
--
15+
-- Transaction shape: the pre-check and the new (empty) tables run inside the runner's batch
16+
-- transaction. The embedded COMMIT then ends it so the index on the hot `user` table can build
17+
-- CONCURRENTLY without write-blocking it. A failure after the COMMIT replays this whole file
18+
-- against tables that are already committed, so every statement here is idempotent: IF NOT
19+
-- EXISTS on tables and indexes, pg_constraint lookups around the foreign keys, and the
20+
-- DROP / IF NOT EXISTS pair on the concurrent build.
21+
DO $$
22+
DECLARE
23+
duplicate_addresses bigint;
24+
BEGIN
25+
SELECT count(*) INTO duplicate_addresses
26+
FROM (SELECT 1 FROM "user" GROUP BY lower(btrim("email")) HAVING count(*) > 1) AS d;
27+
28+
IF duplicate_addresses > 0 THEN
29+
RAISE EXCEPTION
30+
'Cannot create user_email_lower_unique: % email address(es) are held by more than one account. Merge the duplicate accounts, then re-run this migration.',
31+
duplicate_addresses;
32+
END IF;
33+
END $$;--> statement-breakpoint
34+
CREATE TABLE IF NOT EXISTS "knowledge_external_group" (
35+
"id" text PRIMARY KEY NOT NULL,
36+
"workspace_id" text NOT NULL,
37+
"provider_id" text NOT NULL,
38+
"tenant_id" text NOT NULL,
39+
"external_group_id" text NOT NULL,
40+
"last_synced_at" timestamp,
41+
"created_at" timestamp DEFAULT now() NOT NULL,
42+
"updated_at" timestamp DEFAULT now() NOT NULL
43+
);
44+
--> statement-breakpoint
45+
CREATE TABLE IF NOT EXISTS "knowledge_external_group_member" (
46+
"group_id" text NOT NULL,
47+
"email" text NOT NULL,
48+
"created_at" timestamp DEFAULT now() NOT NULL,
49+
CONSTRAINT "knowledge_external_group_member_group_id_email_pk" PRIMARY KEY("group_id","email")
50+
);
51+
--> statement-breakpoint
52+
DO $$ BEGIN
53+
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'keg_workspace_fk') THEN
54+
ALTER TABLE "knowledge_external_group" ADD CONSTRAINT "keg_workspace_fk" FOREIGN KEY ("workspace_id") REFERENCES "public"."workspace"("id") ON DELETE cascade ON UPDATE no action;
55+
END IF;
56+
END $$;
57+
--> statement-breakpoint
58+
DO $$ BEGIN
59+
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'kegm_group_fk') THEN
60+
ALTER TABLE "knowledge_external_group_member" ADD CONSTRAINT "kegm_group_fk" FOREIGN KEY ("group_id") REFERENCES "public"."knowledge_external_group"("id") ON DELETE cascade ON UPDATE no action;
61+
END IF;
62+
END $$;
63+
--> statement-breakpoint
64+
CREATE UNIQUE INDEX IF NOT EXISTS "keg_identity_unique" ON "knowledge_external_group" USING btree ("workspace_id","provider_id","tenant_id","external_group_id");
65+
--> statement-breakpoint
66+
CREATE INDEX IF NOT EXISTS "keg_workspace_synced_idx" ON "knowledge_external_group" USING btree ("workspace_id","last_synced_at" NULLS FIRST);
67+
--> statement-breakpoint
68+
CREATE INDEX IF NOT EXISTS "kegm_email_idx" ON "knowledge_external_group_member" USING btree ("email");
69+
--> statement-breakpoint
70+
COMMIT;--> statement-breakpoint
71+
SET lock_timeout = 0;--> statement-breakpoint
72+
-- migration-safe: replay removes an invalid build left by an earlier attempt; concurrent operations preserve writes.
73+
DROP INDEX CONCURRENTLY IF EXISTS "user_email_lower_unique";--> statement-breakpoint
74+
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS "user_email_lower_unique" ON "user" USING btree (lower(btrim("email")));--> statement-breakpoint
75+
SET lock_timeout = '5s';

‎packages/db/migrations/0321_user_email_case_folded_unique.sql‎

Lines changed: 0 additions & 35 deletions
This file was deleted.

‎packages/db/migrations/0322_knowledge_external_groups.sql‎

Lines changed: 0 additions & 33 deletions
This file was deleted.

‎packages/db/migrations/meta/0321_snapshot.json‎

Lines changed: 192 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"id": "45719bd3-6ef0-4004-ba42-2e7d04edf80c",
2+
"id": "5796ed80-eec0-4507-aa92-05a124950a85",
33
"prevId": "5a6e3414-7fa3-4206-8e48-73d5f0ded1ce",
44
"version": "7",
55
"dialect": "postgresql",
@@ -8885,6 +8885,197 @@
88858885
"checkConstraints": {},
88868886
"isRLSEnabled": false
88878887
},
8888+
"public.knowledge_external_group": {
8889+
"name": "knowledge_external_group",
8890+
"schema": "",
8891+
"columns": {
8892+
"id": {
8893+
"name": "id",
8894+
"type": "text",
8895+
"primaryKey": true,
8896+
"notNull": true
8897+
},
8898+
"workspace_id": {
8899+
"name": "workspace_id",
8900+
"type": "text",
8901+
"primaryKey": false,
8902+
"notNull": true
8903+
},
8904+
"provider_id": {
8905+
"name": "provider_id",
8906+
"type": "text",
8907+
"primaryKey": false,
8908+
"notNull": true
8909+
},
8910+
"tenant_id": {
8911+
"name": "tenant_id",
8912+
"type": "text",
8913+
"primaryKey": false,
8914+
"notNull": true
8915+
},
8916+
"external_group_id": {
8917+
"name": "external_group_id",
8918+
"type": "text",
8919+
"primaryKey": false,
8920+
"notNull": true
8921+
},
8922+
"last_synced_at": {
8923+
"name": "last_synced_at",
8924+
"type": "timestamp",
8925+
"primaryKey": false,
8926+
"notNull": false
8927+
},
8928+
"created_at": {
8929+
"name": "created_at",
8930+
"type": "timestamp",
8931+
"primaryKey": false,
8932+
"notNull": true,
8933+
"default": "now()"
8934+
},
8935+
"updated_at": {
8936+
"name": "updated_at",
8937+
"type": "timestamp",
8938+
"primaryKey": false,
8939+
"notNull": true,
8940+
"default": "now()"
8941+
}
8942+
},
8943+
"indexes": {
8944+
"keg_identity_unique": {
8945+
"name": "keg_identity_unique",
8946+
"columns": [
8947+
{
8948+
"expression": "workspace_id",
8949+
"isExpression": false,
8950+
"asc": true,
8951+
"nulls": "last"
8952+
},
8953+
{
8954+
"expression": "provider_id",
8955+
"isExpression": false,
8956+
"asc": true,
8957+
"nulls": "last"
8958+
},
8959+
{
8960+
"expression": "tenant_id",
8961+
"isExpression": false,
8962+
"asc": true,
8963+
"nulls": "last"
8964+
},
8965+
{
8966+
"expression": "external_group_id",
8967+
"isExpression": false,
8968+
"asc": true,
8969+
"nulls": "last"
8970+
}
8971+
],
8972+
"isUnique": true,
8973+
"concurrently": false,
8974+
"method": "btree",
8975+
"with": {}
8976+
},
8977+
"keg_workspace_synced_idx": {
8978+
"name": "keg_workspace_synced_idx",
8979+
"columns": [
8980+
{
8981+
"expression": "workspace_id",
8982+
"isExpression": false,
8983+
"asc": true,
8984+
"nulls": "last"
8985+
},
8986+
{
8987+
"expression": "last_synced_at",
8988+
"isExpression": false,
8989+
"asc": true,
8990+
"nulls": "first"
8991+
}
8992+
],
8993+
"isUnique": false,
8994+
"concurrently": false,
8995+
"method": "btree",
8996+
"with": {}
8997+
}
8998+
},
8999+
"foreignKeys": {
9000+
"keg_workspace_fk": {
9001+
"name": "keg_workspace_fk",
9002+
"tableFrom": "knowledge_external_group",
9003+
"tableTo": "workspace",
9004+
"columnsFrom": ["workspace_id"],
9005+
"columnsTo": ["id"],
9006+
"onDelete": "cascade",
9007+
"onUpdate": "no action"
9008+
}
9009+
},
9010+
"compositePrimaryKeys": {},
9011+
"uniqueConstraints": {},
9012+
"policies": {},
9013+
"checkConstraints": {},
9014+
"isRLSEnabled": false
9015+
},
9016+
"public.knowledge_external_group_member": {
9017+
"name": "knowledge_external_group_member",
9018+
"schema": "",
9019+
"columns": {
9020+
"group_id": {
9021+
"name": "group_id",
9022+
"type": "text",
9023+
"primaryKey": false,
9024+
"notNull": true
9025+
},
9026+
"email": {
9027+
"name": "email",
9028+
"type": "text",
9029+
"primaryKey": false,
9030+
"notNull": true
9031+
},
9032+
"created_at": {
9033+
"name": "created_at",
9034+
"type": "timestamp",
9035+
"primaryKey": false,
9036+
"notNull": true,
9037+
"default": "now()"
9038+
}
9039+
},
9040+
"indexes": {
9041+
"kegm_email_idx": {
9042+
"name": "kegm_email_idx",
9043+
"columns": [
9044+
{
9045+
"expression": "email",
9046+
"isExpression": false,
9047+
"asc": true,
9048+
"nulls": "last"
9049+
}
9050+
],
9051+
"isUnique": false,
9052+
"concurrently": false,
9053+
"method": "btree",
9054+
"with": {}
9055+
}
9056+
},
9057+
"foreignKeys": {
9058+
"kegm_group_fk": {
9059+
"name": "kegm_group_fk",
9060+
"tableFrom": "knowledge_external_group_member",
9061+
"tableTo": "knowledge_external_group",
9062+
"columnsFrom": ["group_id"],
9063+
"columnsTo": ["id"],
9064+
"onDelete": "cascade",
9065+
"onUpdate": "no action"
9066+
}
9067+
},
9068+
"compositePrimaryKeys": {
9069+
"knowledge_external_group_member_group_id_email_pk": {
9070+
"name": "knowledge_external_group_member_group_id_email_pk",
9071+
"columns": ["group_id", "email"]
9072+
}
9073+
},
9074+
"uniqueConstraints": {},
9075+
"policies": {},
9076+
"checkConstraints": {},
9077+
"isRLSEnabled": false
9078+
},
88889079
"public.mcp_server_oauth": {
88899080
"name": "mcp_server_oauth",
88909081
"schema": "",

0 commit comments

Comments
 (0)