Skip to content

[Bug]: LanceDB row IDs collide across project_id scopes and overwrite searchable episodes #320

Closed

Description

@zeronxpbee-droid

Area

src/everos

What happened?

Summary

EverOS stores Markdown correctly under separate project_id directories, but LanceDB episode rows can overwrite each other across projects.

The episode primary key is currently constructed as:

id=f"{owner_id}_{entry.entry_id}"

Because app_id and project_id are not included, the same owner’s first daily episode in two different projects receives the same LanceDB ID.
Environment
EverOS: 1.1.0
Python: 3.12
Platform: macOS
Storage: local Markdown + LanceDB
API: /api/v1/memory/add, /flush, /search, /get
All test content and identifiers below are fabricated.
Reproduction
Use the same:
app_id
user_id
date
But two different project IDs:
project-a-test
project-b-test
Add and flush a fabricated memory under Project A.
Confirm that Project A search returns the extracted episode.
Add and flush another fabricated memory under Project B.
Search or list Project A again.
Inspect the Markdown files and LanceDB episode table.
Actual behavior
Both Markdown files exist:
<root>/<app_id>/project-a-test/users/<owner>/episodes/...
<root>/<app_id>/project-b-test/users/<owner>/episodes/...
Both independently allocate:
entry_id = ep_<date>_00000001
They therefore produce the same LanceDB primary key:
<owner_id>_ep_<date>_00000001
After Project B is indexed:
/memory/get for Project A returns 0 episodes.
/memory/get for Project B returns 1 episode.
Project A’s Markdown remains intact.
Direct LanceDB inspection finds only one row for the colliding ID, pointing to Project B.
Project A was searchable before Project B was indexed, confirming that the indexed row was replaced rather than never created.
Expected behavior
Project A and Project B episodes should coexist in LanceDB.
Writing memory under one project_id must not replace an episode belonging to another project_id.
Both of these queries should remain valid:
project_id = project-a-test
project_id = project-b-test
Suspected root cause
EpisodeHandler constructs the row ID without application or project scope:
id=f"{owner_id}_{entry.entry_id}"
The LanceDB repository then performs an upsert keyed by id.
The project scope exists as row fields and query filters, but it is not part of the upsert identity.
Similar ID construction may also exist in other daily-log tables, so atomic_fact, agent_case, and foresight may need review as well.
Suggested direction
Use a deterministic scoped row identity that incorporates at least:
app_id
project_id
owner_id
entry_id
The fix should also include:
Cross-project collision regression tests.
A documented LanceDB rebuild/migration path.
Review of dependent episode/fact references.
Verification that Markdown remains the source of truth.
I can provide additional sanitized probe output if needed.

### Steps to reproduce

1. Start EverOS 1.1.0 with a clean local memory root.

2. Use the same `app_id` and `user_id`, but two different project IDs:

   - `project-a-test`
   - `project-b-test`

3. Add and flush one fabricated memory under `project-a-test`.

4. Confirm `/api/v1/memory/search` or `/api/v1/memory/get` returns the Project A episode.

5. Add and flush another fabricated memory under `project-b-test` on the same day.

6. Query Project A again.

7. Inspect both project Markdown directories. Both episode files exist and both use the same first daily entry ID:

   `ep_<date>_00000001`

8. Inspect the LanceDB `episode` table. Only one row exists for:

   `<owner_id>_ep_<date>_00000001`

   The row now points to Project B.

9. Observe that Project A Markdown still exists, but Project A search/get returns no episode.

### Environment

_No response_

### Logs or screenshots

```shell

Activity

  1. gloryfromca commented on Oct 12, 2026

    @gloryfromca
    Member

    Fixed on main by #481.

    The collision is gone at the key level rather than in the id string: every business table now declares PRIMARY_KEY = ("app_id", "project_id", "id"), LanceDB merges on that key, and Milvus keys its collections on a composed __pk column. Profile search and get, the handlers' change detection and the vector backfill all address rows by the full key, and reflection deprecates atomic facts only inside the project it merged.

    A root indexed before the fix heals itself: the first full pass after upgrading (server start, or everos cascade sync) re-reconciles every indexed markdown file once and records that in .index_reconcile_version. Unchanged entries are skipped by digest, so a healthy root pays one read pass and no embedding. Two roots need a command instead — a server started with EVEROS_DISABLE_CASCADE=1 (everos cascade sync --force-all once), and a multi-project root where reflection ran before the fix (everos cascade rebuild once).

    This ships in 1.4.2. Thanks for the precise reproduction — the two-project / same-date / same-entry-id walkthrough is exactly what made the root cause obvious.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions