Write files of any size: usage maps grow, inserts do not slow

[markmario]

The engine could not write a file past 2 MB, and every insert was slower than the last. Both broke ODES's Egress on any Ringtail export of real size (qcatco/exportmdb.interop#20, qcatco/odes#155). Issues are disabled on this fork, so they are tracked on exportmdb.interop.

What changed

• Usage maps grow. A table's pages, and each long-value column's, are listed in a usage map. Its inline form covers one window of pages: 512 in Access's own rows, 1,600 in this engine's. Adding a page outside the window threw. Now UsageMap.AddPage moves the window when every page the map holds still fits in one. Otherwise it turns the map into a reference map, whose map pages are allocated as needed. This is what Access and Java Jackcess do (UsageMap.InlineHandler / ReferenceHandler).
• Inserts cost the same however large the table.
  • DataPageWriter adds a row to the table's last data page, as Access does, instead of reading every owned page to find room. Rows also stay in insertion order now. Only a data page of this table takes the row: a map that lists another table's page last is passed over.
  • LvalWriter tries the column's last page, and the space values of the same column freed in the session (FreedLvalPages, kept per column with the room each page has). It reads a freed page only when it is known to have room, and a value's last chunk can take a freed page that was too small for its full chunks. Only a long-value page of this column takes a chunk.
• A row's address is Jet's own: the page in 24 bits, the row in 8 (RowPointer, used by the insert, the index writer, reader and cursor). Held as page << 16, a row on a page past 32,767 was indexed at the wrong page once files could grow that far.
• A page holds at most 255 rows. Jet points at a row with one byte. One-byte rows went 600 to a page.
• Text longer than its column is refused, as Access refuses it, with ColumnValueTooLongException (an ArgumentException) naming the column, its width and the value's length. Jet3 widths count characters, Jet4's bytes. Written whole, ACE read such a value cut short and typed readers refused the table. Table.Validate(row) checks a row without writing it, and an update checks the merged row before it touches the old one.

Red, then green

Each test was seen failing for the reason it names, and passing after. For a guard added after its fix, the red comes from undoing the fix in place (a mutation):

Test	Red	Green
LargeFileTests.Rows_past_the_first_usage_map_window_are_written_and_read_back_in_order	NotSupportedException: Page 1600 is beyond the inline bitmap capacity	2,000 rows written and read back
LargeFileTests.Each_insert_reads_about_as_many_pages_however_large_the_table	the first 500 inserts read 192,571 pages, the next 500 570,220	flat
LargeFileTests.A_data_page_holds_at_most_255_rows	600 rows on one page	at most 255
UsageMapTests (9 tests)	8 fail on the inline-only map; the in-window case already worked	pass
TextColumnLengthTests.Text_longer_than_its_column_is_refused_rather_than_written	no exception; the 12-character value was written	ColumnValueTooLongException naming the column and its width
ReviewFindingsTests.A_row_on_a_page_past_32767_is_indexed_and_found_by_its_primary_key, rows on pages 32,768, 40,001, 65,536 and 70,001	with the 16-bit address: the index names page 16,744,448 for 32,768 and 0 for 65,536	the row's own page, and the row found by its key after reopening
ReviewFindingsTests.An_update_refused_for_text_longer_than_its_column_leaves_the_row_as_it_was	the row was deleted: 1 row where 2 were	the row, its memo and its key lookup unchanged, in the session and after reopening
ReviewFindingsTests.Deleting_a_long_value_in_one_table_does_not_slow_long_value_inserts_in_another	after 1 MB is deleted from one table, 254 reads per insert into another, against 8	8
ReviewFindingsTests.Space_a_deleted_long_value_freed_is_used_again_without_reading_every_page_it_freed	498 reads per insert into the freed space, against 8 onto a new page	no new pages, and no more reads
ReviewFindingsTests.A_Jet3_text_column_counts_its_length_in_characters	mutation (Jet3 counted as Jet4) fails	pass
ReviewFindingsTests.Long_values_share_the_column_s_last_page_while_it_has_room	mutation (every value on a fresh page) fails	pass
ReviewFindingsTests.Space_freed_in_one_long_value_column_is_not_used_by_another	mutation (freed pages not checked against the column's map) fails	pass
ReviewFindingsTests.A_row_never_goes_onto_another_table_s_page_listed_in_its_map	mutation (no type or owner check on the last page) fails	pass
ReviewFindingsTests.A_long_value_never_goes_onto_a_page_that_is_not_a_long_value_page	fails without the check	pass
FreedSpaceTests.The_last_chunk_of_a_long_value_reuses_a_page_a_delete_left_partly_free	five 5,000-byte values after a delete took 7 new pages	5, one per full chunk
FreedSpaceTests.Room_left_on_a_freed_page_is_noted_however_the_value_reached_it	noted with 2,080 bytes free, the page has 1,146	noted as it is
FreedSpaceTests.A_freed_page_holding_as_many_rows_as_a_page_can_is_forgotten_whatever_its_room	kept as having room	forgotten
FreedSpaceTests.A_column_s_freed_space_is_its_own_in_a_table_with_two_long_value_columns	mutation (freed pages kept for the whole file) fails	pass
FreedSpaceTests.A_freed_page_that_is_not_on_the_column_s_map_is_never_written_to	mutation (no map check) fails	pass
FreedSpaceTests.A_freed_page_that_is_not_a_long_value_page_is_never_written_to	mutation (no LVAL signature check) fails	pass
FreedSpaceTests.A_freed_page_with_less_room_than_recorded_is_read_before_anything_goes_onto_it	mutation (the record trusted without reading the page) fails	pass
FreedSpaceTests.Space_a_delete_frees_in_a_file_Access_wrote_is_used_again_and_nothing_else_changes (blobV2007.accdb)	mutation (freed space never used again) fails	pass: same page count, every other row unchanged

The read-count tests in ReviewFindingsTests also reopen the file and read every value back.

The map tests cover a reference map's second, third and fourth map pages with page numbers alone, without a 128 MB file.

Reviews

• A fresh review of 5ecc214 found the update that deleted its row and the row addresses past page 32,767, fixed in c4bda54 with the guards above.
• Review 5412715460 on 5ecc214 found those two as well, and the file-wide freed-page scan. All three are answered in its threads; the scan is fixed in ff08da3.
• A fresh review of ff08da3 found no data-integrity problem. It found one space regression - a value's last chunk no longer reused partly freed pages - and two room records that could go stale, fixed in 0f6ee30, and five gaps in the tests, filled there.

Verified

• The suite on 0f6ee30: 587 passed, 0 failed (net8.0). CI green on 0f6ee30.
• In exportmdb.interop (qcatco/exportmdb.interop#21, whose submodule points at 0f6ee30):
  • both editions' suites pass;
  • ACE reads a large engine-written export identically to what was written;
  • the engine reads a large ACE-written export identically to ACE.
• Exports written by the engine and read back, with ACE's row counts matching in every table, are measured on #21. The largest, 70,000 documents in 162.5 MB, is past the 32,736 pages one map page covers. An export ACE itself wrote, 65,536 documents in 544.5 MB, is read by the engine with every document, page and note checked.

Known, tracked

• ACE cannot see a table the engine itself creates, of any size: qcatco/exportmdb.interop#22. It was so before this change, and ExportMdb never creates a table.
• Space freed in an earlier session is reused only on a column's last page: qcatco/exportmdb.interop#23. ExportMdb never deletes.

https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY

[markmario]

Reviewed commit 5ecc2145b5eb9240dbfcc9d2de96328a77392fb4 against base 3721d0418350c79cc8fd7fe458c1398852e79edb.

Recommendation: fix the three findings below before merging. Two affect data integrity; one introduces a cross-table performance regression.

Priority	Finding	Verified effect
P1	Text-width rejection happens after the original row is deleted during an update	Rejected update permanently removes the record, including after reopening
P1	Newly enabled large-file inserts exceed the existing packed row-address representation	Incorrect primary-key index addresses at page 32,768; wrapped addresses at page 65,536
P2	Freed LVAL pages are scanned globally for every long-value write	Deleting a 1 MB value in one table raises unrelated-table inserts from 8–9 page reads on base to 254 on this PR

Validation:

• Built and ran the complete net8.0 suite for the exact PR head: 566 passed, 0 failed, 0 skipped.
• Built the base revision separately and ran the targeted reproductions against both DLLs.
• No tracked source changes were made during the review.
• The large-file boundary probe checks the library's indexed lookup and the raw index entry; it did not use ACE. The new growth support turns an existing pointer-width limitation into successful writes with invalid indexes, whereas the base rejects these allocations.

Detailed reproduction steps, observed results and suggested corrections are attached inline. The existing green suite does not cover these scenarios.

[markmario]

[P1] Validate the merged row before deleting the existing record

This new rejection runs inside InsertRow, but DataPageWriter.UpdateRowByPrimaryKey first frees the original row's LVAL chains and marks its slot deleted (lines 235–244), then calls InsertRow for the replacement. An ordinary overlength-text validation error therefore permanently deletes the record the caller was trying to update.

Minimal reproduction against the PR head:

using (var db = Database.Create(path, JetVersion.Jet4))
{
    var table = db.CreateTable("T", new[]
    {
        new ColumnBuilder("Id", DataType.Long).Build(),
        new ColumnBuilder("Name", DataType.Text).MaxLength(3).Build()
    }, primaryKey: "Id");

    table.Insert(new Row { ["Id"] = 1, ["Name"] = "old" });
    try
    {
        table.UpdateByPrimaryKey(1, new Row { ["Name"] = "toolong" });
    }
    catch (ArgumentException) { }

    // PR head: Count == 0; primary-key lookup also finds nothing.
    Console.WriteLine(table.ReadAllRows().Count);
}
using (var db = Database.Open(path))
    Console.WriteLine(db.GetTable("T").ReadAllRows().Count); // Still 0.

The head reports Column 'Name' holds at most 3 characters; the value has 7. and leaves zero rows. The exact base accepts this update and retains one row before and after reopening. Rejecting an oversized value is the intended new behaviour; losing the original record is the regression.

Validate the merged replacement before freeing old long values or marking the old slot deleted. Add an update-rejection regression test asserting the original row and its values survive both immediately and after reopening.

[markmario]

Fixed in c4bda54: UpdateRowByPrimaryKey now checks the merged row (RowEncoder.Validate) before it frees the old row's long values or marks its slot deleted, so a refused update leaves the row alone.

ff08da3 widens the test to what you asked for. ReviewFindingsTests.An_update_refused_for_text_longer_than_its_column_leaves_the_row_as_it_was checks the row, its memo and the primary-key lookup both in the session and after the file is reopened.

Red/green: with the check moved back after the delete, as it was in 5ecc214, the test fails (Expected: 2, Actual: 1 rows). On ff08da3 it passes.

[markmario]

[P1] Preserve full row addresses before enabling these larger allocations

Moving/converting the usage map now permits data pages beyond 32,767, but DataPageWriter.InsertRow still packs (dataPage << 16) | rowNum into a signed int. IndexWriter.BuildLeafEntry extracts the page with (rowPointer >> 16) & 0xFFFFFF: at page 32,768 this sign-extends to 0xFF8000 instead of 0x008000; at page 65,536 the page bits wrap to zero. The existing representation predates this PR, but the newly accepted allocations make corrupting writes reachable.

Bounded reproduction, without inserting hundreds of megabytes of records:

1. Create a Jet4 database and a table T with a single Long primary-key column Id.
2. Before the first insert, extend that temporary file to 32768L * 4096 bytes using a FileStream opened with FileShare.ReadWrite. This places the next allocated data page at the boundary; the empty intermediate pages are not owned by the table.
3. Insert new Row { ["Id"] = 42 }, dispose the database, reopen it, then call NewIndexCursor().FindRowByPrimaryKey(42).
4. Repeat with 65536L * 4096 bytes.

Observed results:

Boundary	PR head	Exact base
Page 32,768 / 128 MiB	Insert succeeds; raw leaf entry is 7F8000002A FF8000 00; indexed lookup after reopening throws EndOfStreamException for page 16,744,448	Insert rejected with NotSupportedException because it exceeds the inline map capacity
Page 65,536 / 256 MiB	Insert succeeds; stored index page address is 000000; indexed lookup returns null	Same allocation rejection

Plain table scans still find the inserted rows on the head, so row-count/readback checks alone miss the invalid index entries.

Correct the packed row-address representation and all its producers/consumers to preserve the full Jet page address, or reject unsupported addresses before mutating the file. Add indexed readback tests at both boundaries alongside the new usage-map growth tests.

[markmario]

Fixed in c4bda54. A row pointer now uses Jet's own layout, with the page in the high 24 bits and the row in the low 8 (Indexes/RowPointer.cs). The row insert, the index writer, the index reader and the index cursor all use it. RowPointer.Pack throws for a page it cannot hold, past 16,777,215, which is a 64 GiB file and far beyond Jet's 2 GB.

A_row_on_a_page_past_32767_is_indexed_and_found_by_its_primary_key checks two things after reopening: the page in the raw leaf entry (IndexReader.FindRowPointers) and the primary-key lookup. ff08da3 adds your two boundaries: rows on page 32,768 and on page 65,536, alongside the existing 40,001 and 70,001.

Red/green: I put the 16-bit shift back. The leaf entry then names page 16,744,448 for 32,768 and page 0 for 65,536, your figures, and all four cases fail. On ff08da3 all four pass.

[markmario]

[P2] Restrict freed-page candidates to the owning LVAL column

PageFile.FreedLvalPages covers the entire database. This loop sorts every freed page and calls UsageMap.Contains for it on every Memo/OLE write, even when the page belongs to another table or column. Each membership check rereads the target column's usage map. Unrelated pages never enter this picker's _pages, so Done() cannot remove them and the overhead persists for the rest of the session unless their owner reuses them.

Reproduction with two otherwise small tables A and B, each having Long Id and OLE Blob columns:

1. Insert a 1,000,000-byte OLE value in A.
2. Measure reads while inserting a 3,000-byte OLE value in B.
3. Delete A's value with a.DeleteRow("Id", 1).
4. Measure reads for two further 3,000-byte inserts in B.

The identical probe against base and head counted page reads using a delegating ICodecHandler on PageFile.Codec:

B insert	Base	PR head
Before deleting A's value	7	7
First after deletion	8	254
Second after deletion	9	254

Deleting one unrelated 1 MB value adds roughly 247 map reads to every subsequent long-value insert in B. Larger deleted values make the penalty scale with storage elsewhere in the database, counteracting the bounded insert cost this PR is intended to provide.

Index or cache freed-page candidates by their owning column/usage-map identity instead of scanning the file-wide set for each value. Add a performance regression test covering deletes in one table followed by inserts in another.

[markmario]

Fixed in ff08da3. Freed pages are now kept per column (Pages/FreedLvalPages.cs), keyed by the owned-pages map of the column whose value was freed, with the room each page had when last seen. A long-value write:

• looks only at its own column's freed pages,
• reads one only when it was last seen with room for the chunk, and at most once per value,
• checks it against the column's map and the LVAL signature before writing to it,
• notes the room left once the value is written.

Pages freed in other tables or columns are never read.

Your probe's shape: reads for each 3,000-byte insert after a 1,000,000-byte value is deleted.

Insert	c4bda54	ff08da3	Nothing deleted
Into another table	254	8	8
Into the same column	499	7	8

The delete reads 249 pages in both builds. Reusing freed space is now cheaper than taking a new page.

Two new tests, both red on c4bda54 and green on ff08da3:

• Deleting_a_long_value_in_one_table_does_not_slow_long_value_inserts_in_another requires the same reads as the same inserts with nothing deleted. On c4bda54: [254, 254, 254] against [8, 8, 8].
• Space_a_deleted_long_value_freed_is_used_again_without_reading_every_page_it_freed checks that five values after the delete add no pages and read no more than inserts onto new pages. On c4bda54: 498 reads against 8.

Full suite on ff08da3: 579 passed, 0 failed.

[markmario]

A fresh review of ff08da3, and what 0f6ee30 does about it.

The reviewer worked read-only, with its own probes, including on copies of corpus files Access wrote. It found nothing that writes a chunk where it doesn't fit, breaks a chain, writes onto another column's or table's page, or damages a file Access wrote.

Finding	Fixed in 0f6ee30	Test, red then green
P2: a value's last chunk no longer reused pages a delete left partly free (7 new pages where 5 do)	a last chunk smaller than the full chunks walks the freed pages again, reading no page twice	The_last_chunk_of_a_long_value_reuses_a_page_a_delete_left_partly_free: 7, then 5
P3: a freed page reached as the column's last page kept an overstated room record	the room on every page read or written is noted again when the column has it noted	Room_left_on_a_freed_page_is_noted_however_the_value_reached_it: 2,080 noted for 1,146, then the page's own
P3: a page holding 255 rows kept its room and was read again by every write that fits	such a page is noted as having none	A_freed_page_holding_as_many_rows_as_a_page_can_is_forgotten_whatever_its_room
Test gaps: no value read back after the read-count tests; two long-value columns in one table; the freed path's checks; reuse in a file Access wrote	FreedSpaceTests, and ReviewFindingsTests reopening the file	each guard shown failing with its check removed (5 mutations, 8 failures)

Not changed: finding room costs in-memory work in proportion to a column's freed pages, with no page reads. This only matters after deleting hundreds of MB and refilling with small values; ExportMdb never deletes.

Suite 587/0 (net8.0); CI green on 0f6ee30.