Repository navigation
Access can see and use the tables the engine creates; freed space across sessions - #3
Merged
Merged
Conversation
A text value's index key (General Legacy sort order) differed from Access's for
many characters, so Access could not find such a value through an index - and a
table whose name has one, through MSysObjects' ParentIdName index:
- The codes files drop a code's leading zero ('#' is "SC", 0x0C; the space is
"S7"), and 1,065 lines of the two files have such a code. The port read them
as no bytes, so '#', '$', the space, control characters and many accents left
their keys. Jackcess pads the zero back.
- A "Z" character with no extra codes counted as one with an empty run, moving
the accent of every later character one place.
- Access sorts 28 characters as two or three letters (AE for the ligature, ss,
ffi) and gives each letter its own place in the accents that follow. The
codes files do not say which; the list is what ACE 16 wrote.
- Surrogate pairs stay out of the key: ACE 16 writes none for them (Jackcess's
computed codes are not what Access writes).
Measured with ACE 16 indexing every BMP character between 'a' and an accented
letter, plus supplementary characters: 63,499 values, every key now equal. The
golden file holds ACE's keys, ascending and descending, for 1,863 values; on the
old encoder 2,370 of its 3,726 keys differ.
Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
A table Database.CreateTable made was invisible to ACE (qcatco/exportmdb.interop#22). Every reason, each found by ACE 16 refusing something, and fixed as Access does it: - The catalog row: ParentId was 0 and Owner empty. Access gives a table the "Tables" container's Id and the Admin user as owner; the Admin user's SID is the system objects' owner with 0x0102 mixed in, in every Access-made table of the corpus (Access 97 to 2019). - MSysObjects' indexes (Id, and ParentIdName, through which Access finds a table by name) had no entry for the row; MSysACEs had no permissions for the table. The table now gets an entry in each, and an ACE for each inheritable ACE of the Tables container holding the full table permissions - what Access writes, and what the corpus's Access-made tables have. - MSysObjects' Name does not allow Unicode compression, and the engine compressed every text value: ACE read the name as UTF-16 and its table list came back empty. Text is compressed only in columns that allow it; the engine's own Text and Memo columns allow it, as Access's new fields do. - A table's usage maps were on a page of type 0x05, a reference map's page type; Access keeps the rows on a data page owned by no table, and refused the table ("Not a valid bookmark"). - The index writer is rewritten to Access's rules, so Access reads and adds to its trees: entries sorted by key then row; a node holds the last entry under each child but its last, which its tail leads to; each level linked through prev/next with 0 for no page; the page's TDEF and level in its header; every page in the index's used-pages usage map (a primary key now has one, on its table's usage-map page as Access keeps it); pages Access wrote with a shared prefix read and rewritten; splits at any depth, the new root patched into the right index block. Pages earlier versions wrote are read either way. - Keys are written per column as Access writes them (IndexKeys): a flag for each column of a composite key, and a GUID big-endian in 8-byte segments with their length bytes. Checked against ACE for Long, Int, Byte and GUID, both directions. - Pages the engine appended stayed free in the database's global usage map (page 1), so Access handed them out again and wrote over them: tables went missing when ACE added rows, and DAO's compact dropped them. A page the engine appends now leaves the map; past the inline map's window the map becomes a reference map, as Jackcess keeps it. This reaches any file the engine adds pages to, ExportMdb's exports included. - A second table of the same name, and a primary key on a column whose keys are not written, are refused. Checked with ACE 16 against ten table names (#, $, accents, Greek, spaces), a primary key of 3,000 rows ACE then added 2,000 to, text and two-column keys, a memo, 300 tables, and DAO compacting the file: every check passes, and none did before. The new tests here need no Access: the corpus's Access-made tables are the reference. All 32 fail on the previous code; 624 of 624 pass now. Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
Space a delete or an update freed on a long-value column was kept only in memory, so in a later session (another Database.Open) it was used again only on the column's last page (qcatco/exportmdb.interop#23). The column's free-space usage map now lists those pages, as Access keeps it: a delete that leaves a page with room adds it; a page a value fills, or one that is not the column's long-value page, leaves it (UsageMap.RemovePage); and the first write to a column in a session notes the pages the map lists, read for their room when a chunk could fit. ExportMdb, which never deletes, writes no more than before. Tests, each seen failing first: the map stayed empty after the deletes; the next session took two new pages instead of the five freed ones; a filled page stayed listed. ACE 16 reads every value written into the freed space, adds to the table, and DAO compacts the file. Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
A deleted row's primary-key entry stayed in the index, and an update added an entry for the moved row beside the old one. Access counts and finds rows through the key: once ACE could see the engine's tables, it counted 90 rows in a table of 70, the extra twenty being rows the engine had deleted. A delete now takes its entry out (IndexWriter.Remove), and an update moves its entry to the rewritten row. Removal keeps Access's rules, as Jackcess does: a page left empty leaves the tree, its level's chain and the index's usage map; a tail leaving its node makes the node's last entry's child the tail; the entry copying a page's last entry follows it when that changes; a root left with one child gives way to it, and one left with none is an empty leaf again. Tests, each seen failing first: deleting a middle range of keys (whole leaves empty), deleting the top range (the tail empties) then adding after it, deleting every row then adding again, and an update - in each the key holds exactly the table's rows, its usage map exactly its pages, and the tree Access's rules. ACE 16 counts the rows correctly after the engine's deletes. Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
The free-space map change used it, which every target but netstandard2.0 has; the CI build of that target failed. Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
… kept What the fresh review found, each seen failing first (EngineReviewTests): - Text keys are written in General - Legacy only. A column's sort order is now read from its TDEF (SortOrder), and a table is not registered in a catalog that sorts names in Access 2010's General order (or Access 97's): refused before a page is added, the file left byte for byte as it was. A text index in another order is not searched with keys it does not hold; the table is scanned instead (indexCodesV2010). - The engine's Text and Memo columns say General - Legacy (09 04 00 00), as Access 2000-2007 write them. With no sort order, ACE keyed the rows it added in another order, and the engine could not find them by key. - An index root keeps its page, as Jackcess's does: a split moves its entries to two new pages beneath it, and a root left with one child takes that child's entries back. A second Table object held the old root and wrote into it; 150 tables registered in an Access 2003 file broke the catalog's ACE index the same way. - A node an earlier version wrote (an entry for every child, each with row pointer 0, no tail) gets the row of the last entry under each child before the tree is changed, so a delete of a leaf's last key no longer leaves a stale entry behind. - Insert, DeleteRow and UpdateByPrimaryKey refuse a table whose primary key the engine cannot keep before changing anything (a text key in another order, Access 97's included). - A table cannot take a name a query or linked table has; the name check, like every other refusal, comes before a page is allocated. - The header and the global usage map are read once and kept (PageFile.ReadKeptPage): appending 200 pages read 400 pages before. ACE 16 matrix (tables the engine makes, ACE adding to them, DAO compacting): 0 failed. The build before this commit fails "engine finds ACE's text keys". Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
… gave, scans by the index's columns What the second fresh review found, each seen failing first (SecondReviewTests): - A delete or update in a primary key whose pages Access prefix-compressed (indexCodes Table14) changed the row and then failed rewriting the key page uncompressed, leaving the key behind. Pages are now written the way Access writes them, the bytes their keys share stored once, and a page that does not fit after any change - a removal's parent update included - splits. A key sharing nothing with a packed page's other keys can leave a half that still does not fit, so a split halves until every piece does. - A root left with one child that itself had one child kept it: the collapse now repeats, and a root that collapses into a node takes its level. - A table registered by an earlier version (ParentId 0) no longer held its name; any table's name, in any container, is refused again. - FindRowByEntry's scan matched the table's first columns, not the index's. - An earlier version's node is brought to the rules children and all: TDEF named, 0 for no page, its level set. - The definition is laid out before a page is added: a key of eleven columns or names too long for one TDEF page refuse with no pages left behind (a definition over one page is NotSupportedException). - A table made from an Access 97 table's columns sorts in General - Legacy. - IndexKeys.Encode refuses a column whose keys it does not write, so the low-level IndexWriter methods refuse as Table does. - IndexCursor's primary-key path no longer needs a lone uncompressed leaf. Each fix and each sort-order rule the review found untested is held by a test: 19 mutations, one per fix, each caught. Engine tests 661/661. ACE 16: Access-made compressed keys after the engine's deletes, updates and inserts (V2003, V2007) 10/10 each - counts, seeks, ACE's own inserts, DAO compaction, the key's order unchanged by the rebuild; the matrix 52/52 on a new file and on Access 2003 and 2010 templates. Claude-Session: https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY
Author
Review summaryTwo fresh read-only reviews ran on this PR: one on the whole branch (head 270d475), one on the fix commit 8032223. Each finding was seen failing first, then fixed ( First review (270d475) - fixed in 8032223
Deferred, filed where the engine's work is tracked (issues are off here): qcatco/exportmdb.interop#26 (port Access 2010's General order so those files can be written), qcatco/exportmdb.interop#27 (secondary indexes of an Access-made table are not kept). Neither touches ExportMdb: its template sorts in General - Legacy and the shipped one has no indexes. Evidence
Second review (8032223)
Not changed: two Mutations: 19, one per fix, each caught by a test. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups to #2: qcatco/exportmdb.interop#22 (Access could not see a table the engine creates) and #23 (space freed in an earlier session was not used again). Fixing #22 with Access as the referee turned up the rest, and every item is checked against ACE 16.
Commits
"SC"is 0x0C) as no bytes at all, so#,$, the space, control characters and many accents left their keys.aandé, plus supplementary characters: 63,499 values, every key now equal to ACE's.ParentId= the Tables container, the Admin user asOwner.IdandParentIdNameindexes, and MSysACEs permissions as Access gives its own new tables.0for no page, TDEF and level in each page's header, used-pages maps, shared-prefix pages read, splits at any depth.Evidence
ACE 16 matrix, run against the engine's tables. Before this PR nothing passed: ACE could not find the table.
#Hash,$Dollar,Café,naïve résumé,ÆØÅ,Ελληνικά, spaces,O'Brien.Typed keys: Long, Int, Byte and GUID keys equal ACE's in both directions.
Tests: the new engine tests need no Access. The corpus's Access-made tables are the reference: a new table's catalog row and permissions must equal those of an Access-made table in the same file, for files from Access 2000 to 2010. Each new test was seen failing on the code before it:
dotnet test: 631 of 631.Owner and permissions
How these were derived:
0x0102. This holds for every Access-made table of the corpus, from Access 97 to 2019; the only exception is a table Jackcess made.0x000FFEFF. This matched Access's own tables in nearly every corpus file. The exceptions are compaction'sMSysCompactErrortables, tables created under other permissions, and Jackcess-made tables.The ExportMdb.Interop submodule bump, with ACE tests in its classic build, follows in a PR there. That PR closes #22 and #23.
https://claude.ai/code/session_01AYm7ruNjAV5SnM2ZRs1MxY