Skip to content

Represent Uint128 as two uint64 words (STF-1828) - #268

Open
oschwald wants to merge 2 commits into
mainfrom
greg/stf-1828-uint128
Open

oschwald wants to merge 2 commits into
mainfrom
greg/stf-1828-uint128

Conversation

@oschwald

@oschwald oschwald commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Uint128 currently stores a big.Int and reconstructs one when decoding the reader's high and low words. Replace it with a Uint128 struct containing High and Low uint64 fields before v2 ships. Every instance is in range, copies are values, and decoding no longer needs arbitrary-precision integers. MMDB encoding stays unchanged.

Closes #39
Fixes STF-1828

  • Add Uint128FromBig with nil, negative, and overflow checks, plus BigInt returning an independent integer.
  • Accept values and non-nil pointers during insertion. Normalize pointers like other scalar types and remove the Uint128 identity-cache special case.
  • Decode and materialize values directly from words or encoded bytes.
  • Document migration from casts and pointer type assertions. Copy returns a value and Equal compares against a value, matching the other scalars. Nil pointers remain invalid insertion inputs.

Validation

Passed go test ./..., go test -race ./..., MMDBWRITER_REFCOUNT_AUDIT=1 go test ./..., GOARCH=386 go test ./..., and precious lint --all on Go 1.27.1.

Tests cover every bit boundary, existing encoding fixtures, conversion rejection and independence, write errors and byte counts, decode errors, pointer/value deduplication, and write/load/rewrite equality at a fixed build epoch.

Allocation measurements

Compared 8f49d9b with this implementation using prebuilt test binaries, GOMAXPROCS=4, and three runs per version in baseline/candidate/candidate/baseline/baseline/candidate order. The copy benchmark cycles through 256 interface-held values. The load benchmark reads 4,096 distinct maps containing Uint128 values.

Benchmark Before bytes/op After bytes/op Before allocs/op After allocs/op
Copy 80 16 2 1
Load 6,537,668 6,078,908 20,774 4,391

Bytes are medians. Timing varied substantially: load samples ranged from 5.39–7.25 ms before and 5.18–7.93 ms after. These measurements establish lower allocation costs, not a general build-speed improvement. Interface boxing still allocates for copying.

Implemented by Codex on behalf of Greg.

Summary by CodeRabbit

  • New Features
    • Uint128 values now use explicit high and low 64-bit words, with conversions to and from big.Int.
    • Decoded Uint128 values are returned as values rather than pointers.
  • Behavior Changes
    • Floating-point equality follows wire encodings: positive and negative zero differ, while NaNs with identical bit patterns compare equal. Signed-zero values are consistently retained during insertion.
    • Custom inserters receiving unsupported input must replace or discard it. Raw-pointer validation applies to direct inserts and inserter results.

@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 73a298b8-8d0d-4db4-9b4a-ce89c296a0cc

📥 Commits

Reviewing files that changed from the base of the PR and between 8f49d9b and 269b5c9.

📒 Files selected for processing (11)
  • CHANGELOG.md
  • inserter/inserter.go
  • mmdbtype/types.go
  • mmdbtype/types_test.go
  • mmdbtype/uint128_test.go
  • mmdbtype/unmarshal_test.go
  • mmdbtype/unmarshaler_cache_test.go
  • store_decoder.go
  • tree_test.go
  • value_store.go
  • value_store_test.go

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


📝 Walkthrough

Walkthrough

Uint128 now uses High and Low uint64 fields with big.Int conversion helpers. Encoding, decoding, value storage, and tree insertion and retrieval use the value representation. Tests cover conversion boundaries, encoding errors, storage normalization, and database round trips.

Changes

Uint128 Value Representation

Layer / File(s) Summary
Uint128 value type and codec
mmdbtype/types.go, mmdbtype/types_test.go, mmdbtype/uint128_test.go, mmdbtype/unmarshal_test.go, mmdbtype/unmarshaler_cache_test.go
Uint128 uses two uint64 fields and adds checked conversion from big.Int. Encoding, decoding, copying, and equality use value semantics. Tests cover conversions, bit positions, malformed input, write errors, and unmarshalling.
Value storage and decoding
value_store.go, store_decoder.go, value_store_test.go
Storage and decoding intern and materialize Uint128 values. Tests cover value and pointer inputs, maximum values, reference handling, and identity behavior.
Tree integration and release notes
tree_test.go, CHANGELOG.md, inserter/inserter.go
Tree tests use Uint128 values and check database round trips. The changelog describes the representation and equality notes; the inserter comment removes out-of-range Uint128 as an unsupported-input example.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 269b5

No identified Uint128 conversion, storage, or tree-integration issue remains to resolve before merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 27.59% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 10 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: replacing Uint128 storage with two uint64 words. The issue identifier does not reduce clarity.
Linked Issues check ✅ Passed Issue #39 requires a two-word Uint128 representation, direct serialization and deserialization, and a conversion from *big.Int. The diff implements Uint128 with High and Low uint64 fields. WriteTo and…
Out of Scope Changes check ✅ Passed The changed production files support the Uint128 representation, encoding, decoding, value normalization, identity handling, and API migration for issue #39. The added and updated tests verify those b…
Full details: Docstring Coverage

Explanation

Docstring coverage is 27.59% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 10 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit counts two limbs of eight,
And stores them neatly by their weight.
It checks each bit from low to high,
Then writes the bytes and lets them fly.
The tree reads back each value right,
And thumps its paws beneath the moonlight.

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Modver result

This report was generated by Modver,
a Go package and command that helps you obey semantic versioning rules in your Go module.

This PR requires an increase in your module’s major version number.
If the new major version number is 2 or greater,
you must also add or update the version suffix
on the module path defined in your go.mod file.
See the Go Modules Reference for more info.

no object *Uint128.Copy in new version of package github.com/maxmind/mmdbwriter/v2/mmdbtype
  Major

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The representation change is consistently applied and thoroughly covers conversions, encoding boundaries, errors, deduplication, and round trips.

Review effort: Balanced
Findings: None

What changed in this PR

Replaces Uint128’s big.Int storage with two uint64 words while preserving MMDB encoding.

Changes:

  • Adds checked big.Int conversion helpers.
  • Updates insertion, decoding, copying, and equality semantics.
  • Expands boundary, error, deduplication, and round-trip tests.
File Description
value_store.go Stores and materializes word-based values.
value_store_test.go Tests pointer normalization and deduplication.
tree_test.go Tests insertion and database round trips.
store_decoder.go Decodes directly from high/low words.
mmdbtype/​unmarshaler_cache_test.go Updates cached decoding expectations.
mmdbtype/​unmarshal_test.go Updates cursor round-trip coverage.
mmdbtype/​uint128_test.go Adds comprehensive conversion and encoding tests.
mmdbtype/​types.go Implements the new Uint128 representation.
mmdbtype/​types_test.go Updates encoding and equality tests.
inserter/​inserter.go Updates inserter documentation.
CHANGELOG.md Documents the breaking API migration.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@horgh horgh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Claude had some comments.

Comment thread CHANGELOG.md
error. The wire encoding holds only the magnitude, so a negative value
previously encoded as its absolute value and produced incorrect data.
- The two validations above apply to direct inserts and to inserter results. A
- `mmdbtype.Uint128` now uses `High` and `Low` uint64 fields instead of

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The migration note does not warn about v1 code that checks for *mmdbtype.Uint128. That code still compiles, because *Uint128 implements DataType through the value-receiver methods. But Tree.Get, Load, the Unmarshaler and the inserter's existing value now return mmdbtype.Uint128 values. So case *mmdbtype.Uint128: never matches, v, ok := x.(*mmdbtype.Uint128) gives ok == false, and an assertion without ok panics. I confirmed that the assertion on a Get result returns false.

The note says only that "Decoding" returns values, which does not clearly cover Get or inserter arguments. This also makes the claim at line 24 false: "This is the one change that does not fail to compile." Suggest that you add an explicit warning here and update line 24.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread mmdbtype/types.go
func (t Uint128) Copy() DataType { return t }

// Equal checks for equality.
func (t Uint128) Equal(other DataType) bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Equal returns false when other is a *Uint128 with the same value. The store accepts the pointer form and treats it as the same value. v1 callers had to use &u, and that form still compiles.

Example: the tree holds Map{"v": Uint128{Low: 42}}. InsertFunc with DeepMerge and Map{"v": &Uint128{Low: 42}} goes to the default branch of deepMerge (inserter.go:327), which calls Equal and gets false. The existing map is then cloned and interned again for every record, although the output does not change. In v1 both sides were pointers and compared equal.

Equal is also asymmetric now: (&u).Equal(u) is true, but u.Equal(&u) is false. Suggest that you also accept *Uint128 (non-nil) in the type check.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread mmdbtype/types.go

// Equal checks for equality.
func (t Uint128) Equal(other DataType) bool {
otherT, ok := other.(Uint128)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling Equal on a nil *Uint128 now panics: value method Uint128.Equal called using nil *Uint128 pointer. Map{"v": (*Uint128)(nil)}.Equal(other) also panics, because Map.Equal calls Equal on each child.

Before this PR, this case returned false. This PR removes the tests for both nil cases ("nil Uint128 compared to ...") and the CHANGELOG entry that promised this behavior. Was this change intentional? If yes, the CHANGELOG should say so. If no, suggest that you keep a pointer-receiver path or restore the tests.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread tree_test.go
@@ -1421,6 +1409,39 @@ func TestReinsertingGetViewsUsesIdentityCache(t *testing.T) {
// TestEmptyContainersRoundTrip pins empty values through serialization and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestUint128TreeRoundTrip sits between the doc comment for TestEmptyContainersRoundTrip and that function. The comment now documents the wrong test, and TestEmptyContainersRoundTrip (line 1445) has no comment. Suggest that you move the new test above the comment.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread mmdbtype/uint128_test.go
assert.Equal(t, test.decimal, input.String(), "conversion changed its input")
output := value.BigInt()
assert.Equal(t, test.decimal, output.String())
input.SetInt64(7)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These storage-sharing checks (input.SetInt64(7), output.SetInt64(9) and the second check of value) can never fail now. Uint128 is a plain value struct and cannot share storage with a big.Int. The input-unchanged check at line 32 and the BigInt round trip are enough. Suggest that you remove lines 35–38.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread CHANGELOG.md
`big.Int`. Use `Uint128FromBig` and `BigInt` for conversion. Decoding now
returns values instead of pointers. Thanks to Luiz Ferraz (@Fryuni) for the
suggestion. GitHub #39.
- Raw-pointer validation applies to direct inserts and to inserter results. A

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new Uint128 bullet now separates this bullet from the raw-pointer bullet at line 109 that it qualifies. Readers see a bullet that refers back past an unrelated entry. Suggest that you merge this text into the bullet at line 109, or move the Uint128 bullet.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread store_decoder.go
integer.Add(integer, new(big.Int).SetUint64(lo))
value := mmdbtype.Uint128(*integer)
ref, err = d.store.internUncached(&value)
ref, err = d.store.internUncached(mmdbtype.Uint128{High: hi, Low: lo})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: internUncached boxes the 16-byte struct into DataType. On Load, this causes one heap allocation for each Uint128 at a new offset, only to call WriteTo. internScalar exists to avoid this boxing, and it works now that Uint128 is a value type. Suggest that you add Uint128 to internScalarValue and use internScalar here, as for the other scalars.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread mmdbtype/types.go
numBytes += int64(written)
if err != nil {
return numBytes, fmt.Errorf("writing uint128: %w", err)
// Match the other unsigned types: omit leading zero bytes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: this loop makes up to 16 WriteByte interface calls, each with a word and shift branch. One buffer write does the same job:

var buf [16]byte
binary.BigEndian.PutUint64(buf[:8], t.High)
binary.BigEndian.PutUint64(buf[8:], t.Low)
n, err := w.Write(buf[16-size:])

This matches how BigInt encodes its value, and the byte count comes from the Write result.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Comment thread mmdbtype/types.go
// in its input, since only direct inserts and inserter results are
// validated.
type Uint128 big.Int
// Uint128 is the MaxMind DB unsigned 128-bit integer type.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type doc no longer says that inserting a nil *Uint128 returns an error. It also does not say that the store still accepts the pointer form on insert and converts it to a value, so Get returns a Uint128, not the pointer that was inserted. Suggest that you add both points here, because v1 callers will still pass &u.

🤖 Comment by Claude (Claude Code) on behalf of Will.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: Represent Uint128 as two uint64 instead of big.Int

3 participants