[vishalchangrani]

Closes: #998

Description

debug_traceTransaction has been returning -32000 "used gas mismatch, expected: <X>, got: <Y>" for any transaction whose index inside its block is greater than 0, unless the request uses callTracer. The failure reproduces against every recent gateway release (verified back to v1.5.7) with the default struct logger, prestateTracer, 4byteTracer, muxTracer, and other non-callTracer tracers.

Root cause

traceTransaction loads the target's receipt once, then calls BlockExecutor.Run(tx, receipt, txTracer) for every preceding transaction in the block to warm up state — passing the target's receipt to each call. BlockExecutor.Run (services/evm/executor.go:98) compares res.GasConsumed to receipt.GasUsed and returns an error on mismatch, so the loop always failed on the first preceding tx. The error's got value equals the gasUsed of tx #0 in the block, expected equals the target's gasUsed.

callTracer's hook path through flow-go's emulator incidentally produces a GasConsumed reading that happens to satisfy the (incorrect) check, which is why that one tracer worked.

The sibling helper traceBlockByNumber in the same file already does this correctly — it fetches d.receipts.GetByBlockHeight(block.Height) and pairs each tx with its own receipt. This PR applies the same pattern to traceTransaction.

Changes

• api/debug.go: load d.receipts.GetByBlockHeight(block.Height) once, build a txHash → receipt map, pass the correct per-tx receipt into each BlockExecutor.Run() call.
• tests/web3js/debug_traces_test.js: regression test that traces the coinbase fee-transfer tx (at index 1 of its block) with the default (nil) tracer and expects success. Fails on main, passes with this change.

Files most critical to review

• api/debug.go:293-343 — the replay loop in traceTransaction.

---

For contributor use:

• Targeted PR against main branch
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
• Code follows the standards mentioned here.
• Updated relevant documentation
• Re-reviewed Files changed in the Github PR explorer
• Added appropriate labels

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Fixed transaction tracing for transactions other than the first in a block. Traces now use the receipt belonging to each transaction during block replay, helping return accurate results instead of reusing another transaction’s receipt.
  • Tracing now reports an error with the transaction hash and block height when a transaction in the block has no matching receipt.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 4af6f7ce-07fc-4c71-a56a-bb2887beeec3

📥 Commits

Reviewing files that changed from the base of the PR and between 337968b and db68ff4.

📒 Files selected for processing (2)

• api/debug.go
• tests/web3js/debug_traces_test.js

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

traceTransaction now uses each block transaction’s receipt during replay. A regression test checks tracing the second transaction with the default struct logger.

Changes

Transaction tracing

Layer / File(s)	Summary
Receipt pairing and regression coverage api/debug.go, tests/web3js/debug_traces_test.js	traceTransaction indexes the block’s receipts by transaction hash and passes each replayed transaction its matching receipt. It returns an error if a receipt is missing. The regression test checks a successful trace for the second transaction in a block.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: m-peter

Merge Risk: ⚪ Minimal · up to db68f

The change corrects receipt selection when tracing later transactions and adds targeted regression coverage. No actionable merge-blocking risk remains; normal test checks should complete before merging.

Security Architecture Review

Security architecture risk: 🔵 Low · up to db68f

The fix preserves the existing request gate and isolates replay state while correcting receipt selection. No introduced security flaw was established. Additional receipt processing and unverified resource and consistency limits leave limited uncertainty.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — A caller choosing an indexed transaction can cause additional receipt decoding and map allocation for its containing block, and corrected replay can proceed where it previously failed. The inspected change does not establish expanded credential authority, persistent writes or cross-tenant access. Effective resource limits remain unverified.

Trust Boundaries and Controls

• observed — The replay consumer uses indexed receipt hashes for identity and retains executor gas checks. The receipt store checks block-height consistency when storing a receipt set, but the inspected store and consumer do not establish uniqueness or immutability across the separate reads. No attacker-controlled index mutation was demonstrated.

Resilience and Maintainability Implications

• observed — The tracing executor is constructed over ephemeral storage backed by a read-only register snapshot. Request context is used at the entrypoint gate but is not passed into the internal replay function; interruption behavior remains an existing, incompletely assessed property rather than a demonstrated regression.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: fixing the debug_traceTransaction gas mismatch for transactions after the first transaction in a block.
Linked Issues check	✅ Passed	The changes satisfy issue #998. api/debug.go loads receipts for the block, indexes them by transaction hash, and passes each replayed transaction its matching receipt. It returns an error when a mat…
Out of Scope Changes check	✅ Passed	The changes stay within issue #998. The production change fixes receipt pairing during debug_traceTransaction replay. The added test verifies the reported index-greater-than-zero failure. No unrelat…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[m-Peter]

LGTM!

[vishalchangrani]

Verified against FF testnet and mainnet gateways

Ran the fix (tagged v1.5.9-rc.1) through a before/after smoke test on both FF gateways. All non-callTracer tracers failed on both before deploy, and all five flipped to success after.

Testnet (evm-001.devnet0.nodes.onflow.org:8000)

Repro target: block 0x804dd94 (134,536,596), tx 0x516d9977dae43ffd588c71ce3430a5438f64c809648c93bdece7df620aeeb829 at index 1 (preceded by 0x608f939e… at index 0, gasUsed = 21055; target gasUsed = 21000).

Tracer	Before (v1.5.8-rc.1)	After (v1.5.9-rc.1)
default (struct logger)	❌ used gas mismatch, expected: 21000, got: 21055	✅ gas: 21000, failed: false
prestateTracer	❌ same error	✅ returns prestate map
4byteTracer	❌ same error	✅ {} (value transfer, no dispatch)
callTracer	❌ same error	✅ type=CALL, gasUsed=0x5208
default on index-0 (control)	✅ opcode trace	✅ opcode trace (unchanged)

Mainnet (evm-001.mainnet0.nodes.onflow.org:8000)

Repro target: the original externally-reported tx 0xcab2ef27617a8845f958792b2fc88b09cb87cc7c8dd5163a68ff9beb9b5ff5b7 (block 0x4c2d74f, index 10, gasUsed = 3,056,754, CREATE).

Tracer	Before (v1.5.8-rc.2)	After (v1.5.9-rc.1)
default (struct logger)	❌ used gas mismatch, expected: 3056754, got: 36747	✅ gas: 3056754, failed: false + full structLogs
prestateTracer	❌ same error	✅ sender nonce=148, balance=0x4e72fff6b14c50400; coinbase prestate also returned
4byteTracer	❌ same error	✅ {} (CREATE, no 4-byte dispatch)
callTracer	❌ same error	✅ type=CREATE, gasUsed=0x2ea472 (3,056,754), created contract 0x69d8cf3570def8cdf7d14ff041b2f4febf030905
muxTracer (nesting callTracer)	❌ same error	✅ nested callTracer result returned

All reconcile against the on-chain receipt (gasUsed, nonce, created address). Fix verified end-to-end.

Side observation

On the FF public mainnet gateway (mainnet.evm.nodes.onflow.org, still on v1.5.7), callTracer on this same tx accidentally dodged the bug and returned a correct trace before the fix was deployed anywhere. On the FF internal mainnet node and on testnet, every tracer (including callTracer) failed uniformly. That difference is consistent with slight register-store state divergence between the two FF mainnet nodes shifting the GasConsumed reading past the (incorrect) comparison on one but not the other. Worth flagging that the public gateway should also be upgraded off v1.5.7 once this is merged.