Add core watchdog safeguards

[seifreed]

Summary

• add watchdog state scaffolding and config
• track written memory regions for progress detection
• implement loop/timeout/trap safeguards and execute-from-written OEP detection

Testing

• not run

[AbhiTheModder]

CC @trufae @ulexec

[trufae]

intent says:

PR #7 Review: Add Core Watchdog Safeguards

Summary

This PR by @seifreed adds a watchdog subsystem to radius2 for detecting and breaking out of problematic execution scenarios like infinite loops, trap instructions, and stalled execution. The implementation is focused on unpacker/packer analysis (detecting Original Entry Points from self-modifying code).

Verdict: ⚠️ Needs Discussion

The PR adds useful functionality for a specific use case (unpacker analysis), but there are concerns about:

1. Architectural fit - Is this the right layer for this functionality?
2. Integration completeness - No CLI flags, no tests, no documentation
3. Alternative home - Some of this might belong in radare2's ESIL VM instead

---

Understanding the PR

What radius2 Is

radius2 is a symbolic execution framework built on top of radare2. It uses r2's ESIL (Evaluable String Intermediate Language) to emulate programs while tracking symbolic values and constraints. Key use cases:

• CTF challenge solving (finding inputs that satisfy conditions)
• Automated reverse engineering
• Program exploration with symbolic inputs

What This PR Adds

1. WatchdogConfig (processor.rs) - Configuration for loop/stall detection:

• max_steps - Hard limit on execution steps
• pc_repeat_limit - Stop if same PC executed N times consecutively
• pc_window_size / pc_window_repeat_limit - Detect repeating PC patterns (loops)
• no_progress_steps_limit - Stop if no new memory writes for N steps
• trap_repeat_limit - Skip past trap instructions (int3, ud2, hlt)
• exec_from_written_streak_limit + exec_from_written_stable_steps - OEP detection

2. WatchdogState (state.rs) - Per-state tracking:

• Step counters, PC history windows
• Memory write epoch tracking
• Trap hit counters
• OEP (Original Entry Point) detection result

3. WriteLog (memory.rs) - Tracks which memory chunks have been written:

• Uses 256-byte chunk granularity
• Increments epoch on new chunk writes
• Enables "execute from written memory" detection

---

Component-by-Component Analysis

✅ Parts That Make Sense for radius2

• 🟢 Max Steps Limit (max_steps)

• 🟢 PC Repeat Detection (pc_repeat_limit)

• 🟡 PC Window Pattern Detection (pc_window_size, pc_window_repeat_limit)

⚠️ Parts That Need Discussion

• 🟠 Memory Write Epoch Tracking (WriteLog, no_progress_steps_limit)

• 🟠 Trap Instruction Skipping (trap_repeat_limit, is_trap_instruction)

• 🟠 Execute-from-Written OEP Detection

---

Could This Move to radare2?

What Could Move to r2

Feature	Move to r2?	Rationale
Max steps limit	Maybe	r2's aes (step) could have built-in limits
PC repeat detection	Maybe	Could be an ESIL VM option (e esil.maxrep)
Trap skipping	Yes	Anti-debug emulation belongs in r2's analysis layer
Memory write tracking	No	r2 doesn't do symbolic execution, this is radius2's domain
OEP detection	No	Symbolic execution specific heuristics

What Should Stay in radius2

1. Progress detection - Requires symbolic execution context
2. OEP detection - Specific to symbolic unpacker analysis
3. Per-state watchdog tracking - radius2's multi-state model is unique

---

Missing from This PR

• 🔴 No CLI Integration

• 🔴 No Tests

• 🔴 No Documentation

---

Performance Considerations

The implementation adds overhead to every step:

1. step_count increment (minimal)
2. PC history tracking when enabled (O(window_size) memory per state)
3. Write epoch checks when enabled (hash lookups)
4. Trap instruction string matching when enabled (string ops per instruction)

Most overhead is gated behind self.watchdog.enabled, but the WriteLog struct is always allocated in Memory even when watchdog is disabled.

---

Recommendations Summary

Component	Verdict	Action
max_steps	✅ Include	Core safeguard, expose via CLI
pc_repeat_limit	✅ Include	Useful debugging feature
pc_window detection	✅ Include	Consider ring buffer optimization
WriteLog tracking	⚠️ Conditional	Make opt-in (feature flag or lazy init)
Trap skipping	⚠️ Conditional	Document as anti-debug, consider r2
OEP detection	⚠️ Conditional	Consider separate "unpacker mode"

Before Merge Requires:

1. Add CLI flags for basic watchdog options
2. Add at least one test demonstrating loop detection
3. Lazy-initialize WriteLog only when watchdog enabled
4. Document the feature

Optional Improvements:

1. Ring buffer for PC window instead of Vec with remove(0)
2. Architecture-agnostic trap instruction detection
3. Separate "unpacker analysis mode" module