Zyntax Compiler - Development Backlog

Last Updated: November 27, 2025 Current Status: Production-Ready Core (100% tests passing, 71/71 Zig tests)

Recent Progress:

✅ Switch Expression Pattern Matching Complete (November 27, 2025)
- Multi-case switch with literal patterns and else clause
- Tagged union patterns (.some, .none) with graceful non-enum handling
- Error patterns (error.OutOfMemory)
- Grammar support for all pattern types (literal, wildcard, range, struct, enum, array, pointer, error)
- Documentation updated (book/05-semantic-actions.md, book/08-zig-example.md)
✅ LLVM Backend Working (AOT and JIT modes fully functional - functions, structs, generics)
- Parameter value mapping for function calls
- Struct initialization with InsertValue
- Generic function instantiation with comptime
✅ Comptime Parameter Support (Zig grammar: fn foo(comptime T: type, x: T) T)
✅ ZynPEG VM Runtime (No Rust compilation required! JSON command pattern for dynamic AST construction)
- pest_vm for dynamic grammar parsing
- JSON action blocks in .zyn files → runtime interpretation
- CommandInterpreter executes zpeg commands via host functions
- Full pipeline: .zyn grammar → .zpeg module → parse source → TypedAST → HIR → JIT
✅ ZynPEG Grammar Specification (docs/ZYN_GRAMMAR_SPEC.md - Version 2.0 with JSON action blocks)
✅ ZynPEG Integration Test with JIT (Full pipeline: ZynPEG grammar → pest + AST builder → TypedAST → HIR → Cranelift JIT → Execution)
✅ Grammar Conventions Documentation (crates/zyn_peg/GRAMMAR_CONVENTIONS.md - best practices for .zyn files)
✅ Fixed fn_params bug (all_children iteration instead of .find() to collect ALL parameters)
✅ Fixed binary expressions (fold_binary pattern for proper left-associative parsing)
✅ Try expressions in loops fixed (SSA continuation_block handling for Jump terminators)
✅ Generic function monomorphization (comptime type params, type_args extraction, full pipeline)
✅ Pointer JIT execution (address-taken variables allocated on stack, full load/store support)
✅ Address-of operator (&expr creates Reference expression)
✅ Pointer dereference (ptr.* creates Dereference expression)
✅ Compound assignment operators (+=, -=, *=, /=, %=, &=, |=, ^=, <<=, >>=)
✅ Null/undefined literals (parsing support for optional type values)
✅ Union declarations (grammar: union { } and union(enum) { })
✅ Error set declarations (grammar: error { NotFound, ... })
✅ Lambda/closure expressions (full closure support with captured variables)
✅ Indirect function calls (call_indirect via function pointers)
✅ Bitwise operators (&, |, ^, <<, >>, ~)
✅ Defer/errdefer statements (grammar support)
✅ Orelse/catch operators (error handling operators)
✅ Enum declarations (grammar support)
✅ Try expression / error propagation (full try operator implementation)
✅ Switch expressions working (test_zig_jit_switch_expression passing)
✅ Fixed array indexing in loops bug (SSA phi node cycle breaking)
✅ Pattern matching infrastructure complete (Some/Ok/Err working)
✅ All loop tests passing (for, while, continue, break)
✅ Lambda tests passing (basic lambda, capture, calls in loops)
✅ 71/71 Zig E2E JIT tests passing (100%)

🎯 Active Development

0. ZynPEG Parser Generator (HIGHEST PRIORITY) ✅ PHASE 3 COMPLETE

Goal: Build multi-language frontend infrastructure using PEG parser generator

Status: Phase 1-3 Complete, Phase 4 (Multi-language) Planned

Phase 1: Calculator POC ✅ COMPLETE

Create crates/zyn_parser workspace crate
Integrate pest PEG parser (v2.7)
Implement calculator.pest grammar
Build TypedAST generator (pest parse trees → TypedAST)
Integrate with lowering API (LoweringContext::lower_program())
End-to-end JIT execution tests (16/16 passing - 100%)
Bug Fix: Fixed I32 literal translation in ssa.rs (was incorrectly using I64)
Documentation: Phase 1 Completion Report

Key Achievement: Demonstrated full pest → TypedAST → HIR → JIT pipeline ✅

Test Results:

✅ 8 unit tests (grammar parsing + TypedAST building)
✅ 3 E2E JIT execution tests ("2 + 3" = 5, "(2 + 3) * 4" = 20, etc.)
✅ 5 TypedAST validation tests

Phase 2: Zig Subset ✅ COMPLETE

Phase 3: TypedAST Integration ✅ COMPLETE

ZynPEG grammar generates pest + AST builder code
Generated code builds zyntax_typed_ast::TypedProgram directly
Full JIT compilation test (parse → TypedAST → HIR → Cranelift → Execute)
add(10, 20) = 30 works in integration test
Grammar Conventions Documentation (crates/zyn_peg/GRAMMAR_CONVENTIONS.md)
Fixed fn_params bug (use all_children.filter() not .find())
Fixed binary expression rules (use fold_binary pattern)
NOTE: JIT tests require --test-threads=1 (Cranelift not thread-safe)

Phase 3.5: ZynPEG VM Runtime ✅ COMPLETE

No Rust compilation required for grammar users! Dynamic interpretation of JSON action blocks.

JSON Action Block Syntax - .zyn grammars use JSON commands instead of Rust code
ZPEG Module Format - Compiled .zpeg contains pest grammar + JSON command mappings
pest_vm Integration - Dynamic grammar parsing at runtime (no pest_derive compilation)
CommandInterpreter - Executes zpeg commands via host functions
AstHostFunctions Trait - Pluggable AST construction interface
TypedAstBuilder - Uses TypedASTBuilder fluent API from zyntax_typed_ast
CLI Integration - zyntax compile --grammar foo.zyn --source code.lang --format zyn
Grammar Specification - docs/ZYN_GRAMMAR_SPEC.md (Version 2.0)
End-to-end JIT execution - Calculator example returns correct value (42)!

Architecture:

┌─ Compile Time ────────────────────────────────────────┐
│  .zyn grammar  →  ZynPEG Compiler  →  .zpeg module   │
│                                                       │
│  .zpeg contains:                                     │
│  - pest grammar (string)                             │
│  - Rule command mappings (JSON)                      │
└───────────────────────────────────────────────────────┘

┌─ Runtime ─────────────────────────────────────────────┐
│  source.lang + .zpeg  →  pest_vm  →  Commands  → AST │
│                                                       │
│  No Rust compilation required!                       │
└───────────────────────────────────────────────────────┘

Remaining Items (deferred):

Binary expression folding - fold_binary command for proper left-associative arithmetic
Pattern matching - None literal (arena symbol resolution issue)
String operations (needs stdlib integration via plugin system)

Documents:

Grammar Specification ← NEW (Version 2.0 JSON format)
Implementation Plan
Phase 1 Completion
Phase 2 Plan
Parser Library Comparison

1. Reflaxe/Haxe Integration (HIGH PRIORITY) 🚧 IN PROGRESS

Goal: Create reflaxe.Zyntax backend to tap into Haxe's mature ecosystem

Status: Phase 1 Complete, Phase 2 In Progress

Completed Tasks:

Set up Reflaxe project structure (reflaxe.zyntax/)
Implement Haxe Typed AST → Zyntax TypedAST JSON translator
- compileClassImpl() - Classes to JSON
- compileEnumImpl() - Enums to JSON
- compileExpressionImpl() - Full expression support
Create Zyntax CLI (crates/zyntax_cli/) for JSON compilation
Documentation: HAXE_INTEGRATION.md
Test setup with HelloWorld example

Current Phase (2/4): TypedAST JSON → HIR Conversion

Implement typed_ast_to_hir() in CLI
- Function declarations → HirFunction
- Expressions → HIR instructions
- Type mapping (TypedAST → HIR types)
- Control flow translation
Test with simple functions (add, fibonacci)

Remaining Phases:

Phase 3: Standard library mapping
- Haxe stdlib → Zyntax runtime bridge
- String/Array/Map implementations
- I/O and file operations
Phase 4: Advanced features
- Generics instantiation
- Closures with captures
- Async/await integration
- Performance benchmarking

Benefits:

Instant access to thousands of production-tested libraries
Validation at scale with real-world code
Positioning as high-performance backend for Haxe

Compiler Infrastructure

2. LLVM AOT Backend (MEDIUM PRIORITY) ✅ CORE COMPLETE

Status: Working for core features (functions, structs, generics) Location: crates/compiler/src/llvm_backend.rs

Completed (November 27, 2025):

Function calls with parameters (SSA parameter value mapping)
Struct initialization (InsertValue instruction)
Struct field access (ExtractValue instruction)
Generic function instantiation with comptime
Both AOT and JIT modes working
Basic control flow (if/while)
Arithmetic operations

Remaining Work:

Advanced control flow (switch, pattern matching)
Error union types (try/catch)
Implement optimization passes
Cross-platform testing

3. GPU AOT Backend - NVPTX via LLVM (MEDIUM PRIORITY) 📋 PLANNING

Status: Architecture documented, implementation planned for Q1-Q2 2026 Location: docs/GPU_AOT_ARCHITECTURE.md Feature Flag: compute (opt-in)

Goal: Compile GPU kernels from HIR to NVPTX via LLVM IR for high-performance compute workloads (ZynML, QuantDSL, ImagePipe).

Key Features:

NVPTX target via LLVM (nvptx64-nvidia-cuda)
GPU primitives in HIR (ThreadIdx, BlockIdx, SyncThreads, SharedMemAlloc, WarpShuffle)
Kernel metadata in TypedAST (@kernel, @device, @workgroup annotations)
Critical path CPU optimizations for ultra-low-latency execution
CUDA driver runtime integration

Building with GPU Support:

cargo build --release --features compute

Implementation Phases:

Phase	Description	Effort
Phase 1	TypedAST kernel metadata, HIR GPU primitives, NVPTX target setup	3 weeks
Phase 2	Thread indexing, synchronization, shared memory, atomics, warp ops	3 weeks
Phase 3	Tensor cores, async copy, critical path optimizer, SIMD codegen	3 weeks
Phase 4	CUDA runtime, memory pool, unified memory, DSL integration	3 weeks

Milestone: End-to-end ZynML compute() working with GPU by Q2 2026.

Related Docs:

GPU AOT Architecture - Full technical specification
ZynML Unified DSL - DSL with compute() syntax
GPU Compute System - Compute IR design

4. Bytecode Interpreter (LOW PRIORITY)

Status: Specification complete Location: docs/BYTECODE_FORMAT_SPEC.md

Remaining Work:

Implement bytecode VM
Serialization/deserialization
JIT fallback integration

Language Features

4. Error Handling

✅ Zig-Style Error Handling (COMPLETE)

Value-based error handling without stack unwinding. Fully working with E2E tests.

Completed:

Error union types: !T (function returns error or value)
try expression (error propagation, auto-return on error)
try in loops (SSA continuation_block handling)
Chained try expressions
catch operator (handle error with default/handler)
orelse operator (unwrap optional with default)
Error set declarations: const FileError = error { NotFound, ... };

E2E Tests Passing:

test_zig_jit_try_expression
test_zig_jit_try_error_propagation
test_zig_jit_try_in_loop
test_zig_jit_chained_try
test_zig_jit_catch_operator
test_zig_jit_orelse_operator
test_zig_jit_error_union_type

🔄 Haxe-Style Exception Handling (PENDING)

Stack-unwinding exceptions for Haxe/Java-style languages. Required for Reflaxe integration.

Tasks:

Design exception model (throw/catch/finally with stack unwinding)
Implement throw expression in parser
Add exception types to type system
Generate exception handling HIR (landing pads, cleanup)
Cranelift backend support for stack unwinding
Integration with Haxe's try/catch semantics

5. Pattern Matching ✅ COMPLETE

Status: Fully working with E2E tests

Completed:

Known Issue (minor):

None literal (arena symbol resolution - Symbol 7 doesn't resolve)
- Root cause: Parser and SSA arena symbol lookup mismatch
- Impact: Minor - workaround exists using if-let on None branch

Pending Features:

Or patterns (Some(x) | None)
Pattern guards (if conditions)
Slice patterns ([first, .., last])

Standard Library

6. Extended Collections

Status: Core collections complete (Vec, String, HashMap)

Additions Needed:

HashSet
BTreeMap<K,V> / BTreeSet
LinkedList
VecDeque

7. I/O and File System

Status: Not started

Tasks:

File operations (read, write, open, close)
Directory operations
Path manipulation
Stdio (stdin, stdout, stderr)

8. Networking

Status: Not started

Tasks:

TCP sockets
UDP sockets
HTTP client/server (basic)
Async networking integration

Testing & Quality

9. Integration Testing

Status: 280/284 tests passing

Remaining Work:

Fix 4 failing tests
Add more real-world test cases
Performance regression testing
Continuous integration improvements

10. Fuzzing & Property Testing

Status: Not started

Tasks:

Set up fuzzing infrastructure
Property-based testing framework
Continuous fuzzing in CI

Performance

11. Tiered JIT Optimization

Status: Framework complete, needs tuning

Tasks:

Profile-guided optimization (PGO)
Better hot-path detection
Deoptimization support
Inline caching

12. Memory Management

Status: Basic ownership works

Enhancements:

Escape analysis optimization
Stack allocation promotion
Better lifetime elision
Reference counting optimization

Tooling

13. Language Server Protocol (LSP)

Status: Not started

Tasks:

14. Package Manager

Status: Not started

Design Considerations:

Package registry
Dependency resolution
Build system integration
Version management

Documentation

15. Language Reference

Status: Needs comprehensive docs

Required Documentation:

16. Tutorials & Examples

Status: Limited examples

Needed:

"Getting Started" guide
Common patterns cookbook
Real-world application examples
Performance tuning guide

Research & Exploration

17. Effect System

Status: Research phase

Exploration:

Effect type design
Effect handlers
Integration with async/await
Algebraic effects

18. Dependent Types (Advanced)

Status: Basic framework exists

Enhancements:

Refinement types
Indexed families
Full dependent function types
Proof obligations

Completed (Archive)

🚧 Generic Function Parsing (November 18, 2025) - Phase 1 Complete

Goal: Support generic functions with comptime type parameters Status: Parsing complete, monomorphization integration pending

Completed:

Added type_params: Vec<TypedTypeParam> field to TypedFunction
Grammar support for comptime T: type parameters
Builder parses type params and adds to scope as TypeVar
Test infrastructure in place (test_zig_jit_generic_function)

Pending:

Connect existing monomorphization engine (monomorphize.rs)
Type argument separation at call sites
Generic instantiation in lowering/SSA

Note: Monomorphization engine already exists and passes all 11 tests

✅ Try Expression / Error Propagation (November 18, 2025)

Goal: Implement Zig's try operator for error union types (!T) Problem: try expression test failing - returned 0 instead of 42 Root Causes:

Functions returning !T weren't wrapping plain return values in Result::Ok union
ExtractValue instruction had wrong ty field (aggregate type instead of extracted value type)
Union type not handled in Cranelift ExtractValue case Solution:

Added original_return_type: Option<Type> field to SsaBuilder
Added with_return_type() builder method to pass original return type
Modified process_typed_terminator to auto-wrap return values in Result::Ok when function returns error union
Fixed ExtractValue ty fields: discriminant uses HirType::U32, ok_value uses hir_ok_ty, err_value uses hir_err_ty
Added HirType::Union case to Cranelift backend's ExtractValue handler Files Modified:
crates/compiler/src/ssa.rs: Return value wrapping, fixed ExtractValue types
crates/compiler/src/lowering.rs: Pass return type to SsaBuilder
crates/compiler/src/cranelift_backend.rs: Union ExtractValue handling
crates/zyn_parser/tests/zig_e2e_jit.rs: test_zig_jit_try_expression Impact: Try expressions fully working, 23/24 Zig tests passing (95.8%)

✅ Switch Expression Implementation (November 18, 2025)

Problem: Switch expressions not working - Cranelift only processed entry block, ignoring all match logic Root Cause: When control flow expressions (switch/match/if) were in return position, the Return statement overwrote the entry block's Branch terminator with Return, making other blocks unreachable Solution:

Added continuation_block field to SsaBuilder to track merge/end blocks
Control flow expressions set continuation_block when creating branching structure
Return statement handler places Return on continuation block instead of entry block
Preserves entry block's Branch terminator, allowing full CFG traversal Files Modified:
crates/compiler/src/ssa.rs: Added continuation_block tracking and fixed process_typed_terminator
crates/zyn_parser/src/zig.pest: Switch expression grammar
crates/zyn_parser/src/zig_builder.rs: build_switch_expr and build_switch_pattern functions
crates/zyn_parser/tests/zig_e2e_jit.rs: test_zig_jit_switch_expression Impact: Switch expressions fully working, 22/23 Zig tests passing (95.7%)

✅ Array Indexing in Loops Bug Fix (November 18, 2025)

Problem: Stack overflow when accessing arrays inside while loops Root Cause: Infinite recursion in read_variable_recursive when reading loop-carried array variables. The function would bounce between loop header and back-edge blocks infinitely without creating phi nodes. Solution: Create placeholder phi node and write to block BEFORE recursing to predecessors (breaks the cycle) Files Modified:

crates/compiler/src/ssa.rs (lines 1806-1863): Phi cycle breaking logic
crates/compiler/src/typed_cfg.rs: Pattern check wiring, match arm extraction
crates/zyn_parser/src/zig_builder.rs: Consistent symbol interning Impact: Fixed test_zig_jit_array_in_loop, all loop tests now passing (20/21 Zig tests)

✅ Core Compiler Pipeline

Parser → TypeChecker → HIR → Cranelift JIT
98.9% test coverage (20/21 Zig tests passing)
Full end-to-end compilation

✅ Type System

Generics, traits, lifetimes, dependent types
Structural and nominal typing
Multi-paradigm support (OOP, FP, procedural)
Advanced type features (associated types, GADTs)

✅ Async/Await Runtime

Complete executor, task, waker infrastructure
Parameter capture in async state machines
Integration with tiered JIT
Zero-cost futures

✅ Standard Library (Core)

Vec, String, HashMap<K,V>
Iterator trait with 50+ adapters
Basic memory management
93/100 stdlib functions compile

✅ Tiered JIT Compilation

3-tier optimization (Baseline → Standard → Optimized)
Runtime profiling with atomic counters
Hot-path detection and recompilation
Cranelift (fast) + LLVM (optimized) backends

✅ HIR Builder API

Fluent interface for HIR construction
Type-safe SSA value management
Zero-cost abstractions

Priority Matrix

Priority	Category	Item	Effort	Impact
HIGH	Ecosystem	Reflaxe/Haxe Integration	3-4 weeks	Massive
HIGH	Testing	Fix remaining 4 test failures	1-2 days	High
MEDIUM	Compiler	LLVM AOT Backend (CPU)	2 weeks	High
MEDIUM	Compiler	GPU AOT Backend (NVPTX)	12 weeks	High
MEDIUM	Language	Exception Handling	1 week	Medium
MEDIUM	Stdlib	I/O and File System	1-2 weeks	High
LOW	Tooling	LSP Support	2-3 weeks	Medium
LOW	Compiler	Bytecode Interpreter	2 weeks	Low

Next Milestones

Milestone 1: Core Stabilization (Q4 2025 - Current)

✅ Fix continue statement bug
✅ Complete Zig control flow support
✅ Array types and indexing
✅ Fix array indexing in loops (SSA phi node cycle breaking)
✅ String literals support
✅ Pattern matching infrastructure (Some/Ok/Err working)
🔄 Fix None literal resolution (arena symbol issue)
🔄 String operations
🔄 Switch expressions and generic functions

Milestone 2: Production Features (Q1 2026)

LLVM AOT backend completion (CPU)
GPU AOT backend Phase 1-2 (NVPTX target, HIR primitives)
Exception handling (try/catch/finally)
Complete I/O and networking stdlib
Advanced pattern matching
Generic functions with type parameters

Milestone 3: Ecosystem & GPU Integration (Q2 2026)

Complete Reflaxe/Haxe integration
GPU AOT backend Phase 3-4 (tensor cores, CUDA runtime, DSL integration)
ZynML compute() end-to-end working with GPU
Run 100+ Haxe projects through Zyntax
Performance benchmarks vs other targets
100% test pass rate

Milestone 4: Developer Experience (Q3 2026)

LSP implementation
Package manager
Comprehensive documentation
VSCode/IntelliJ integration

Contributing

See individual task items for details. Most tasks have clear acceptance criteria and can be tackled independently.

Key Resources:

docs/ARCHITECTURE.md - System architecture
docs/HIR_BUILDER_EXAMPLE.md - HIR construction guide
docs/ASYNC_RUNTIME_DESIGN.md - Async runtime internals

Getting Started:

Pick a task from the backlog
Check related documentation
Run existing tests to understand the system
Implement incrementally with tests
Submit PR with clear description

Notes

Keep this document updated as work progresses
Move completed items to the Archive section
Update priority matrix quarterly
Track time estimates vs actuals for planning

Uh oh!

FilesExpand file tree

BACKLOG.md

Latest commit

History

BACKLOG.md

File metadata and controls

Zyntax Compiler - Development Backlog

🎯 Active Development

0. ZynPEG Parser Generator (HIGHEST PRIORITY) ✅ PHASE 3 COMPLETE

1. Reflaxe/Haxe Integration (HIGH PRIORITY) 🚧 IN PROGRESS

Compiler Infrastructure

2. LLVM AOT Backend (MEDIUM PRIORITY) ✅ CORE COMPLETE

3. GPU AOT Backend - NVPTX via LLVM (MEDIUM PRIORITY) 📋 PLANNING

4. Bytecode Interpreter (LOW PRIORITY)

Language Features

4. Error Handling

✅ Zig-Style Error Handling (COMPLETE)

🔄 Haxe-Style Exception Handling (PENDING)

5. Pattern Matching ✅ COMPLETE

Standard Library

6. Extended Collections

7. I/O and File System

8. Networking

Testing & Quality

9. Integration Testing

10. Fuzzing & Property Testing

Performance

11. Tiered JIT Optimization

12. Memory Management

Tooling

13. Language Server Protocol (LSP)

14. Package Manager

Documentation

15. Language Reference

16. Tutorials & Examples

Research & Exploration

17. Effect System

18. Dependent Types (Advanced)

Completed (Archive)

🚧 Generic Function Parsing (November 18, 2025) - Phase 1 Complete

✅ Try Expression / Error Propagation (November 18, 2025)

✅ Switch Expression Implementation (November 18, 2025)

✅ Array Indexing in Loops Bug Fix (November 18, 2025)

✅ Core Compiler Pipeline

✅ Type System

✅ Async/Await Runtime

✅ Standard Library (Core)

✅ Tiered JIT Compilation

✅ HIR Builder API

Priority Matrix

Next Milestones

Milestone 1: Core Stabilization (Q4 2025 - Current)

Milestone 2: Production Features (Q1 2026)

Milestone 3: Ecosystem & GPU Integration (Q2 2026)

Milestone 4: Developer Experience (Q3 2026)

Contributing

Notes