Skip to content

refactor: replace tiered aggregation with flat storage#170

Merged
rajsinghtech merged 10 commits intomainfrom
feat/flat-storage
Apr 16, 2026
Merged

refactor: replace tiered aggregation with flat storage#170
rajsinghtech merged 10 commits intomainfrom
feat/flat-storage

Conversation

@rajsinghtech
Copy link
Copy Markdown
Owner

Summary

  • Replaces 13-table tiered scheme (*_minutely, *_hourly, *_daily, flow_logs_current) with 5 flat tables: node_pairs, bandwidth, bandwidth_by_node, traffic_stats, poll_state
  • Adds idempotent migration in Init() — renames existing minutely tables and drops old tiers, preserving data on upgrade
  • Query-time bucketing in GetBandwidth, GetNodeBandwidth, GetTrafficStats via resolveBucketSize(): ≤2h→1min, ≤48h→1hour, else→1day — no information loss from pre-aggregation
  • Removes raw flow log storage (flow_logs_current + InsertFlowLogs); GET /api/flow-logs returns empty
  • Single TSFLOW_RETENTION env var (default 720h / 30 days) replaces TSFLOW_RETENTION_MINUTELY, TSFLOW_RETENTION_HOURLY, TSFLOW_RETENTION_DAILY

Test Plan

  • go test ./internal/... passes (database, handlers, services)
  • Fresh install: verify Init() creates 5 flat tables, no old tables exist
  • Upgrade: run against a DB with old *_minutely tables — verify data is preserved in renamed tables and old tiers are dropped
  • Analytics page loads and shows correct data across all time ranges (1h, 6h, 24h, 7d, 30d)
  • Bandwidth chart buckets correctly for wide time ranges
  • TSFLOW_RETENTION=168h env var respected at startup

@rajsinghtech rajsinghtech merged commit b63076a into main Apr 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant