feat: use cache for workflow queues

[jason-lynch]

Summary

Updates the workflows backend to use the write-through cache for workflow and activity queue items. This eliminates a frequent, expensive range query and enables us to reduce the queue polling frequency to the default 200ms without overloading Etcd. We were previously performing these range queries every 500ms, but the queries themselves could take up to 1.5s - even when the range was empty.

Testing

• Database operations and overall E2E runtime should be ~30% faster.
• If you enable warn logging for Etcd, you should no longer see the workflow and activity range queries reported by the slow query logger

{
  "profiling_enabled": true,
  "client_addresses": ["127.0.0.1"],
  "etcd_server": {
    "log_level": "warn"
  }
}

Notes for Reviewers

I disabled one of the workflow backend tests and included an explanation in the Makefile. This test passes most of the time, but that occasional failure is annoying.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e8d6e334-169f-47f4-a2a5-ea81992dbf07

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cache-workflow-queues

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}