Scheduled (cron) workflows never fire on a hosted multi-tenant relay; manual trigger works

[yjc801]

Summary

On a hosted multi-tenant Buzz deployment (*.communities.buzz.xyz), workflows
with trigger: {on: schedule, cron: ...} are accepted, listed, and execute
correctly via manual kind:46020 trigger — but the cron scheduler never fires
them. No error is surfaced to the owner anywhere; the workflow simply does
nothing on schedule.

Evidence (2026-08-30)

• A daily workflow (cron: "0 9 * * *", created the previous evening, owner is
  an active member of the target channel) did not fire at 09:00 UTC, and also
  not at 16:00 UTC (ruling out a local-timezone interpretation). Manually
  triggering the same workflow immediately posts its send_message step.
• A fresh probe workflow with cron: "* * * * *" (every minute) in a fresh
  private channel produced zero fires over 6.5 minutes (~6 missed cron
  instants). Same owner, same community, definition created via buzz workflows create (relay-side parse_yaml accepted it).
• Relay NIP-11 reports version 0.2.1.

Client-side causes ruled out against current main:

• WorkflowDef.enabled defaults to true (buzz-workflow/src/schema.rs).
• 5-field cron is normalized for the cron crate by normalize_cron.
• The canonical definition JSON satisfies the scheduler's eligibility filter
  (definition->'trigger'->>'on' = 'schedule').
• check_owner_authority requires only active channel membership for a
  send_message-only workflow — satisfied.

Since only relay logs can distinguish the remaining causes and community
operators/users have no access to them, filing here. Useful log greps:
WorkflowEngine cron loop started, Cron tick: failed to load workflows,
Cron tick: invalid cron expression, Cron trigger fired.

A concrete bug regardless of the above: silent unbounded truncation

list_all_enabled_workflows (crates/buzz-db/src/store/workflow.rs) is the
scheduler's per-tick scan and is platform-wide, ordered oldest-first, capped
at LIST_MAX_LIMIT = 1000 rows:

WHERE w.status = 'active'
  AND w.enabled = TRUE
  AND w.definition->'trigger'->>'on' = 'schedule'
  AND c.archived_at IS NULL
ORDER BY w.created_at ASC
LIMIT $1   -- LIST_MAX_LIMIT = 1000

On any deployment where the total number of active scheduled workflows across
all communities exceeds 1000, every newly created scheduled workflow falls
past the limit and silently never fires — exactly the symptom above — with no
warning logged and no signal to the owner. Even if this isn't the cause here,
the query needs either pagination, removal of the cap, or at minimum a
warn! when rows.len() == LIST_MAX_LIMIT.

Related observability gap

When the scheduler skips a workflow (parse failure, missing channel_id,
authority failure, claim errors), the only signal is a relay-side log line.
The owner-facing surfaces (workflow list, run history) show nothing. A
"last scheduled evaluation" status or an owner-visible error event would make
this class of failure diagnosable without relay access.

Repro

1. On a multi-tenant deployment, create a workflow:

   name: scheduler-probe
   trigger:
     on: schedule
     cron: "* * * * *"
   steps:
     - id: tick
       action: send_message
       text: scheduler tick

2. Wait 3+ minutes: no messages appear.
3. buzz workflows trigger --workflow <id>: message appears immediately.