The archive-audit-data cron job (cron-service) calls the audit-service archive API POST /audit-service/audit-logs/archive once per tenant. This API is owned and hosted by another team — we only invoke it and cannot scale or modify that service's infrastructure. On QA, the first tenant's archive request causes the audit-service to fail mid-request, and it stays unreachable for the rest of the run, so the remaining tenants also fail. As a result no audit data is archived and the cron records a failure entry per tenant.
it should be scale or consider batching/streaming the archive instead of loading the full result set into memory
Two distinct failures observed in a single run:
- First tenant → socket hang up (connection opened, but the server closed it before responding → service crashed/restarted mid-request).
- Subsequent tenants → connect ECONNREFUSED / connect EPERM 10.43.48.20:3000 (service no longer accepting connections → down/restarting after the first request).
Expected behavior
The archive API should handle each tenant's archival request without crashing or becoming unreachable, regardless of how many audit-log rows fall in the requested date window. A single tenant's request should not take the audit-service down for the remaining tenants. Each archive call should complete successfully (or return a clean, handled error) so audit data is archived as intended. it should be scale or consider batching/streaming the archive instead of loading the full result set into memory
Screenshots
N/A — backend cron job. Relevant log output and cron_runs.meta.error payloads are included above.
Additional context
- The archive API (POST /audit-service/audit-logs/archive) is owned and hosted by another team; cron-service only calls it. We are not able to scale or change that service's infrastructure, so the fix must come from the sourceloop team.
- API behavior (sourceloop @sourceloop/audit-service audit.controller.js): it runs auditLogRepository.find({where}) to load the entire matching result set into memory (no pagination/limit), CSV-exports it, then deleteAll. For a busy tenant over the requested window this is a large, unbounded in-memory operation — a likely cause of the crash (suspected OOM/restart).
- Failure cascade: socket hang up on the first tenant (server died mid-request) → ECONNREFUSED/EPERM for subsequent tenants (server down/unreachable). The EPERM may additionally point to a NetworkPolicy/egress or conntrack/socket exhaustion issue on the audit-service side.
- Suggested investigation for the sourceloop team: consider batching/streaming the archive instead of loading the full result set into memory; verify NetworkPolicy/egress between cron-service and audit-service.
The archive-audit-data cron job (cron-service) calls the audit-service archive API POST /audit-service/audit-logs/archive once per tenant. This API is owned and hosted by another team — we only invoke it and cannot scale or modify that service's infrastructure. On QA, the first tenant's archive request causes the audit-service to fail mid-request, and it stays unreachable for the rest of the run, so the remaining tenants also fail. As a result no audit data is archived and the cron records a failure entry per tenant.
it should be scale or consider batching/streaming the archive instead of loading the full result set into memory
Two distinct failures observed in a single run:
Expected behavior
The archive API should handle each tenant's archival request without crashing or becoming unreachable, regardless of how many audit-log rows fall in the requested date window. A single tenant's request should not take the audit-service down for the remaining tenants. Each archive call should complete successfully (or return a clean, handled error) so audit data is archived as intended. it should be scale or consider batching/streaming the archive instead of loading the full result set into memory
Screenshots
N/A — backend cron job. Relevant log output and cron_runs.meta.error payloads are included above.
Additional context