Skip to content

Comments

fix: reduce excessive logging from status polling and health checks#1178

Open
litlmike wants to merge 1 commit intomoghtech:mainfrom
litlmike:fix-issue-1115
Open

fix: reduce excessive logging from status polling and health checks#1178
litlmike wants to merge 1 commit intomoghtech:mainfrom
litlmike:fix-issue-1115

Conversation

@litlmike
Copy link

@litlmike litlmike commented Feb 8, 2026

Problem

Komodo generates excessive logs (300k+ in 24 hours), filling up storage. The primary sources are high-frequency polling operations that log at info! or warn! level on every cycle.

Root Cause

With the default monitoring_interval of 15 seconds (5,760 cycles/day), several log statements fire on every polling cycle for every resource:

  1. /write request handler logged at info! — every resource refresh triggers write requests for each stack/build/repo/sync, generating an info log per resource per cycle
  2. Resource refresh loop errors logged at warn! — persistent DB or cache errors fire on every poll cycle for every resource
  3. Stack service extraction failures logged at warn! — fires every refresh cycle for stacks with parsing issues
  4. Container regex match failures logged at warn! — fires every 15s monitoring cycle per service
  5. Resource state cache errors (build/repo/procedure/action) logged at warn! — fires every 60s per resource on persistent errors

Fix

Move all high-frequency polling log statements from info!/warn! to debug! level. This dramatically reduces log volume at the default info log level while preserving all diagnostic information when debug level is enabled.

Files Changed

  • bin/core/src/api/write/mod.rs — write request logging: info!debug!
  • bin/core/src/api/write/stack.rs — service extraction warnings: warn!debug!
  • bin/core/src/monitor/resources.rs — container regex match: warn!debug!
  • bin/core/src/resource/refresh.rs — all refresh loop errors: warn!debug!
  • bin/core/src/resource/build.rs — build state cache errors: warn!debug!
  • bin/core/src/resource/repo.rs — repo state cache errors: warn!debug!
  • bin/core/src/resource/procedure.rs — procedure state errors: warn!debug!
  • bin/core/src/resource/action.rs — action state errors: warn!debug!

Fixes #1115

Move high-frequency polling logs from info/warn to debug level:

- /write request handler: info! -> debug! (fires on every resource
  refresh cycle for each stack/build/repo/sync resource)
- Resource refresh loop errors in refresh.rs: warn! -> debug!
  (fires every poll cycle per resource on persistent errors)
- Stack service extraction failures: warn! -> debug!
  (fires every refresh cycle per stack with parsing issues)
- Container regex match failures: warn! -> debug!
  (fires every 15s monitoring cycle per service)
- Resource state cache refresh errors (build/repo/procedure/action):
  warn! -> debug! (fires every 60s per resource on persistent errors)

With default monitoring_interval of 15s (5,760 cycles/day) and
multiple servers/resources, these polling logs were the primary
source of 300k+ daily log entries. All messages remain available
at debug level for troubleshooting.

Fixes moghtech#1115
@cveilleux
Copy link

Here from the Reddit thread to contribute human review.

This reduces log level of many important errors from warn to debug and on first glance seems wrong.

Now maybe one of these log level is running in a high frequency loop and risks polluting the logs, but simply lowering the levels to debug is certainly not the solution. Better insight is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Excesive logging: Over 300k logs in the past 24h

2 participants