fix: reduce excessive logging from status polling and health checks#1178
Open
litlmike wants to merge 1 commit intomoghtech:mainfrom
Open
fix: reduce excessive logging from status polling and health checks#1178litlmike wants to merge 1 commit intomoghtech:mainfrom
litlmike wants to merge 1 commit intomoghtech:mainfrom
Conversation
Move high-frequency polling logs from info/warn to debug level: - /write request handler: info! -> debug! (fires on every resource refresh cycle for each stack/build/repo/sync resource) - Resource refresh loop errors in refresh.rs: warn! -> debug! (fires every poll cycle per resource on persistent errors) - Stack service extraction failures: warn! -> debug! (fires every refresh cycle per stack with parsing issues) - Container regex match failures: warn! -> debug! (fires every 15s monitoring cycle per service) - Resource state cache refresh errors (build/repo/procedure/action): warn! -> debug! (fires every 60s per resource on persistent errors) With default monitoring_interval of 15s (5,760 cycles/day) and multiple servers/resources, these polling logs were the primary source of 300k+ daily log entries. All messages remain available at debug level for troubleshooting. Fixes moghtech#1115
|
Here from the Reddit thread to contribute human review. This reduces log level of many important errors from warn to debug and on first glance seems wrong. Now maybe one of these log level is running in a high frequency loop and risks polluting the logs, but simply lowering the levels to debug is certainly not the solution. Better insight is required. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Komodo generates excessive logs (300k+ in 24 hours), filling up storage. The primary sources are high-frequency polling operations that log at
info!orwarn!level on every cycle.Root Cause
With the default
monitoring_intervalof 15 seconds (5,760 cycles/day), several log statements fire on every polling cycle for every resource:/write requesthandler logged atinfo!— every resource refresh triggers write requests for each stack/build/repo/sync, generating an info log per resource per cyclewarn!— persistent DB or cache errors fire on every poll cycle for every resourcewarn!— fires every refresh cycle for stacks with parsing issueswarn!— fires every 15s monitoring cycle per servicewarn!— fires every 60s per resource on persistent errorsFix
Move all high-frequency polling log statements from
info!/warn!todebug!level. This dramatically reduces log volume at the defaultinfolog level while preserving all diagnostic information whendebuglevel is enabled.Files Changed
bin/core/src/api/write/mod.rs— write request logging:info!→debug!bin/core/src/api/write/stack.rs— service extraction warnings:warn!→debug!bin/core/src/monitor/resources.rs— container regex match:warn!→debug!bin/core/src/resource/refresh.rs— all refresh loop errors:warn!→debug!bin/core/src/resource/build.rs— build state cache errors:warn!→debug!bin/core/src/resource/repo.rs— repo state cache errors:warn!→debug!bin/core/src/resource/procedure.rs— procedure state errors:warn!→debug!bin/core/src/resource/action.rs— action state errors:warn!→debug!Fixes #1115