Describe the bug
Description
I noticed this when deleting a very large group, where I restarted both SC and SP processes halfway
Expected behavior
No overflowing
Actual behavior
While the delete is running, "Messages left to delete" can drop below zero and "Messages deleted" can go above the number of messages the group had when the delete started. In the screenshot, the delete started with 666,677 messages and the counters show 670,000 deleted and -3,323 left. The counters reset when the delete finishes.
Versions
2.6.0
Please list the version of the relevant packages or applications in which the bug exists.
Steps to reproduce
Delete a large group and restart both SC and SP processes.
Also, make sure that the large group keeps on getting more and more errors of the same type.
Relevant log output
Additional Information
Workarounds
None for the counters themselves. They are only wrong while the delete is running and correct themselves once it finishes. The messages in the group are still deleted.
Possible solutions
When a delete starts, ServiceControl records how many messages the group has, then deletes the messages in batches and adds each batch to the deleted count. The starting count and the messages that actually get deleted come from separate queries. When failures keep arriving in the group, the delete can pick up more messages than the starting count included. "Messages left to delete" is the starting count minus the deleted count, which is why it goes negative.
A fix could keep the remaining count from going below zero, or raise the total as more messages are picked up. It could also exclude messages ingested after the deletion job starts.
Additional information
Describe the bug
Description
I noticed this when deleting a very large group, where I restarted both SC and SP processes halfway
Expected behavior
No overflowing
Actual behavior
While the delete is running, "Messages left to delete" can drop below zero and "Messages deleted" can go above the number of messages the group had when the delete started. In the screenshot, the delete started with 666,677 messages and the counters show 670,000 deleted and -3,323 left. The counters reset when the delete finishes.
Versions
2.6.0
Please list the version of the relevant packages or applications in which the bug exists.
Steps to reproduce
Delete a large group and restart both SC and SP processes.
Also, make sure that the large group keeps on getting more and more errors of the same type.
Relevant log output
Additional Information
Workarounds
None for the counters themselves. They are only wrong while the delete is running and correct themselves once it finishes. The messages in the group are still deleted.
Possible solutions
When a delete starts, ServiceControl records how many messages the group has, then deletes the messages in batches and adds each batch to the deleted count. The starting count and the messages that actually get deleted come from separate queries. When failures keep arriving in the group, the delete can pick up more messages than the starting count included. "Messages left to delete" is the starting count minus the deleted count, which is why it goes negative.
A fix could keep the remaining count from going below zero, or raise the total as more messages are picked up. It could also exclude messages ingested after the deletion job starts.
Additional information