What happened?
BigqueryIO creates temporary datasets and tables and then later deletes them. Sometimes, the delete succeeds but the work-item commit fails. This means the temporary dataset/table is deleted but the work-item in which the delete happens is retried. Since BigqueryIO considers a 404 to be an error, the work-item is forever stuck when using a streaming job.
This slows down draining of a job.
To be resistant to failure, a 404 for a delete request should be treated as a success.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
BigqueryIO creates temporary datasets and tables and then later deletes them. Sometimes, the delete succeeds but the work-item commit fails. This means the temporary dataset/table is deleted but the work-item in which the delete happens is retried. Since BigqueryIO considers a 404 to be an error, the work-item is forever stuck when using a streaming job.
This slows down draining of a job.
To be resistant to failure, a 404 for a delete request should be treated as a success.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components