Skip to content

Dataset download bug fix#945

Draft
sanjaychelliah wants to merge 2 commits intomasterfrom
dataset_export_bug
Draft

Dataset download bug fix#945
sanjaychelliah wants to merge 2 commits intomasterfrom
dataset_export_bug

Conversation

@sanjaychelliah
Copy link
Contributor

Issue 1: Silent Exception Swallowing ✅ FIXED

Problem: ThreadPoolExecutor wasn't checking for exceptions.
Solution: Added proper exception handling with future.result() to catch and log all failures.

Issue 2: Thread-Safety Race Condition ✅ FIXED (NEW)

Problem: Multiple threads writing to the same ZipFile object simultaneously.
Error: Can't write to ZIP archive while an open writing handle exists.
Solution: Added threading.Lock() to synchronize all ZIP file writes.
Protected operations:

_save_image_to_archive() - line 148
_save_text_to_archive() - line 164
_save_audio_to_archive() - line 183
_save_video_to_archive() - line 206
_save_annotation_to_archive() - line 231

Issue 3: Timeout Handling ✅ FIXED

Problem: No timeouts on HTTP requests.
Solution: Added 60s timeout for images/text/audio, 120s for videos.

Issue 4: Detailed Error Logging ✅ FIXED

Problem: No visibility into what failed or why.
Solution: Added comprehensive logging with input IDs and error messages.

@github-actions
Copy link

Code Coverage

Package Line Rate Health
clarifai 45%
clarifai.cli 67%
clarifai.cli.templates 46%
clarifai.client 65%
clarifai.client.auth 67%
clarifai.constants 100%
clarifai.datasets 100%
clarifai.datasets.export 67%
clarifai.datasets.upload 75%
clarifai.datasets.upload.loaders 37%
clarifai.models 100%
clarifai.modules 0%
clarifai.rag 0%
clarifai.runners 53%
clarifai.runners.models 64%
clarifai.runners.pipeline_steps 41%
clarifai.runners.pipelines 77%
clarifai.runners.utils 62%
clarifai.runners.utils.data_types 72%
clarifai.schema 100%
clarifai.urls 60%
clarifai.utils 60%
clarifai.utils.evaluation 16%
clarifai.workflows 95%
Summary 61% (9969 / 16295)

Minimum allowed line rate is 50%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant