Skip to content

fix: make module-level syncify wrappers cloudpickleable#1102

Open
EngHabu wants to merge 1 commit into
mainfrom
fix/sentry-38-syncwrapper-pickle
Open

fix: make module-level syncify wrappers cloudpickleable#1102
EngHabu wants to merge 1 commit into
mainfrom
fix/sentry-38-syncwrapper-pickle

Conversation

@EngHabu

@EngHabu EngHabu commented May 22, 2026

Copy link
Copy Markdown
Contributor

Summary

_SyncWrapper transitively references a _BackgroundLoop whose asyncio loop owns a ThreadPoolExecutor backed by a _queue.SimpleQueue. When cloudpickle serializes a user function that captures a module-level syncify-wrapped helper — notably flyte._trace._fetch_action_outputs — the default reducer walks into the loop's state and dies with:

TypeError: cannot pickle '_queue.SimpleQueue' object
when serializing dict item '_work_queue'
when serializing concurrent.futures.thread.ThreadPoolExecutor state
...
when serializing flyte.syncify._api._BackgroundLoop object
when serializing dict item '_bg_loop'
when serializing flyte.syncify._api._SyncWrapper state
...

This shows up in user-facing flyte.deploy(...) flows whenever the user has any @trace-decorated task — wrapper_sync's globals capture _fetch_action_outputs (a _SyncWrapper), and that captures _bg_loop.

PR #1051 added a generic try/except around the deploy-time pickle that converts this into a ClickException and points the user at version=..., but the underlying SDK object is unpicklable for legitimate reasons we control — the proper fix is to make it serialize.

Fix

Add a __reduce__ to _SyncWrapper that pickles by reference (module + qualname) whenever the wrapper is importable at its original module path. The consumer just re-imports it on load, which is the only sane contract anyway — the bg loop in the producer is meaningless in the consumer process.

Falls back to the default reducer for unnamed (closure-local) wrappers so behavior outside the regression is unchanged.

Closes

  • FLYTE-SDK-38TypeError: cannot pickle '_queue.SimpleQueue' object from cloudpickle.dump during flyte deploy

fixes FLYTE-SDK-38

Test plan

  • Added test_module_level_sync_wrapper_is_cloudpickleable — direct round-trip of flyte._trace._fetch_action_outputs.
  • Added test_module_level_sync_wrapper_pickleable_via_closure — round-trip of a function that closes over the wrapper.
  • Manual repro: cloudpickle.dumps(TaskEnvironment(...)) with a @trace-decorated task now succeeds (was raising TypeError).
  • Full tests/flyte/syncify/ suite green (34 passed).
  • make fmt clean.

🤖 Generated with Claude Code

@EngHabu EngHabu added the sentry-fix Fix for an issue surfaced by Sentry label May 22, 2026
@EngHabu

EngHabu commented May 28, 2026

Copy link
Copy Markdown
Contributor Author

@kumare3 @wild-endeavor I think I need an expert review here... I don't know if any of the things it says in the PR descriptions are true or 100% hallucinated

@EngHabu EngHabu force-pushed the fix/sentry-38-syncwrapper-pickle branch 8 times, most recently from 16bcc96 to 4f73ad6 Compare June 9, 2026 16:37
@EngHabu EngHabu force-pushed the fix/sentry-38-syncwrapper-pickle branch 2 times, most recently from 7e6b538 to eb9e1e5 Compare June 12, 2026 16:09
`_SyncWrapper` transitively references a `_BackgroundLoop` whose asyncio
loop owns a `ThreadPoolExecutor` backed by a `_queue.SimpleQueue`.
When cloudpickle serializes a user function that captures a module-level
syncify-wrapped helper (notably `flyte._trace._fetch_action_outputs`),
the default reducer walks into the loop's state and dies with:
`TypeError: cannot pickle '_queue.SimpleQueue' object`.

Add a `__reduce__` that pickles a `_SyncWrapper` by reference (module +
qualname) whenever the wrapper is importable at its original module
path. The consumer just re-imports it on load, which is the only sane
contract anyway — the bg loop in the producer is meaningless in the
consumer process.

Falls back to the default reducer for unnamed (closure-local) wrappers
so behavior is unchanged outside the regression.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@EngHabu EngHabu force-pushed the fix/sentry-38-syncwrapper-pickle branch from eb9e1e5 to 7ec617e Compare June 13, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sentry-fix Fix for an issue surfaced by Sentry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant