Replace zstandard with backports.zstd / stdlib compression.zstd (#577) - #877
Conversation
|
Hi @pferate thanks for this! I've just merged a huge backend refactor. Public API hasn't changed, but it's created some conflicts. I'll give this a review this week. |
|
Hi @joe-clickhouse Appreciate it. I just submitted another PR (#887) about 20 minutes after your big changed got merged in, and saw the conflicts. I'll work on rebasing and resolving the conflicts in both, and updating the PRs. |
…kHouse#577) Python 3.14 added compression.zstd to the standard library. backports.zstd provides the identical API for Python 3.10–3.13, so a single call surface now works across all supported versions.
|
Hi @joe-clickhouse I've updated this PR to resolve the conflicts. It should be ready for review at your convenience. |
joe-clickhouse
left a comment
There was a problem hiding this comment.
Thanks @pferate! Nice work here. I hope you don't mind but I pushed a few additions on top of your commits:
- PEP 784 makes
libzstdoptional at CPython build time. Likely rare, but it'd be possible for a 3.14+ interpreter built without it to not havecompression.zstdand the unconditional import would have madeimport clickhouse_connectfail entirely. The driver now imports cleanly on these kinds of builds, drops zstd from the advertised compression methods, and raises an error only ifzstdis then explicitly requested. Standard installs still work as usual. - To fix a typing issue I made the adapter functions private.
zstd_compressand friends became_zstd_compressetc. They are internal and the unannotatedzstd_decompressorreturn was failing in the public type completeness in CI. - I pinned
backports.zstd>=1.3.0. This version guarantees compatibility with modern packaging tools and the standard library's zstd backport without causing potential import collisions.
I also added some unit tests for the adapter and the unavailable path, and merged in latest main. Appreciate the contribution! This closes out the last real gap from #577.
Python 3.14 added compression.zstd to the standard library. backports.zstd provides the identical API for Python 3.10–3.13, so a single call surface now works across all supported versions.
Summary
Replace the zstandard dependency throughout the driver (httputil, asyncclient, streaming, compression) with thin wrappers in compression.py that dispatch to compression.zstd (3.14+) or backports.zstd (<3.14). The zstandard package is removed from install_requires and test requirements entirely for Python 3.14+, and replaced by backports.zstd for older versions.
Closes #577
Checklist
Delete items not relevant to your PR: