Skip to content

Replace zstandard with backports.zstd / stdlib compression.zstd (#577) - #877

Merged
joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
pferate:577-zstd-update
Jul 22, 2026
Merged

Replace zstandard with backports.zstd / stdlib compression.zstd (#577)#877
joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
pferate:577-zstd-update

Conversation

@pferate

@pferate pferate commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Python 3.14 added compression.zstd to the standard library. backports.zstd provides the identical API for Python 3.10–3.13, so a single call surface now works across all supported versions.

Summary

Replace the zstandard dependency throughout the driver (httputil, asyncclient, streaming, compression) with thin wrappers in compression.py that dispatch to compression.zstd (3.14+) or backports.zstd (<3.14). The zstandard package is removed from install_requires and test requirements entirely for Python 3.14+, and replaced by backports.zstd for older versions.

Closes #577

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

@CLAassistant

CLAassistant commented Jul 17, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@joe-clickhouse

Copy link
Copy Markdown
Contributor

Hi @pferate thanks for this! I've just merged a huge backend refactor. Public API hasn't changed, but it's created some conflicts. I'll give this a review this week.

@pferate

pferate commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Hi @joe-clickhouse Appreciate it. I just submitted another PR (#887) about 20 minutes after your big changed got merged in, and saw the conflicts.

I'll work on rebasing and resolving the conflicts in both, and updating the PRs.

…kHouse#577)

Python 3.14 added compression.zstd to the standard library.
backports.zstd provides the identical API for Python 3.10–3.13, so a
single call surface now works across all supported versions.
@pferate

pferate commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Hi @joe-clickhouse I've updated this PR to resolve the conflicts. It should be ready for review at your convenience.

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pferate! Nice work here. I hope you don't mind but I pushed a few additions on top of your commits:

  1. PEP 784 makes libzstd optional at CPython build time. Likely rare, but it'd be possible for a 3.14+ interpreter built without it to not have compression.zstd and the unconditional import would have made import clickhouse_connect fail entirely. The driver now imports cleanly on these kinds of builds, drops zstd from the advertised compression methods, and raises an error only if zstd is then explicitly requested. Standard installs still work as usual.
  2. To fix a typing issue I made the adapter functions private. zstd_compress and friends became _zstd_compress etc. They are internal and the unannotated zstd_decompressor return was failing in the public type completeness in CI.
  3. I pinned backports.zstd>=1.3.0. This version guarantees compatibility with modern packaging tools and the standard library's zstd backport without causing potential import collisions.

I also added some unit tests for the adapter and the unavailable path, and merged in latest main. Appreciate the contribution! This closes out the last real gap from #577.

@joe-clickhouse
joe-clickhouse merged commit 98371b7 into ClickHouse:main Jul 22, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exclude zstandard from Python >= 3.14

3 participants