v0.4.33 gives us "zstd stream did not finish"

[syphar]

Hi,

we're using async-compression in docs.rs,

the recent release gives us plenty of new errors.

One example is db::delete::tests::test_delete_crate::true_expects

What do you need to debug this? Do you have pointers where I could look?

[NobodyXu]

This error is introduced in #412

async-compression/crates/compression-codecs/src/zstd/decoder.rs

Line 90 in 8947fed

"zstd stream did not finish",

It would occur when the zstd file is partially decompressed.

You might want to ignore this error, if this behavior is intentional

[syphar]

where exactly do you mean?

We're decompressing a file we downloaded?

[NobodyXu]

Hmm I don't know how you are using our library.

Is it directly through codec, or indirectly through tokio/futures-io bufread/write implementation?

[EliseZeroTwo]

We are also experiencing this since today. We're using the futures-io bufread/write implementation. I'll try to also look into this.

[NobodyXu]

Thanks! Since there was quite some refactoring in the past release, there could be bugs, but it also could be some issues with your reader/writer doesn't provide enough data of the archive.

You could try git bisect to see which commit is the offending one

[0x676e67]

We used tower-http and relied on the zstd compression component of async-compression, which resulted in the following test error.

#[tokio::test]
async fn test_zstd_empty_body() {
    let server = server::http(move |req| async move {
        assert_eq!(req.method(), "HEAD");

        http::Response::builder()
            .header("content-encoding", "zstd")
            .body(Default::default())
            .unwrap()
    });

    let res = wreq::head(format!("http://{}/zstd", server.addr()))
        .send()
        .await
        .unwrap();

    let body = res.text().await.unwrap();

    assert_eq!(body, "");
}

[0x676e67]

Currently, only the below sub-dependencies need to be downgraded to restore normal operation. I have tried my best to narrow down the scope of the problem.

• compression-core v0.4.30 -> v0.4.29
• compression-codecs v0.4.31 -> v0.4.30

cargo update -p compression-core --precise 0.4.29
cargo update -p compression-codecs --precise 0.4.30

[EliseZeroTwo]

We are also experiencing this since today. We're using the futures-io bufread/write implementation. I'll try to also look into this.

A part of our usage of the crate was not calling close on the encoder. Adding it to this path stopped the error from occuring!

[syphar]

This is also the problem in our case, shutdown (using tokio) wasn't called, so the archive was incomplete.

Trying to find out now if just the EOF was missing, or more :)

[NobodyXu]

It's good to hear that calling close fix it, sounds like our new code is catching bugs!

Though I'd still want confirmation on this

[ctron]

Running into the same issue. I think this version should be yanked as it is definitely breaking things.

[syphar]

Running into the same issue. I think this version should be yanked as it is definitely breaking things.

But that's the thing: it's only raising a pre-existing error that previously was silently ignored.

[ctron]

Ok, maybe I missed that. What exactly was the previous error?

The code (test) we have, and which now fails is:

• take content
• zstd encode
• write to file

Then:

• read file
• zstd decode
• compare with original

This does work (with previous versions), and provides the same content as stored. Now it fails with the error above.

[syphar]

As mentioned above, calling .close (or .shutdown for tokio) on the encoder was missing. Which leads to missing EOF markers at the end of the compressed file.

When you tried to decompress using other libraries, your compressed blob would have likely failed too