FilePut: SignatureDoesNotMatch for files larger than 1 GiB

[mst-mkt]

Summary

FilePut fails with 403 SignatureDoesNotMatch for any file larger than 1 GiB. Files of 1 GiB or smaller are unaffected.

What I found

The following seems to fill the hash buffer for x-amz-content-sha256 with a single Read, discarding the returned count:

simples3/object.go

Lines 159 to 160 in 57fa0ec

	content := make([]byte, fSize)
	_, err = u.Body.Read(content)

io.Reader permits a short read, and on linux/amd64 a single Read on an *os.File appears to be capped at 1 GiB (maxRW = 1 << 30 in internal/poll/fd_unix.go), so content probably ends up as the first 1 GiB plus zero padding. The request body itself is re-read from offset 0 and uploads correctly — in my testing only x-amz-content-sha256 was wrong.

For what it's worth, replacing the Read with io.ReadFull made every case below pass locally.

Reproduction

No real S3 endpoint is needed: a local test server compares the x-amz-content-sha256 header with the hash of the body it actually received. Running this needs about 1.5 GiB of free memory, since FilePut allocates a buffer the size of the object, and a filesystem that supports sparse files.

package main

import (
    "crypto/sha256"
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
    "os"

    "github.com/rhnvrm/simples3"
)

func main() {
    testCases := []struct {
        name string
        size int64
    }{
        {"1 MiB", 1 << 20},
        {"1 GiB (boundary)", 1 << 30},
        {"1 GiB + 1", 1<<30 + 1},
        {"1.5 GiB", 1536 << 20},
    }

    var headerHash, bodyHash string
    ts := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
        h := sha256.New()
        io.Copy(h, r.Body)
        headerHash, bodyHash = r.Header.Get("x-amz-content-sha256"), fmt.Sprintf("%x", h.Sum(nil))
    }))
    defer ts.Close()

    s3 := simples3.New("us-east-1", "id", "secret")
    s3.SetEndpoint(ts.URL)

    fmt.Printf("%-20s %-12s %s\n", "case", "size", "header == body?")
    for _, tc := range testCases {
        headerHash, bodyHash = "", ""

        f, _ := os.CreateTemp("", "repro") // errors omitted for brevity
        f.WriteAt([]byte{0xFF}, tc.size-1) // extends the file sparsely: zeros + non-zero last byte

        if _, err := s3.FilePut(simples3.UploadInput{Bucket: "b", ObjectKey: "k", Body: f}); err != nil {
            fmt.Println("upload failed:", err)
        }

        verdict := "OK"
        if headerHash == "" || headerHash != bodyHash {
            verdict = "MISMATCH -> SignatureDoesNotMatch"
        }
        fmt.Printf("%-20s %-12d %s\n", tc.name, tc.size, verdict)

        f.Close()
        os.Remove(f.Name())
    }
}

Expected: OK for every case.

Actual:

case                 size         header == body?
1 MiB                1048576      OK
1 GiB (boundary)     1073741824   OK
1 GiB + 1            1073741825   MISMATCH -> SignatureDoesNotMatch
1.5 GiB              1610612736   MISMATCH -> SignatureDoesNotMatch

Environment

• simples3: v0.11.1, and the code is unchanged on current master
• Go: 1.26.4
• Platform: linux/amd64
• Endpoint: Cloudflare R2 in production, plus the local server above. I have not tested against AWS S3 itself.

Possibly related: #46 also reports SignatureDoesNotMatch with S3-compatible providers; if the objects involved were larger than 1 GiB, it may share this cause. not the case: comment

[ptman]

@mst-mkt thank you for your suggestion, but my objects are small

[mst-mkt]

@ptman Thanks for confirming! Sounds like #46 is unrelated to this issue then. I've struck out that speculation from the issue description.