ISSUE-21583: httpfs: fall back to full GET when HEAD disallows or Range probe fails

[herbenderbler]

Fixes duckdb/duckdb#21583

Problem

Remote HTTPS URLs used with read_json / read_json_auto (and similar) go through httpfs’s file metadata path: HEAD, then—when HEAD is not 200 OK—a GET with Range: bytes=0-1 to infer size. That matches object storage and static files, but many REST/JSON APIs do not implement HEAD, return 405 / 501, or do not honor Range on JSON responses.

For https://api.openai.com/... with an http secret, users saw HTTP 0 Internal Server Error while the same URL worked with curl (no Range). A GET without the probe path (e.g. SET force_download=true) produced a normal HTTP result (e.g. 401 with a JSON error body), showing the bug was tied to httpfs’s HEAD / Range sequence, not the API being unreachable.

Solution

In HTTPFileHandle::LoadFileInfo():

• If HEAD returns 405 Method Not Allowed or 501 Not Implemented, skip the Range probe and set up the existing full download path (length == 0 → FullDownload in Initialize()), which issues a plain GET.
• If the Range probe fails (unexpected status) or GetRangeRequest throws HTTPException, fall back to that same full GET when auto_fallback_to_full_download is true (default), preserving prior strict behavior when that setting is false.

Replace the bare probe length 2 with a named constexpr (HTTP_RANGE_FILE_SIZE_PROBE_LENGTH).

Changes

• src/httpfs.cpp — LoadFileInfo() branches for 405/501, try/catch around the Range probe with fallback; anonymous-namespace constexpr for the probe size.
• test/sql/copy/test_remote_head_forbidden.test — clarify the test description (regression for REST-style URLs when HEAD or Range probing is unreliable).

Testing

Formatting

• Per CONTRIBUTING.md, use clang-format 11.0.1 on the touched C++ file, for example:
  • clang-format -i src/httpfs.cpp
• If your extension build exposes it, run make format-fix from the duckdb-httpfs tree the same way CI does.

Build (local httpfs + DuckDB)

From the duckdb-httpfs repository root (with the DuckDB submodule initialized):

git submodule update --init --recursive
GEN=ninja make release

That produces build/release/duckdb with httpfs (and bundled deps such as json) linked in. Adjust VCPKG_TOOLCHAIN_PATH if your environment uses vcpkg (see the repo README).

SQL logic tests

From the duckdb-httpfs root (after make release), run the relevant sqllogictest file:

./build/release/test/unittest test/sql/copy/test_remote_head_forbidden.test

That test hits a public HTTPS JSON endpoint (see the .test file). For a broader pass, run the full suite as in the duckdb-httpfs README:

./build/release/test/unittest

(S3/MinIO-heavy tests in this repo need Docker and extra setup per test/README.md; the test above does not.)

A/B check: upstream httpfs.cpp vs this branch

To confirm the behavior change comes from LoadFileInfo, rebuild duckdb twice:

1. Baseline (upstream LoadFileInfo): restore src/httpfs.cpp from main (or git checkout main -- src/httpfs.cpp), then cmake --build build/release --target duckdb -j (or a full make release).
2. This branch: check out your branch again and rebuild the same target.

Run the Manual verification SQL below for each binary. You should see HTTP 0 on the baseline and HTTP 401 on this branch (with a fake Bearer token).

Stock Python duckdb (optional cross-check)

On a release build with INSTALL httpfs, the pre-fix symptom matches HTTP 0; SET force_download=true before the same read_json_auto call typically yields a normal 401 response. After this PR ships in the extension repo, the default path should match that force_download behavior without needing the setting for this class of URL.

Manual verification (upstream httpfs vs this branch)

Using a build/release/duckdb built from duckdb-httpfs with the same fake Bearer secret:

CREATE SECRET openai (
  TYPE http,
  SCOPE 'https://api.openai.com',
  EXTRA_HTTP_HEADERS MAP {'Authorization': 'Bearer sk-fake'}
);
FROM read_json_auto('https://api.openai.com/v1/models');

Or as a one-liner:

./build/release/duckdb -c "
CREATE SECRET openai (TYPE http, SCOPE 'https://api.openai.com', EXTRA_HTTP_HEADERS MAP {'Authorization': 'Bearer sk-fake'});
FROM read_json_auto('https://api.openai.com/v1/models');
"

1. Upstream httpfs (pre-patch LoadFileInfo):
   HTTP Error: HTTP GET error on 'https://api.openai.com/v1/models' (HTTP 0 Internal Server Error) — matches the reported behavior.

2. This branch:
   HTTP Error: HTTP GET error on 'https://api.openai.com/v1/models' (HTTP 401) — a real HTTP status from OpenAI for an invalid key (same class of outcome as curl and as SET force_download=true on stock builds), not HTTP 0.

The query still errors with a fake key (expected); the fix is that the client no longer collapses to HTTP 0 during metadata / probe handling.