Add per-request cancellation hook to HTTPFSCurlClient

[rustyconover]

Summary

Adds an optional should_cancel std::function<bool()> field to HTTPFSParams. When set, it is wired into libcurl as a CURLOPT_XFERINFOFUNCTION progress callback that returns non-zero to abort the in-flight transfer with CURLE_ABORTED_BY_CALLBACK.

Motivation

Extensions doing slow HTTP work from inside a query (remote-function execution, long S3 fetches, etc.) currently have no way to honor a user Ctrl-C while a request is in flight. libcurl is run with CURLOPT_NOSIGNAL=1 (correctly, for thread-safety in DuckDB's multi-threaded executor), so SIGINT does not unblock the underlying poll()/recv() syscall. There is no in-band cancel primitive between the extension and libcurl, so a stuck endpoint hangs the query until CURLOPT_TIMEOUT fires (or forever, if no timeout is set).

I hit this concretely while debugging a hung VGI extension query: the main thread was blocked in curl_easy_perform → multi_wait → poll, completely unresponsive to Ctrl-C. There's no clean fix possible at the extension layer alone — extensions can't add curl options to clients owned by httpfs. The interrupt primitive has to live in httpfs.

This patch is modeled on libcurl's own blessed pattern for cooperative cancellation (XFERINFOFUNCTION returning non-zero), which is the standard way to abort an in-flight transfer from another context without removing CURLOPT_NOSIGNAL.

Behavior

• Default (no hook): zero behavior change. Empty std::function makes the callback return 0 cheaply on every progress tick.
• Hook returning true: libcurl aborts the transfer; the resulting HTTPResponse has status set to 499 ("Client Closed Request", matching nginx's convention) and request_error = \"Request cancelled by callback\".
• Throwing hook: treated as cancel (libcurl callbacks cannot propagate C++ exceptions across the C boundary; throwing would crash the process).

Caller pattern

auto params = http_util.InitializeParameters(context, url);
auto &httpfs_params = params->Cast<HTTPFSParams>();
httpfs_params.should_cancel = [&context]() {
    return context.interrupted.load();
};

PostRequestInfo post(url, headers, *params, body.data(), body.size());
auto response = http_util.Request(post);
if (response->status == static_cast<HTTPStatusCode>(499)) {
    throw InterruptException(\"Request cancelled\");
}

Implementation notes

• The hook is stored as a member on HTTPFSCurlClient so its address remains stable across cached-client reuse (Initialize() is called again with new params on each request, but &cancel_fn doesn't move).
• The progress callback is installed unconditionally on every Initialize(), even when the hook is empty — the empty-std::function check inside the callback is a single branch and cheaper than conditionally setopt-ing per request.
• Field is appended to the end of HTTPFSParams to preserve the existing layout expected by duckdb-wasm.
• WASM client (httpfs_client_wasm.cpp) is unchanged; cancellation there is a separate concern (emscripten_fetch / sync XHR have different abort primitives).

Latency to cancel

Bounded by libcurl's progress-callback cadence — typically every few hundred ms during active I/O, less often during long DNS resolves or TLS handshakes. In practice, well under one second for the common cases. Setting a reasonable CURLOPT_CONNECTTIMEOUT complements this for the connect-phase edge case.

Follow-up

A small duckdb-core PR could add Cancelled_499 = 499 to the HTTPStatusCode enum so callers don't need to cast through the integer value:

// duckdb/common/enums/http_status_code.hpp
Cancelled_499 = 499,  // Client Closed Request (nginx convention)

Once that lands, the cast in TransformResponseCurl can be replaced with the named constant.

Test plan

• CI build passes on all platforms
• Existing httpfs tests pass (no behavior change for callers that don't set the hook)
• Manual: extension with cancel hook returning true after 100ms aborts an in-flight POST and surfaces status 499

🤖 Generated with Claude Code