Skip to content

feat: send cookies with every request during a clone - #58

Open
didimitrie wants to merge 1 commit into
tamnd:mainfrom
didimitrie:feat/request-cookies
Open

feat: send cookies with every request during a clone#58
didimitrie wants to merge 1 commit into
tamnd:mainfrom
didimitrie:feat/request-cookies

Conversation

@didimitrie

@didimitrie didimitrie commented Jun 22, 2026

Copy link
Copy Markdown

What

Adds a --cookie flag to kage clone so a site behind a login or a region/cookie wall can be mirrored. The cookies ride on every request a clone makes, not just the page render:

  • Chrome page navigations — seeded into the browser before the first load
  • Asset downloadsCookie header on the HTTP fetch
  • robots.txt fetch
  • sitemap.xml fetch

This keeps the authenticated session consistent across the whole run rather than only the rendered pages.

Usage

# single cookie
kage clone --cookie "session=abc" example.com

# repeat the flag
kage clone --cookie "session=abc" --cookie "theme=dark" example.com

# paste a whole Cookie header straight from devtools
kage clone --cookie "session=abc; theme=dark" example.com

Notes

  • Browser cookies are scoped to the seed host and its subdomains; the HTTP fetches send the same header.
  • Because off-domain assets are left on the live web by default, the cookies are not leaked to third-party hosts (a nice security property that falls out of the existing asset policy).
  • --cookie accepts a single name=value, repeats, or a full header in one value; malformed input is a clear usage error.

Tests

Added coverage for:

  • Config.CookieHeader serialisation (incl. empty-name skipping)
  • the asset downloader sending / omitting the Cookie header
  • fetchSitemap sending the cookie
  • the Chrome cookie-param building
  • CLI parseCookies, including malformed-input errors (the cli package had no tests before)

go build ./..., go vet ./..., gofmt, and the full go test ./... all pass.

README, CLI reference (docs/content/reference/cli.md), and CHANGELOG (Unreleased) updated.

🤖 Generated with Claude Code

Closes #9

Add a --cookie flag to `kage clone` that attaches cookies to all four
request paths a clone makes, so a site behind a login or a region/cookie
wall can be mirrored:

  - Chrome page navigations (seeded into the browser before first load)
  - asset downloads
  - the robots.txt fetch
  - the sitemap.xml fetch

The flag takes a single name=value pair, repeats for several cookies, or
accepts a whole Cookie header in one value ("a=1; b=2") pasted straight
from devtools. Browser cookies are scoped to the seed host and its
subdomains; the HTTP fetches send the same header. Off-domain assets are
left on the live web by default, so the cookies are not leaked to
third-party hosts.

Tests cover the header serialisation, the downloader/sitemap sending the
cookie, the Chrome cookie-param building, and the CLI flag parsing
(including malformed input). README, CLI reference, and CHANGELOG updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zhaobokai341

Copy link
Copy Markdown

No one process?

@SihanTeng

Copy link
Copy Markdown

The current HTTP path can send the supplied session cookie outside the seed origin.

asset.Downloader.try adds the raw Cookie header to every URL it receives. The default asset policy reduces the exposure, but --all-asset-hosts deliberately downloads third-party assets, so using that flag with --cookie sends the seed credentials to those hosts. Same-registrable-domain asset hosts also receive the header even though the cookie was supplied for the seed host.

Sitemaps are a second path: robots.txt may advertise a cross-origin sitemap, and a sitemap index may point to children on another host. collectSitemaps follows those URLs and fetchSitemap attaches the same cookie without checking their origin.

I think the HTTP cookie needs to be selected per request URL, with the safest default being the exact seed host. Requests to other hosts should omit it unless the user explicitly supplied cookie domain semantics. Tests should cover --all-asset-hosts, an external robots sitemap, and an external child sitemap. The Chrome cookie jar can remain scoped independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cookies?

3 participants