feat: send cookies with every request during a clone - #58
Conversation
Add a --cookie flag to `kage clone` that attaches cookies to all four
request paths a clone makes, so a site behind a login or a region/cookie
wall can be mirrored:
- Chrome page navigations (seeded into the browser before first load)
- asset downloads
- the robots.txt fetch
- the sitemap.xml fetch
The flag takes a single name=value pair, repeats for several cookies, or
accepts a whole Cookie header in one value ("a=1; b=2") pasted straight
from devtools. Browser cookies are scoped to the seed host and its
subdomains; the HTTP fetches send the same header. Off-domain assets are
left on the live web by default, so the cookies are not leaked to
third-party hosts.
Tests cover the header serialisation, the downloader/sitemap sending the
cookie, the Chrome cookie-param building, and the CLI flag parsing
(including malformed input). README, CLI reference, and CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
No one process? |
|
The current HTTP path can send the supplied session cookie outside the seed origin.
Sitemaps are a second path: robots.txt may advertise a cross-origin sitemap, and a sitemap index may point to children on another host. I think the HTTP cookie needs to be selected per request URL, with the safest default being the exact seed host. Requests to other hosts should omit it unless the user explicitly supplied cookie domain semantics. Tests should cover |
What
Adds a
--cookieflag tokage cloneso a site behind a login or a region/cookie wall can be mirrored. The cookies ride on every request a clone makes, not just the page render:Cookieheader on the HTTP fetchrobots.txtfetchsitemap.xmlfetchThis keeps the authenticated session consistent across the whole run rather than only the rendered pages.
Usage
Notes
--cookieaccepts a singlename=value, repeats, or a full header in one value; malformed input is a clear usage error.Tests
Added coverage for:
Config.CookieHeaderserialisation (incl. empty-name skipping)CookieheaderfetchSitemapsending the cookieparseCookies, including malformed-input errors (theclipackage had no tests before)go build ./...,go vet ./...,gofmt, and the fullgo test ./...all pass.README, CLI reference (
docs/content/reference/cli.md), and CHANGELOG (Unreleased) updated.🤖 Generated with Claude Code
Closes #9