Description
Encountering a Forbidden error from defense.gov due to missing or improper HTTP headers.
Steps to Reproduce
- Run mildew with its standard configuration to scrape DoD directories.
- Observe the
Forbidden error when the target domain rejects the request.
Expected Behavior
Scraping should proceed successfully with valid HTTP headers set (e.g., a proper User-Agent or other relevant headers).
Additional Context
It appears that defense.gov requires specific HTTP headers or a valid User-Agent string. Adding appropriate headers may resolve the issue.
PR incoming
Description
Encountering a
Forbiddenerror fromdefense.govdue to missing or improper HTTP headers.Steps to Reproduce
Forbiddenerror when the target domain rejects the request.Expected Behavior
Scraping should proceed successfully with valid HTTP headers set (e.g., a proper User-Agent or other relevant headers).
Additional Context
It appears that
defense.govrequires specific HTTP headers or a valid User-Agent string. Adding appropriate headers may resolve the issue.PR incoming