Filter the normalized query in strict check_url path filtering

[gaoflow]

In check_url, strict content filtering ran path_filter against the raw query string:

if strict and path_filter(parsed_url.path, parsed_url.query) is False:

but the query the function ultimately returns has been normalized (tracker parameters and non-whitelisted keys stripped). So an index-style path could survive strict filtering only because of a query parameter that normalization then removes, leaving check_url to accept a URL whose own cleaned output it would reject.

That breaks idempotence: check_url(check_url(url, strict=True)[0], strict=True) did not equal check_url(url, strict=True). This runs path_filter against clean_query(parsed_url.query, strict, language) (the query as it will actually survive), so the strict decision matches the returned URL.

test_path_filter was extended to assert the round-trip is stable for tracker params, non-whitelisted keys, and query-stripped index paths. ruff check, ruff format --check, and mypy are clean.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (61452a4) to head (affe6cf).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #140   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        12    +1     
  Lines          853       863   +10     
  Branches       165       167    +2     
=========================================
+ Hits           853       863   +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[adbar]

@gaoflow Thanks! Since we're at it we could tackle other similar issues here. It is a broader pattern, several filters run on the raw input before normalize_url transforms it, so anything normalization changes can change the result.

[gaoflow]

Agreed, it's the same shape as this fix. Mapping check_url, the filters that run on the pre-normalize_url value are type_filter (scrubbed url), lang_filter (scrubbed url), extension_filter(parsed_url.path) and domain_filter(parsed_url.netloc) — each tests a component in a form normalize_url will later change:

• extension_filter tests the raw path, but normalize_url runs normalize_part (percent-encoding normalization, //→/, /../ removal). So a disguised extension like /file%2Eexe is only revealed as .exe after normalization — the filter can pass a URL that normalizes into something it would have rejected.
• domain_filter validates the raw netloc, but normalize_url lowercases + punycode-decodes + strips the default port — so it validates a different host string than the one emitted (case / xn-- vs unicode / :80).

The clean generalization of this PR is to feed each filter the value that survives normalization (as I did for the query via clean_query). But I don't think a blanket "normalize everything first, then filter" is safe: type_filter looks like it deliberately inspects the raw url to catch spam that normalization would hide, so moving it after normalization could weaken it.

Two questions so I scope this right:

1. Which do you want to cover — I'd prioritize extension_filter (normalized path) and domain_filter (normalized netloc), and leave type_filter on raw input by design?
2. Fold them into this PR, or land this one and open a follow-up per filter?

[adbar]

Yes, you got my point. I think you could address extension_filter and domain_filter in this PR.

[gaoflow]

Done — extended in affe6cf. I pulled normalize_authority() and normalize_path() out of normalize_url() (pure extraction, behaviour unchanged) and fed them to the two filters, mirroring the clean_query() treatment of the query.

One honest note on scope, since I checked both before wiring them up:

• domain_filter has a real, reproducible case. An IP host with an explicit default port was being dropped: check_url("http://1.2.3.4:80/page.html") returned None, even though it normalizes to http://1.2.3.4/. domain_filter saw 1.2.3.4:80, failed the IP parse (the port is attached), then hit the split(".")[0].isdigit() guard and rejected it — whereas the bare 1.2.3.4 passes. Feeding it the normalized (port-stripped, lower-cased, punycode-decoded) authority fixes it. A non-default port like :8080 is preserved by normalization, so it's still filtered out, which is correct. Added to test_urlcheck_port.

• extension_filter turned out to be a no-op in practice. I couldn't find any input where it differs on the raw vs normalized path — including the /file%2Eexe shape from earlier: normalize_part() is quote(..., safe=...), so it never decodes %2E into ., and the structural rewrites (// → /, leading /../ removal) don't change the trailing extension. I still routed it through normalize_path() for consistency (it's the value that will actually be emitted, and it's future-proof against normalize_part changes), but I didn't want to imply it fixes a live case — happy to drop that one line if you'd rather keep the change strictly to domain_filter.

type_filter/lang_filter left on the raw input as discussed. ruff, mypy and the test suite are green.