fix: use locale-aware collation for language name sorting#13141
fix: use locale-aware collation for language name sorting#13141okarachidera wants to merge 3 commits into
Conversation
Replace casefold()-only sort key with locale.strxfrm(name.casefold()) so that diacritics (Č, Ć, É, etc.) sort alongside their base characters rather than at the end of the alphabet. Adds _sort_key_name() helper with graceful fallback to casefold() if locale transformation fails (e.g. lone surrogates). Fixes internetarchive#11962
|
Thanks for opening this PR, @okarachidera — and welcome, this looks like your first contribution to Open Library! 🎉 @RayBB is assigned to this PR and currently has:
Note for reviewers: PR #12802 was opened earlier for the same issue (#11962) and is still open. Maintainers may want to compare both approaches before deciding which to move forward with. PR triage checklist (maintainers / Richy)
Note This comment was automatically generated by PAM, Open Library's Project AI Manager. PAM provides status visibility, performs basic project management functions, and gives actionable feedback so contributors aren't left waiting. — Richy, via PAM (Open Library's Project AI Manager) |
|
Thanks for the triage, @openlibrary-bot. I've looked at #12802 and believe this PR offers a simpler approach:
Happy to adjust if maintainers prefer the PyICU path in #12802. |
- Remove blank line between import groups - Use tuple in startswith() call - Add fmt: skip on multi-except clause to prevent formatter from producing Python-3-invalid comma-separated except syntax
2bda992 to
b302e9e
Compare
for more information, see https://pre-commit.ci
Summary
Fixes #11962: Language names are now sorted using locale-aware Unicode collation so that characters with diacritics (Č, Ć, É, etc.) sort alongside their base characters rather than at the end of the alphabet.
What was done
_sort_key_name(name)helper inopenlibrary/plugins/worksearch/languages.pythat useslocale.strxfrm(name.casefold())for Unicode-aware collation with graceful fallback tocasefold()when locale transformation failsget_top_languages()to use the new sort key when sorting by nameopenlibrary/plugins/worksearch/tests/test_language_sort.pywith 11 tests covering: case-insensitivity, diacritic ordering, mixed-language sorting, stability, empty strings, surrogate fallback, and locale-aware collationTesting
The sort key logic was verified locally:
Unit tests are included in the PR.
Before / After
Before:
.casefold()only — characters with diacritics sort at the end.After:
locale.strxfrm(.casefold())— Č sorts next to C, É sorts next to E, etc.Closes #11962