Skip to content

i18n: keep messages.pot location comments on one line to prevent merge conflicts#12903

Open
lokesh wants to merge 8 commits into
internetarchive:masterfrom
lokesh:12837/refactor/pot-disable-line-wrapping
Open

i18n: keep messages.pot location comments on one line to prevent merge conflicts#12903
lokesh wants to merge 8 commits into
internetarchive:masterfrom
lokesh:12837/refactor/pot-disable-line-wrapping

Conversation

@lokesh

@lokesh lokesh commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Closes #12837

Refactor. Stops messages.pot from generating spurious merge conflicts between unrelated PRs.

The problem, in plain terms

openlibrary/i18n/messages.pot is a generated file we commit. Every translatable string carries a #: "location comment" listing the files that use it:

#: search/authors.html search/lists.html search/subjects.html work_search.html
msgid "Checking Search Inside matches"
msgstr ""

Babel wraps those lines at 76 characters. So when a PR adds one file to a string, the line tips over the limit and Babel re-flows the whole block — rewriting filenames that didn't actually change:

-#: search/authors.html search/lists.html search/subjects.html work_search.html
+#: search/authors.html search/editions.html search/lists.html
+#: search/subjects.html work_search.html

subjects.html and work_search.html got pushed onto a new line even though nothing about them changed. Now if a second in-flight PR also touches one of those templates, both branches have rewritten the same physical lines → git reports a conflict, even though the two PRs added completely unrelated files. The wrapping is what turns a non-overlapping change into an overlapping one.

What this PR does

Write the file with a very large width, which disables Babel's line wrapping. Each string's #: location comment now stays on a single line, so adding a file just extends that one line:

-#: search/authors.html search/lists.html search/subjects.html work_search.html
+#: search/authors.html search/editions.html search/lists.html search/subjects.html work_search.html

Unrelated string changes no longer share any lines, so they stop conflicting.

write_po(f, catalog, include_lineno=False, width=POT_WIDTH)

That's the whole change. No post-processing pass, no custom string-munging to maintain.

Why not width=0

The issue suggested write_po(..., width=0). It does the opposite of what we want. Babel deliberately copies xgettext's behaviour — it always wraps comments, even when wrapping is otherwise off. From Babel's source:

# xgettext always wraps comments even if --no-wrap is passed;
# provide the same behaviour
comment_width = width if width and width > 0 else 76

So width=0 leaves the #: location comments wrapped at 76 (the actual problem — untouched) and only unwraps the msgid/msgstr text. A large positive width, on the other hand, raises comment_width too — so the locations finally stop wrapping. That's the lever.

About the message text

A large width also single-lines the msgid/msgstr text (Babel's one width knob governs both). For a generated template that's a wash, and arguably better:

  • A string's text only changes when the English itself changes — rare, and localized to that one string. The churn that caused conflicts was always the location lists, which change constantly as templates move around.
  • A one-line msgid can't reflow its neighbours either, so it carries the same conflict-resistance the locations now have.

Note on history: an earlier version of this PR kept message text wrapped via a custom _unwrap_location_comments post-processing pass. That was replaced with the one-line width= approach after confirming (with a full-catalog diff) that the two produce byte-identical location output — the only difference is message text, which is cosmetic for a generated file. Fewer moving parts, no helper to maintain. Thanks to @mekarpeles for prompting the look at other comment types, which is what surfaced that the simpler lever was sufficient.

For developers

  • One fewer source of "why is messages.pot conflicting again?" when rebasing or merging concurrent PRs that touch templates.
  • The location breadcrumbs (which templates use this string) are kept — we didn't drop them to solve this.
  • Heads-up: this PR's first regeneration reflows essentially the entire file (every wrapped line collapses to one), so the diff is large but boring — it's a one-time formatting pass, not a content change. After this lands, the file stays stable. Best merged when no other .pot-touching PRs are mid-flight.

For translators

  • Nothing changes in what gets translated. The regenerated .pot has the exact same set of msgids and the exact same locations; only line wrapping differs. No string was added, removed, or altered.
  • The #: "where is this string used" hints translators rely on for context are preserved.

Technical

  • openlibrary/i18n/__init__.pyextract_messages now calls write_po(f, catalog, include_lineno=False, width=POT_WIDTH) where POT_WIDTH = 1_000_000 (a comment by the constant records why width=0 doesn't work and why a large width does). The previous _unwrap_location_comments helper and its buffer round-trip are removed.
  • Other gettext comment types — #, flags (e.g. #, fuzzy, #, python-format), #. translator notes, #| previous-msgid — are written by Babel exactly as before; nothing custom touches them. (fuzzy in particular can't appear in a .pot template anyway — it's a flag on a translation in .po files.)

Testing

  • Updated unit tests in openlibrary/tests/core/test_i18n.py (Test_pot_width): generate a catalog through the real write_po(..., width=POT_WIDTH) path and assert a multi-file string's locations stay on a single #: line, and that flags (#, python-format) and message text survive untouched.
  • Regenerated messages.pot with the actual extraction script and verified via read_po that it round-trips to an identical set of msgids and per-string locations vs. the previous version (semantic no-op): 1953 msgids, all locations identical.
  • Confirmed zero wrapped #: continuation lines remain in the regenerated file.
  • ruff lint + format pass on the changed Python files.

Possible follow-ups (out of scope)

  • One #: line per file would make even two PRs editing the same string's location list merge cleanly. It's the most conflict-proof but unconventional and makes the file taller; this PR keeps the conventional space-joined single line.
  • A .gitattributes merge driver that regenerates messages.pot on conflict, or having CI own the file outright (both noted in the issue).

Stakeholders

@cdrini (i18n lead)

Babel wraps the #: location comments at 76 chars, so adding or removing one
file from a string's location list re-flows the whole block and rewrites
filenames that didn't change. That turns unrelated PRs into overlapping diffs
and causes spurious messages.pot merge conflicts (internetarchive#12837).

The issue proposed write_po(..., width=0), but Babel mirrors xgettext and
always wraps comments regardless of width, so width=0 only unwraps msgid text
(the downside) without unwrapping the location comments (the actual fix).

Instead, post-process write_po's output to collapse each #: block onto a
single line. Adding a file now just extends that line; message text stays
wrapped and readable. This first regeneration reflows the whole file as a
one-time change.

Closes internetarchive#12837
@lokesh
lokesh marked this pull request as ready for review June 11, 2026 23:24
@lokesh
lokesh marked this pull request as draft June 11, 2026 23:25
@lokesh
lokesh marked this pull request as ready for review June 15, 2026 18:01
@mekarpeles mekarpeles self-assigned this Jun 15, 2026
@mekarpeles

Copy link
Copy Markdown
Member

Important: Are there other types of comments (like fuzzy) that might break?

lokesh added 2 commits June 19, 2026 23:57
…t-disable-line-wrapping

# Conflicts:
#	openlibrary/i18n/messages.pot
@lokesh
lokesh requested a review from cdrini June 26, 2026 18:02
Replace the _unwrap_location_comments post-processing pass with
write_po(..., width=POT_WIDTH). A large width disables Babel's line
wrapping entirely, so each #: location comment stays on a single line —
the same conflict fix, with no custom string-munging code to maintain.

This also single-lines the msgid/msgstr text. For a generated template
that is cosmetic and, if anything, more merge-friendly: a string's text
only changes when the English itself changes (rare and localized), and a
one-line msgid can't reflow its neighbours either.

width=0 still does not work here (Babel mirrors xgettext and always wraps
comments at 76 regardless), so a large width is the way to turn wrapping
off; the reasoning is captured in a comment by POT_WIDTH.

Tests now assert the real generation path (write_po at POT_WIDTH) keeps
locations on one line and leaves flags/message text intact, replacing the
unit tests for the deleted helper.
@lokesh

lokesh commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator Author

Good question — and digging into it is actually what led me to simplify the whole PR.

Short answer: no, nothing else breaks.

  • fuzzy is a flag on a translation (#, fuzzy) and lives in .po files. messages.pot is a template with empty msgstrs — there are no fuzzy flags in it to begin with.
  • The other gettext comment types — #, flags (incl. #, python-format), #. translator notes, #| previous-msgid — are all written by Babel exactly as before. The earlier version of this PR had a custom #:-only post-processing pass; your question prompted me to double-check it never touched those... and in doing so I realized the post-processing wasn't needed at all.

I've replaced _unwrap_location_comments with a one-liner: write_po(..., width=POT_WIDTH). A large width just turns Babel's wrapping off, so the #: locations stay on one line with no custom code touching any comment type. I confirmed against the full catalog that it produces byte-identical location output to the old approach (the only difference is message text, which is cosmetic for a generated file), and added a test asserting flags + message text survive. Updated the description with the details.

@lokesh lokesh added translations Needs: Review This issue/PR needs to be reviewed in order to be closed or merged (see comments). [managed] labels Jun 26, 2026
@lokesh

lokesh commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

@cdrini heads-up — I checked this against your JS i18n experiment (#12968) and there's no directional conflict: that PR changes what gets extracted, this one only changes how the .pot is serialized. Your __init__.py edits (METHODS list) are in a different part of extract_messages than the write_po call here, and the messages.pot overlap resolves itself with a regeneration on rebase.

If anything this helps #12968: adding JS extraction appends new files to lots of existing strings' location lists — exactly the reflow churn this PR eliminates.

@lokesh

lokesh commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

The issue is present on testing right now:

image

lokesh and others added 2 commits July 17, 2026 23:58
…t-disable-line-wrapping

# Conflicts:
#	openlibrary/i18n/messages.pot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs: Review This issue/PR needs to be reviewed in order to be closed or merged (see comments). [managed] translations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

i18n: prevent messages.pot merge conflicts by disabling location-comment line wrapping

2 participants