fix(x-ui): fix OOM crash in generate_subscription_url_mapping for large deployments

[phoseinq]

Problem

On large x-ui deployments (10k+ users), generate_subscription_url_mapping.py is killed by the OOM killer (exit code 137 / SIGKILL) and produces no output file. The script appears to start (Fetching users from x-ui...) but then silently dies.

Root cause

The query does a LEFT JOIN between client_traffics and inbounds:

SELECT ct.id, ct.email, ct.inbound_id, i.settings AS inbound_settings
FROM client_traffics ct
LEFT JOIN inbounds i ON ct.inbound_id = i.id

The inbounds.settings column contains a JSON blob that embeds all clients for that inbound (can be 3,500+ clients, ~7 MB per inbound). Joining this into client_traffics and calling fetchall() duplicates the full blob for every user row.

Example: 10,654 users × 3 inbounds × ~7 MB per inbound ≈ ~75 GB resident memory → OOM kill before any output is written.

Fix

Pre-load an email → subId dict by reading each inbound's settings JSON once (O(inbounds) passes), then query client_traffics without the LEFT JOIN and do an O(1) dict lookup per user.

Memory usage drops from O(users × inbound_json_size) to O(inbounds × inbound_json_size + users).

The existing extract_subscription_token_from_inbound() helper is no longer called in the hot path (it is still used by the per-row branch for deployments that store the token directly on the client_traffics row).

Test results

Tested on a real production deployment:

	Before	After
Users	10,654	10,654
Inbounds	3	3
RAM	2 GB (no swap)	2 GB (no swap)
Result	OOM killed after ~5 s, no output	Completes in ~3 s, 10,652 mappings generated

Summary by CodeRabbit

• Refactor
  • Improved subscription URL mapping generation process for better performance.

[coderabbitai]

Warning

Review limit reached

@phoseinq, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 50 minutes and 50 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ae356474-4f12-4694-a102-bb578a323076

📥 Commits

Reviewing files that changed from the base of the PR and between 28fa498 and 8928f2d.

📒 Files selected for processing (1)

• x-ui/migration/generate_subscription_url_mapping.py

Walkthrough

The pull request optimizes subscription URL mapping generation by pre-loading subscription tokens into a lookup map once per inbound, removing the per-row inbounds table join and inbound_settings JSON parsing, replacing it with a fallback map-based lookup during user processing.

Changes

Subscription URL Mapping Optimization

Layer / File(s)	Summary
Pre-load subscription map and simplify query x-ui/migration/generate_subscription_url_mapping.py	New pre-load step builds email_to_subid map by scanning inbounds once and parsing settings JSON per inbound. Main x-ui user query updated to select lightweight columns without LEFT JOIN inbounds, eliminating per-row join overhead.
Token extraction fallback lookup x-ui/migration/generate_subscription_url_mapping.py	Removes per-row inbound_settings retrieval and replaces per-user extraction call with fallback lookup into the pre-loaded map when subscription_token is not present on the user row.

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 Once queries joined with care,
Now maps hold tokens in the air,
Per-row parsing gone away,
Pre-load happens just one day,
Lighter queries lead the way!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main fix: resolving an OOM crash in the generate_subscription_url_mapping function for large deployments, which directly aligns with the primary changeset objective.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@x-ui/migration/generate_subscription_url_mapping.py`:
- Around line 228-254: The email_to_subid map currently keys only by email
causing later inbounds to overwrite earlier subIds; change email_to_subid to use
a composite key (inbound_id, email) when populating it (use ib_row['id'] as
inbound_id alongside client_email and sub_id) and then update the lookup site
(where client_traffics rows are matched — the code that currently looks up by
client_email, referenced as email_to_subid[client_email] around the later
lookup) to use the same (inbound_id, email) tuple key so each inbound's client
entry is preserved and looked up correctly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ecb32dd6-5974-46d8-9157-ba674bf8e2d1

📥 Commits

Reviewing files that changed from the base of the PR and between c9c227a and 28fa498.

📒 Files selected for processing (1)

• x-ui/migration/generate_subscription_url_mapping.py

[phoseinq]

PR #4 is ready for review/merge.

It includes the original OOM fix plus the follow-up review improvements:

• removed the heavy LEFT JOIN on inbounds.settings
• preloads subscription mappings using (inbound_id, email)
• iterates cursors for both inbounds and client_traffics
• handles empty/invalid settings safely
• narrows exception handling
• adds COUNT(*) based progress logging

Tested successfully on a large deployment: 10,654 users across 3 inbounds on a 2GB RAM server.