fix(deployment): give the tile server a resource budget, cap allocator arenas, and spread its replicas

[arnoldcastro5000]

Part of Greenstand/treetracker-infrastructure#315. Companion to #113 (code fixes); this PR is deployment configuration only, no code changes.

---

Background: what went wrong on July 2

The tile server pods currently declare no resource requests or limits. In Kubernetes terms they are "BestEffort" pods, which has two consequences, and we watched both happen in production on July 2:

1. The tile server's memory grows over days (investigated under issue #315). One pod grew to about 2 GB on a 4 GB node, and the node ran out of memory.
2. When a node runs out of memory, Kubernetes kills the pods with no reservation first. So the tile server was killed. But because the pod reserves nothing, the scheduler believed the starving node still had plenty of room, and placed the replacement pod right back on it, where it was killed again immediately. This repeated roughly 90 times in a few minutes before the node recovered, and other services on the node were at risk the whole time.

Change 1: memory request and limit (production overlay)

resources:
  requests:
    memory: 768Mi
  limits:
    memory: 1536Mi

These numbers come from measurement campaigns against production on July 2 and 3 (over 1,000 one-minute samples; details in issue #315):

• Normal steady usage per pod is 510 to 650 MB. The request (768Mi) covers that with margin, so the scheduler makes honest placement decisions.
• Short rendering spikes of up to +458 MB happen for a few minutes when users browse the map at high zoom, then the memory is given back. The gap between request and limit exists exactly for these: they should be absorbed as burst, not reserved permanently (reserving them would tie up ~2.3 GB for two pods on 2.9 GB-usable nodes).
• The slow growth keeps going until something stops it. With the limit in place, a grown pod is OOM-killed and cleanly restarted in place (projected roughly every 2 to 3 days until the code fixes land) instead of taking the whole node down.
• The budget is coherent with the app's existing V8 heap cap: the start script has run node --max-old-space-size=1024 since 2021, so 1024Mi of JavaScript heap plus ~500Mi of native rendering headroom is what 1536Mi represents.
• A final 6-hour campaign on July 3 (360 one-minute samples per pod, zero restarts) reinforced the numbers: the highest working set observed was 1131 MB, about 480 MB under the limit, and no sample reached 1280 Mi. Mean usage was 622 and 641 MB per pod. The hourly memory floors also moved in both directions during the run (one pod kept a +150 MB step for its final hour, the other gave back about 120 MB after a spike), which matches the "slow ratchet plus occasional give-back" picture from the earlier runs.

One honest note for reviewers: the busier pod spends roughly 9 to 12% of its time above 768Mi depending on the day (9.4% in the final campaign, all of it transient spikes). That is expected and fine for a Burstable pod. If the team prefers a more conservative reservation, 896Mi also fits the node math.

Change 2: cpu request, and deliberately no cpu limit (production overlay)

requests:
  cpu: 100m

Measured baseline from a 6-hour campaign on July 3 (359 one-minute intervals per pod): mean 43 to 46 millicores, median 27 to 30, p90 100 to 106, p99 234 to 253, single worst minute 398. So the request sits right at the measured p90 and at roughly twice the mean: without any request a pod gets almost no CPU weight when the node is contended, and slow renders stack up concurrent requests, which raises the very memory spikes this PR contains.

No cpu limit on purpose: cpu limits work by throttling, and they throttle even when the node has idle CPU. For a render server that means longer requests, deeper queues, and higher peak memory. Bursts above the request are short and modest (about 10% of intervals exceeded 100 millicores, only 3 or 4 of 359 exceeded 250, none reached 500) and are absorbed by spare node capacity instead: the two pool nodes averaged 213 and 311 millicores of their 2000 allocatable during the campaign.

Pre-warm forecast: when Airflow's Redis is repaired (issue #321) the map pre-warm job resumes at 182 requests per 5 minutes; the measured CPU cost per rendered tile (46 core-milliseconds, from 41,616 served requests) predicts that adds about 28 millicores average, well within the budget.

Change 3: cap the memory allocator's arenas (production overlay)

- name: MALLOC_ARENA_MAX
  value: "2"

The measurements found that the multi-GB growth is mostly NOT JavaScript memory (the JS heap is capped at 1 GB and the killed pods held 2.0 to 2.5 GB). The best-fit explanation is fragmentation in glibc's memory allocator. Plain-language version: the allocator keeps up to 8 memory pools per CPU (16 on our 2-CPU nodes), and memory freed into one pool cannot be reused by the others, and is almost never returned to the operating system. The tile renderer allocates and frees 300 to 450 MB per render burst, all day, scattered across those 16 pools, so each pool slowly grows toward its own record peak. The process looks like it frees everything (it does, internally) while its footprint ratchets upward for days. Capping to 2 pools makes each burst reuse the previous burst's freed memory instead of growing a fresh pool. This is a standard, widely used setting for exactly this workload shape; the cost is a tiny amount of lock contention between the handful of rendering threads.

We verified on July 3 that no MALLOC setting exists in production today (checked the live pods directly). This change doubles as the experiment that confirms the mechanism: if the slow memory creep flattens after this deploys, we have found the ratchet. If it does not, we have cheaply eliminated a suspect.

Change 4: prefer separate nodes for the replicas (base)

There are 2 replicas but nothing today prevents both landing on the same node, and this service's history shows why that is risky. The measurements add a concrete reason: the load balancer spreads each browsing session across both replicas, so their memory spikes are simultaneous. During the July 3 campaign both pods hit their spike peaks in the same minute (one at 996 MB and the other at 959 MB at 21:04 UTC). On separate nodes that is fine; on one shared 4 GB node it would be a combined 2 GB surge on top of everything else. The added podAntiAffinity is a soft preference: spread across nodes when possible, still schedule if only one node has room. It changes nothing for the single-replica development overlay.

What this PR deliberately does not change

• The existing pin to microservices-node-pool stays exactly as it is. Capacity check: all 5 pool nodes have room for the new requests today (four with ~2.3 GB unreserved, one with ~1.0 GB; cpu allocated is ~592m of ~1900m).
• Resources and the arena cap are added to the production overlay only, because the numbers were measured on production traffic. Dev/test can follow once observed.
• It does not fix the underlying memory behavior; those are code changes tracked in issue #315. This PR is the containment that makes them survivable in the meantime. Values should be re-baselined after those fixes ship.

Deploy notes

• Merging auto-deploys the development overlay via CI. Production needs the usual manual kustomize apply.
• The production rollout restarts both pods; that is fine (they restart regularly anyway), and the first hour of a fresh pod is a measurement issue #315 wants regardless.

Validation

Manifests were structurally validated (both files parse, the patch container name matches the base so the strategic merge binds, env lists merge by name, values and affinity land where expected). Reviewer with kubectl handy: kubectl kustomize deployment/overlays/production should render clean; that is the one check the draft environment could not run.