The operator currently sets cluster.routing.allocation.enable to new_primaries before terminating nodes and therefore also during rolling upgrades:
|
await set_cluster_setting( |
|
conn_factory, |
|
logger, |
|
setting="cluster.routing.allocation.enable", |
|
value="new_primaries", |
|
mode="PERSISTENT", |
|
) |
We recently found that using new_primaries across a rolling upgrade can cause an existing primary shard to become unavailable. The CrateDB documentation has therefore been changed to recommend primaries instead (crate/crate#19751).
The operator is not directly affected by the reproduced scenario. It sets/resets the allocation setting before/after each individual node restart rather than at the beginning/end of the complete upgrade workflow.
Nevertheless, I suggest changing the operator to use primaries as well. primaries provides the same relevant behavior for the operator's workflow, while eliminating two different variants of the rolling upgrade process.
Otherwise, there is a risk that new_primaries is perceived as a generally safe setting for rolling upgrades. If the same approach is later applied in another deployment scenario without the important condition that allocation is restored after every node restart, it can lead to primary shards becoming unavailable.
Standardising on primaries establishes a single canonical recommendation that works with both per-node allocation changes and allocation changes spanning the complete rolling upgrade. The additional restriction imposed by new_primaries does not appear necessary and could become problematic if the operator is ever changed to use a different restart approach.
Proposed change
Replace: value="new_primaries"
With: value="primaries"
The operator currently sets
cluster.routing.allocation.enabletonew_primariesbefore terminating nodes and therefore also during rolling upgrades:crate-operator/crate/operator/operations.py
Lines 777 to 783 in 4a45724
We recently found that using
new_primariesacross a rolling upgrade can cause an existing primary shard to become unavailable. The CrateDB documentation has therefore been changed to recommendprimariesinstead (crate/crate#19751).The operator is not directly affected by the reproduced scenario. It sets/resets the allocation setting before/after each individual node restart rather than at the beginning/end of the complete upgrade workflow.
Nevertheless, I suggest changing the operator to use
primariesas well.primariesprovides the same relevant behavior for the operator's workflow, while eliminating two different variants of the rolling upgrade process.Otherwise, there is a risk that
new_primariesis perceived as a generally safe setting for rolling upgrades. If the same approach is later applied in another deployment scenario without the important condition that allocation is restored after every node restart, it can lead to primary shards becoming unavailable.Standardising on
primariesestablishes a single canonical recommendation that works with both per-node allocation changes and allocation changes spanning the complete rolling upgrade. The additional restriction imposed bynew_primariesdoes not appear necessary and could become problematic if the operator is ever changed to use a different restart approach.Proposed change
Replace:
value="new_primaries"With:
value="primaries"