Skip to content

Align cluster.routing.allocation.enable with CrateDB recommendation to use primaries #863

Description

@hammerhead

The operator currently sets cluster.routing.allocation.enable to new_primaries before terminating nodes and therefore also during rolling upgrades:

await set_cluster_setting(
conn_factory,
logger,
setting="cluster.routing.allocation.enable",
value="new_primaries",
mode="PERSISTENT",
)

We recently found that using new_primaries across a rolling upgrade can cause an existing primary shard to become unavailable. The CrateDB documentation has therefore been changed to recommend primaries instead (crate/crate#19751).

The operator is not directly affected by the reproduced scenario. It sets/resets the allocation setting before/after each individual node restart rather than at the beginning/end of the complete upgrade workflow.

Nevertheless, I suggest changing the operator to use primaries as well. primaries provides the same relevant behavior for the operator's workflow, while eliminating two different variants of the rolling upgrade process.

Otherwise, there is a risk that new_primaries is perceived as a generally safe setting for rolling upgrades. If the same approach is later applied in another deployment scenario without the important condition that allocation is restored after every node restart, it can lead to primary shards becoming unavailable.

Standardising on primaries establishes a single canonical recommendation that works with both per-node allocation changes and allocation changes spanning the complete rolling upgrade. The additional restriction imposed by new_primaries does not appear necessary and could become problematic if the operator is ever changed to use a different restart approach.

Proposed change

Replace: value="new_primaries"
With: value="primaries"

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions