Bug: Shared ConfigMaps will be promoted with any Canary succeeding

[Arman17Babaei]

Describe the bug

Config maps (and perhaps secrets, I haven't checked) used by multiple canary deployments will be promoted with only one successful Canary, even if other canaries fail or are still in progress.

This will cause the failed canaries to use the new broken configmap on scale-up/down or event signals and propagate the issue to primary pods unexpectedly.

This is kind of similar to #1046, and builds on top of it.

To Reproduce

1. Create configmap cm with data (key=value)
2. Create deployment A using cm along its proper canary and A is okay with all the values in cm
3. Create deployment B using cm along its proper canary and B is okay only with key=value and will break on any other key/value.
4. Change cm to have data key=new-value
5. Canary A succeeds and copies cm to cm-primary.
6. Canary B Fails and does nothing.
7. Deployment B-primary will use new cm value for new pods and breaks.

Expected behavior

B's Canary should stop the config change in canary stage, and should not let it affect primary pods.

I have two proposals to solve this:

1. Keep all canaries in WaitingPromotion as long as all other canaries sharing the configmap/secret are in progress/failed. This may create frustration in service owners as they might have to fix other teams' problems as well, but is safer and makes more sense to me.
2. Generate a new primary configmap for each Canary (e.g. including Canary's name in the primary configmap). I don't like this idea as this generates a lot of copies of shared configmaps and may lead to diversion between each primary version of the configmap.

We are actively adopting Flagger and I'd be happy to create a PR once we agree on the design.

If we go with option 1, I think it might not necessarily need a feature flag but I understand that it still might cause frustration. (I'm a little worried about too many feature flags added for my cause, so I would like to keep them at minimum)

Additional context

• Flagger version: 1.43.0
• Kubernetes version: -
• Service Mesh provider: istio
• Ingress provider: -