Add change management process

[drewr]

Describes how we use a google calendar to coordinate human-orchestrated changes.

[ecv]

Roger that production changes are calendar events.

Non-production changes however, do not need to be calendar events, and dry-running maintenances against non-production non-traffic-serving sites does not appear to warrant calendaring. I think this was my confusion this morning.

So, for instance, I do not have to announce to anyone when I am scripting a maintenance, only when I am performing it.

[scotwells]

@drewr would production releases that are automatically orchestrated through Flux also require calendar events? Example being creating a new tag in our infra repo: https://github.com/datum-cloud/infra/releases/tag/v0.30.0

[drewr]

@scotwells A system-initiated change doesn't need a calendar entry because it already has an anchor point in time. A human-generated change in the automation which creates the release is perhaps calendar-worthy. It doesn't solve the passive knowledge of something happening to production, but we just expect that robots are constantly changing the system in small ways (including users!) and that those behaviors are normal instead of exceptional.

[scotwells]

@drewr I think where I get tripped up there is that the creation of the release is still human-generated, but the deployment of that change to production is automated.

Agree with needing proper documentation / knowledge sharing that something is going to happen with production.

[drewr]

You're right, and ideally there would be some foreknowledge that a release was going to happen. However, I don't know that we want to gate every release with that level of friction, or we un-do the purpose of frequent software releases, even though they have the potential to break things. This level of change management is more about the types of changes that are difficult to automate or we haven't yet automated.

[scotwells]

@drewr fully agree there. Likely worth having a note about that in the doc just to provide clarity.

[drewr]

@scotwells Ack. Updating now

Non-production changes however, do not need to be calendar events

@ecv It's not limited to production. Treat the calendar as a tool. If there is a non-production change that you are jumping over to slack to 📣 to everyone about it, it probably should be in the calendar. It just means something with a big blast radius at a particular time is happening that folks should know about.