Skip to content

Add LCOW live migration support across the controller stacks#2790

Open
rawahars wants to merge 1 commit into
microsoft:mainfrom
rawahars:lm_save
Open

Add LCOW live migration support across the controller stacks#2790
rawahars wants to merge 1 commit into
microsoft:mainfrom
rawahars:lm_save

Conversation

@rawahars

Copy link
Copy Markdown
Contributor

Implement end-to-end live migration support for LCOW pods so a running guest and its workloads can be moved between hosts. Every controller in the stack — VM, pod, Linux container, process, network, and SCSI/Plan9/VPCI devices — gains a save/import lifecycle plus the destination-side patch and resume plumbing needed to rebind local resources and bring the workload back online.

Migration lifecycle:

  • Source: Save serializes a controller's state into a self-describing protobuf envelope and freezes the controller (StateSourceMigrating) so no mutating ops race the transfer; Resume rolls the freeze back, and a finalize Stop or VM teardown terminates it.
  • Destination: Import rehydrates a controller into a migrating state, Patch repoints saved resources (layer VHDs, process IO/bundle, network namespace) at the destination host, and Resume binds the live VM, guest, and devices and republishes events so containerd treats the task as locally running. AbortMigrated discards an imported-but-never-resumed controller and emits synthetic exits so Delete can proceed.

VM controller:

  • Add the source/destination migrating states and the full HCS migration lifecycle: InitializeLiveMigrationOnSource, StartLiveMigrationOnSource, StartLiveMigrationTransfer, FinalizeLiveMigration, plus StartWithMigrationOptions on the destination.
  • Exchange the opaque compatibility blob, retain the final HCS document so the destination can recreate an identical VM, and recover the GCS port/bridge-id allocator floors so reissued ids cannot collide.
  • Make SCSI initialization lazy (built on first use from the HCS document) and handle never-started/destination teardown paths, including the already-stopped HCS error.

Controller-specific changes:

  • SCSI controller switches to an RWMutex and rejects all ops while migrating; ReserveForRootfs now carries the full disk config.
  • Process, network, container, and VM state machines document and enforce the new migrating states and transitions.
  • Pod gains a migrating guard, AbortMigrated fan-out, and routes new containers through lazy SCSI init.

Includes accompanying unit tests for the new save/import/patch/resume paths across all controllers.

@rawahars rawahars requested a review from a team as a code owner June 24, 2026 12:02
@rawahars rawahars force-pushed the lm_save branch 4 times, most recently from a33ffa9 to 3ae59a0 Compare June 24, 2026 14:35
Implement end-to-end live migration support for LCOW pods so a running guest and
its workloads can be moved between hosts. Every controller in the stack —
VM, pod, Linux container, process, network, and SCSI/Plan9/VPCI devices —
gains a save/import lifecycle plus the destination-side patch and resume
plumbing needed to rebind local resources and bring the workload back
online.

Migration lifecycle:
- Source: Save serializes a controller's state into a self-describing
  protobuf envelope and freezes the controller (StateSourceMigrating) so
  no mutating ops race the transfer; Resume rolls the freeze back, and a
  finalize Stop or VM teardown terminates it.
- Destination: Import rehydrates a controller into a migrating state,
  Patch repoints saved resources (layer VHDs, process IO/bundle, network
  namespace) at the destination host, and Resume binds the live VM, guest,
  and devices and republishes events so containerd treats the task as
  locally running. AbortMigrated discards an imported-but-never-resumed
  controller and emits synthetic exits so Delete can proceed.

VM controller:
- Add the source/destination migrating states and the full HCS migration
  lifecycle: InitializeLiveMigrationOnSource, StartLiveMigrationOnSource,
  StartLiveMigrationTransfer, FinalizeLiveMigration, plus
  StartWithMigrationOptions on the destination.
- Exchange the opaque compatibility blob, retain the final HCS document so
  the destination can recreate an identical VM, and recover the GCS
  port/bridge-id allocator floors so reissued ids cannot collide.
- Make SCSI initialization lazy (built on first use from the HCS document)
  and handle never-started/destination teardown paths, including the
  already-stopped HCS error.

Controller-specific changes:
- SCSI controller switches to an RWMutex and rejects all ops while
  migrating; ReserveForRootfs now carries the full disk config.
- Process, network, container, and VM state machines document and enforce
  the new migrating states and transitions.
- Pod gains a migrating guard, AbortMigrated fan-out, and routes new
  containers through lazy SCSI init.

Includes accompanying unit tests for the new save/import/patch/resume
paths across all controllers.

Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant