Use VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE to pause, snapshot, and
resume migratable VFIO devices. pause() transitions to STOP,
resume() back to RUNNING, and snapshot() walks STOP_COPY, drains
the data_fd to EOF, and returns to STOP. The opaque state blob
is attached to the device snapshot as a base64 encoded child
snapshot.

All new behavior is gated on migration_flags.is_some(), so
devices without migration support (including vfio-user) retain
their previous snapshot behavior.

On any transition failure during save, STOP is attempted as best
effort before bubbling the error. Full recovery including device
reset is deferred.

Mirror every non BAR, non MSI config write into the
PciConfiguration shadow via write_byte / write_word / write_reg
so snapshot() captures PCI_COMMAND. The non BAR read path goes
directly to VFIO, leaving the shadow at post init zeros. Without
this, snapshots encode cmd=0 and mlx5_vfio_pci loses bus master
across restore, wedging its firmware cmd queue with ACCESS_REG
timeouts. Avoid write_config_register because its drain of
pending_bar_reprogram when MSE=1 would swallow the BAR reprog
params that the caller reads immediately below.

Signed-off-by: Saravanan D <saravanand@crusoe.ai>

When a snapshot is loaded, walk the migration v2 state machine
in VfioCommon::new() after interrupt state has been restored.
If the device supports migration and a blob is present, drive
RUNNING to RESUMING in a single transition and write the blob
to the data_fd. The kernel handles the intermediate STOP arc
internally. An explicit STOP dwell was observed to make
mlx5_vfio_pci re initialize SQ, CQ, and EQ indices on top of
the just loaded blob, wedging queue state on resume. The
device is left in RESUMING and resume() drives it to RUNNING
during VM resume.

After the blob load, push PCI_COMMAND to the device via
write_config(). set_state() rebuilds in memory MSI / MSI-X
structs but does not touch the kernel's view of PCI_COMMAND,
so without this the VF sits at post reset defaults with no
bus master and mlx5_core ACCESS_REG times out. Rearm
VFIO_DEVICE_SET_IRQS via enable_msi or enable_msix because
set_state() does not reissue the ioctl and the kernel has
no eventfds for this device until this is done. Both match
QEMU vfio_pci_load_config().

In allocate_bars, skip add_pci_bar and add_pci_rom_bar on
restore. PciConfiguration::new(Some(state)) already populated
the BAR registers with used=true, so the extra call trips
BarInUse. The bars vec and mmio_regions pushes still need to
happen so the caller can wire bus mappings.

The new state machine is gated on migration_flags.is_some(),
so devices without migration support (including vfio-user)
skip it entirely.

On any transition or write failure during restore, STOP is
attempted as best effort before bubbling the error.

Signed-off-by: Saravanan D <saravanand@crusoe.ai>

Cover the Vfio trait defaults (query_migration_support returns
Ok(None), set_migration_state and get_mig_data_size return
NoMigrationSupport), VfioMigrationState round trip and invalid
value handling, and the VfioCommon save and load helpers using
a mock Vfio wrapper driven through a libc pipe for the data_fd.

Happy paths verify the transition sequence and the transferred
byte payload. The save happy path asserts the full STOP_COPY to
STOP arc. The load happy path asserts the device is left in
RESUMING so resume() can drive it back to RUNNING. Error paths
verify the best effort STOP recovery when set_migration_state
fails mid sequence.

Also verify the save side PciConfiguration shadow sync: writing
a non BAR config register through VfioCommon::write_config_register
mirrors the bytes into the local shadow so a later snapshot picks
up the live value instead of the post init zero.

Signed-off-by: Saravanan D <saravanand@crusoe.ai>

Add a Snapshot and Restore section to docs/vfio.md covering the
migration v2 requirements (Linux 5.18 kernel, variant VFIO
driver such as mlx5_vfio_pci) and the classification into
migratable and non migratable modes.

The migratable mode description covers the full restore sequence:
the RUNNING to RESUMING single transition (the kernel walks the
intermediate STOP arc), the post load PCI_COMMAND push to the
device, and the MSI or MSI-X eventfd rearm that the kernel state
does not carry. Behavior matches QEMU vfio_pci_load_config().

Note one limitation: the snapshot format stores the opaque
device blob as base64 inside the snapshot JSON, which may
benefit from a binary transport path for very large state.

Extend docs/snapshot_restore.md with a short VFIO section that
distinguishes migratable from non migratable devices and
redirects to docs/vfio.md for details. Replace the stale
"VFIO devices is out of scope" limitation with a note that
live migration support for VFIO devices is tracked as a follow
up.

Signed-off-by: Saravanan D <saravanand@crusoe.ai>