Skip to content

Accept pre-opened VFIO cdev and iommufd FDs#8287

Merged
rbradford merged 11 commits into
cloud-hypervisor:mainfrom
likebreath:0506/accept_vfio_cdev_iommufd_fds
Jun 17, 2026
Merged

Accept pre-opened VFIO cdev and iommufd FDs#8287
rbradford merged 11 commits into
cloud-hypervisor:mainfrom
likebreath:0506/accept_vfio_cdev_iommufd_fds

Conversation

@likebreath

@likebreath likebreath commented May 28, 2026

Copy link
Copy Markdown
Member

Enable cloud-hypervisor to consume pre-opened VFIO cdev and iommufd file descriptors supplied by the caller, rather than always opening them itself from sysfs and /dev/iommu. This lets a privileged orchestrator open and configure the device nodes once and hand the FDs to an unprivileged VMM, both at boot (CLI) and at runtime (HTTP API). It is also a prerequisite for live migration for VMs using VFIO devices, where the destination VM cannot be tied to the exact sysfs device node on the source host.

@likebreath likebreath force-pushed the 0506/accept_vfio_cdev_iommufd_fds branch 3 times, most recently from 0d10fc1 to cd380da Compare May 28, 2026 19:56
@likebreath likebreath marked this pull request as ready for review May 28, 2026 20:27
@likebreath likebreath requested a review from a team as a code owner May 28, 2026 20:27
@rbradford

rbradford commented May 28, 2026

Copy link
Copy Markdown
Member

@arctic-alpaca How is this going to interact with your new proposed FD work - I want to make sure we end up having these clash

@likebreath

likebreath commented May 29, 2026

Copy link
Copy Markdown
Member Author

How is this going to interact with your new proposed FD work - I want to make sure we end up having these clash

I thought about this and don't think it should be a problem. The main overlap is in how FDs are passed via the CLI or API, and in both cases this change follows the existing status quo (e.g., virtio-net FDs). Whatever redesign we apply to FD handling later should apply to the changes here in the same way.

saravan2
saravan2 suggested changes May 30, 2026
View reviewed changes

@saravan2 saravan2 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first commit build: Bump iommufd-ioctls and vfio-ioctls needs to update fuzz/Cargo.lock because a committed lockfile in our repo that doesn't match the manifests is misleading.

Our CI builds fuzz without --locked so it repairs the stale committed lockfile automatically instead of erroring on it.

I cannot say for sure whether this relaxed setup was intentional so that contributors need not bother updating fuzz/Cargo.lock

@arctic-alpaca

arctic-alpaca commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

@arctic-alpaca How is this going to interact with your new proposed FD work - I want to make sure we end up having these clash

Thanks for the ping! The changes in this PR will lead to a bit more churn, but don't conflict with anything I'm working on.

The main overlap is in how FDs are passed via the CLI or API, and in both cases this change follows the existing status quo (e.g., virtio-net FDs). Whatever redesign we apply to FD handling later should apply to the changes here in the same way.

I'm (also) working on replacing the i32/RawFd types with OwnedFd to improve type-safety, but as you said, that will apply to the changes from this PR the same as to the existing FD handling.

rbradford
rbradford reviewed Jun 1, 2026
View reviewed changes
Comment thread vmm/src/vm_config.rs Outdated
Comment thread vmm/src/vm_config.rs Outdated
Comment thread vmm/src/seccomp_filters.rs
@likebreath likebreath force-pushed the 0506/accept_vfio_cdev_iommufd_fds branch 2 times, most recently from 2e7fa00 to c4e988c Compare June 1, 2026 23:37
@likebreath

likebreath commented Jun 1, 2026

Copy link
Copy Markdown
Member Author

All comments are addressed. PTAL. Thank you!

saravan2
saravan2 suggested changes Jun 2, 2026
View reviewed changes
Comment thread vmm/src/device_manager.rs
Comment thread vmm/src/device_manager.rs
@rbradford

rbradford commented Jun 4, 2026

Copy link
Copy Markdown
Member

@likebreath Thanks! We're in an unfortunate situation where VFIO integration tests have been red for some time. I don't really have the bandwidth or knowledge to address that - would you be happy landing this first with those tests known failing or would you prefer to try and get those fixed?

@likebreath

likebreath commented Jun 4, 2026

Copy link
Copy Markdown
Member Author

@rbradford Thank you for bringing this to my attention. I'd prefer to land this PR first and then address the CI issue around VFIO separately. The changes here are pretty self-contained and is tested by a new integration test. This would also enable @saravan2's work on vfio device live migration as well as the @arctic-alpaca's work on FD refactoring.

I also noticed more flakyness around the VFIO CI worker recently, particularly around vfio device hotplug, and I believe it is related to the nested virtualization setup (that relates to virtio-iommu being used for the L1 VM). Need to take further look.

@likebreath likebreath requested review from rbradford and sboeuf June 8, 2026 23:25
rbradford
rbradford requested changes Jun 11, 2026
View reviewed changes
Comment thread vmm/src/vm_config.rs Outdated
Comment thread vmm/src/api/http/http_endpoint.rs Outdated
Comment thread vmm/src/seccomp_filters.rs
@rbradford rbradford mentioned this pull request Jun 12, 2026
Draft
@likebreath likebreath force-pushed the 0506/accept_vfio_cdev_iommufd_fds branch 2 times, most recently from 8d99ab5 to d0a1dff Compare June 15, 2026 18:39
rbradford
rbradford approved these changes Jun 16, 2026
View reviewed changes
Comment thread vmm/src/device_manager.rs
@likebreath likebreath force-pushed the 0506/accept_vfio_cdev_iommufd_fds branch 2 times, most recently from dbe2172 to 91cb6e7 Compare June 17, 2026 00:35
@likebreath
build: Update rust-vmm dependencies in /fuzz
ee65991 
Signed-off-by: Bo Chen <bchen@crusoe.ai>
@likebreath
vmm: seccomp: Allow IOMMU_DESTORY ioctl
659339a 
Update the seccomp filter for vmm and vcpu thread, because `Drop for
VfioIommufd` since vfio-ioctls v0.6.1 now issues IOMMU_DESTROY
to release the IOAS allocated for each VM boot.

Signed-off-by: Bo Chen <bchen@crusoe.ai>
@likebreath
vmm: DeviceConfig: Store path as Option<PathBuf>
b55bb64 
Relax DeviceConfig::path from PathBuf to Option<PathBuf> in preparation
to accept an externally-opened vfio cdev FD. The parser and OpenAPI spec
still enforces that `path` is set, so callers see no behavior change.

Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
vmm: DeviceConfig: Add fd field for an externally-opened vfio cdev
8bf9c35 
Add a new `fd: Option<i32>` field to DeviceConfig so a caller can
supply a pre-opened vfio cdev FD (e.g. /dev/vfio/devices/vfioN) in
addition to the existing sysfs path. The CLI `--device` option now
accepts `fd=<n>`, parsed alongside the existing options.

Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
vmm: http_api: Accept a VFIO device FD via SCM_RIGHTS
ea24b5c 
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
vmm: Enable FD-based VFIO devices
33e61cf 
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
vmm: OpenAPI: Make DeviceConfig path optional
0bbe930 
`path` is no longer required in the DeviceConfig, since a VFIO
device may also be supplied via a pre-opened cdev FD passed via
SCM_RIGHTS alongside the /vm.add-device request.

Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
ch-remote: Support FD-based VFIO devices
79d4059 
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
vmm: Accept an externally-opened iommufd FD
04416d2 
The CLI `--platform` option now accepts `iommufd_fd=<n>` alongside the
existing `iommufd=on|off`.

Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
vmm: device_manager: Use externally-supplied iommufd FD
26a01bc 
Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath
tests: Add integration test for FD-based VFIO device
b895cd2 
This also covers the usage of pre-opened iommufd FD.

Signed-off-by: Bo Chen <bchen@crusoe.ai>
Assisted-by: Claude:Opus-4.7
@likebreath likebreath force-pushed the 0506/accept_vfio_cdev_iommufd_fds branch from 91cb6e7 to b895cd2 Compare June 17, 2026 00:40
@likebreath likebreath enabled auto-merge June 17, 2026 01:07
@likebreath likebreath added this pull request to the merge queue Jun 17, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 17, 2026
@rbradford rbradford added this pull request to the merge queue Jun 17, 2026
Merged via the queue into cloud-hypervisor:main with commit ca2f847 Jun 17, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@phip1611 phip1611 phip1611 left review comments

@rbradford rbradford rbradford approved these changes

@sboeuf sboeuf sboeuf approved these changes

+1 more reviewer

@saravan2 saravan2 saravan2 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@likebreath @rbradford @arctic-alpaca @phip1611 @saravan2 @sboeuf