This is the second PR in a series adding support for x86_64 CPU profiles to Cloud hypervisor which is requested in #7068.

The x86_64 CPU profile feature in its entirety is currently tracked in #7068 (comment), but you may also want to read the "full implementation overview" section below to get the big picture of what the complete implementation will look like.

Reviewers unfamiliar with CPU profiles/templates/models may want to jump directly to the "Motivation and background" section in this description first.

What this PR does

This PR does not add any user facing functionality to Cloud hypervisor. All it does is introduce data structures and logic for updating and filtering MSRs in accordance with a (at this point) hypothetical CPU profile.

The logic introduced here is (mostly) oblivious to the CPU manufacturer (as long as it is an x86_64 CPU), but focuses on the KVM hypervisor. This means that there are a few unimplemented!() invocations in the MSHV path, but note that these paths should not be encountered in the case of the host CPU profile.

If/When CPU profile support for MSHV is implemented then it should be possible to work with a lot of the code introduced in this PR though.

Differences with the implementation in the Cyberus Technology fork

For those familiar with the working prototype available in the Cyberus Technology fork there are a few differences to note here with regards to the implementation:

1. We let KVM take care of most feature MSR compatibility checks for us. Since this is done when setting MSRs we unfortunately had to perform a more invasive refactor, but it saves us from introducing a lot of relatively complex code.
2. The check for IA32_ARCH_CAPABILITIES is still necessary though because KVM does not check compatibility when setting that particular MSR. I noticed a bug in the initial implementation during this work and it has been fixed here.
3. We now use a filter that explicitly permits MSRs specified by the CPU profile and denies everything else instead of declaring what to be denied in the filter and allowing everything else. We expect the explicit permit mode used here to be more robust and future proof.

Our working prototype

We have a full implementation for Intel CPUs on our fork.

Generating a new CPU profile

See our docs for instructions on how to generate a new CPU profile for your own Intel CPU(s)

Experimenting with existing CPU profiles

You can also test running cloud hypervisor with one of our pre-generated profiles: Intel Sapphire Rapids or Intel Skylake; Simply add ,profile=sapphire-rapids or ,profile=skylake to the --cpus <cpu option> argument when bringing up the VM.

How we have tested our implementation

We have performed a range of tests on deployments of various sizes including:

• Live migration from an Intel Granite Rapids to an Intel Sapphire Rapids processor where the Sapphire Rapids CPU profile is being utilized. We ensured that the migration was successful while the guest was executing a workload involving the intel AMX feature (available on both processors).
• CPU profile restricted VMs boot with several images (CirrOS, Ubuntu, Windows Server, NixOS) and also with direct kernel boot.
• Checks that MSRs not permitted by CPU profiles are not available to the guest.
• A plethora of successful live migrations on our customer's clusters.

Full implementation overview

At a high level our implementation consists of:

1. A tool for generating a CPU profile for the hardware and hypervisor the tool is executed on. The output of the tool are JSON files describing required CPUID and MSR adjustments.
2. A build script that traverses all pre-generated CPU profile JSON files, then bakes their compressed versions into the Cloud hypervisor binary at build time and also generates a CpuProfile enum with a variant per CPU profile together with logic for extracting the compressed JSON files.
3. Cloud hypervisor is adapted to decompress and deserialize the aforementioned CPUID and MSR adjustment descriptions and sets CPUID and MSR values visible to the guest accordingly.
4. We perform compatibility checks at runtime to determine whether the chosen CPU profile can be applied given the hardware and hypervisor the VM is running on. If these checks fail a hard error is returned and the VM will not start.

We plan to bring all of this functionality to Cloud hypervisor through several PRs:

1. Plumbing for enabling Cloud hypervisor to adjust CPUID entries according to a CPU profile (this PR).
2. Plumbing for enabling Cloud hypervisor to adjust MSR entries according to a CPU profile.
3. The CPU profile generation tool limited to CPUID adjustments on Intel CPUs with the KVM hypervisor.
4. Extending the CPU profile generation tool to also consider MSRs on Intel CPUs with the KVM hypervior.
5. Introducing the build script that generates Rust code based on the CPU profile JSON files in the tree.
6. Including some CPU profiles generated by our tool (Intel Sapphire Rapids and possibly also Intel Granite Rapids and Intel Skylake). This could be split into more PRs.
7. Adding support for AMD CPUs. Note that we have yet to do this on our fork, but we want to get it done in the not too distant future. This step may of course also possibly be split into more PRs.

The CPU profile generation tool in more detail

Our CPU profile generation tool is aware of pretty much all (Intel) CPUID (sub) leaves and MSRs as well as CPUID and MSR entries tied to KVM.

It uses these hard coded lists together with our specified policies when the tool is executed to select, or prevent CPU features to become part of the generated CPU profile.

We will do our best to make the tool automatically warn or error when it encounters CPUID leaves and/or MSRs it is not aware of as this is a good sign that the tool needs to be updated.

If/when individual reserved CPUID and/or MSR bits within a CPUID or MSR register become specified, then one may also want to update the CPU profile generation tool to take this into account. If this is forgotten then we primarily expect this to just lead to new profiles having a slightly reduced set of supported features. This is something we can fix upon detection by re-generating the profile (with a "v2" suffix for backward compatibility reasons).

Additional binary size per new CPU profile

Our experiments show that with compression each new CPU profile adds between 3 and 4 KB to the Cloud hypervisor binary. Without compression the pretty printed JSON files sum up to about 50 - 60 KB per CPU profile.

Further optimizations to binary size are definitely possible, but we consider 3 - 4 KB per CPU profile good enough for the time being.

Motivation and background

Recall that software is usually developed to run on a variety of processors with various features. In order for the software to dynamically discover which hardware features may be utilized one typically uses the CPUID instruction to query the CPU for information, or in some often more low-level cases one uses so called MSRs (model specific registers) to obtain relevant processor specific information.

In the context of live migration this can of course lead to a time of check to time of use bug if the guest obtains processor information through CPUID or MSRs from the migration source and then ends up making decisions based on these findings on the migration destination that may have a different processor that might not support the same instructions as the migration source.

To mitigate this problem Cloud hypervisor performs CPUID checks (MSR checks are done by KVM, but it is debatable whether these are sufficient) at the beginning of a live migration. If the CPUID entries reported by the destination's hypervisor are not compatible with those of the source VM then the migration is aborted.

Such compatibility checks, although important, are thus also somewhat limiting in clusters with hosts/nodes running on different processors. There is no way to perform a live migration from a host with a say Intel Granite Rapids processor to a destination with a Intel Sapphire Rapids processor, even if the guest is not utilizing any functionality outside of the capabilities of the Intel Sapphire Rapids machine.

The aforementioned checks also prevent migrations from older hardware to newer in some cases where the older hardware supports deprecated CPU features (such as for example Intel MPX).

There are also certain CPU features that are unlikely to ever give accurate results in the context of live migration such as performance counters and debugging capabilities. We do not want guests making decisions based on these capabilities during live migration, even when all CPUs involved are identical!

Luckily hypervisors are capable of manipulating what guests see when executing the CPUID instruction or reading MSRs. This is a fact that we can and will use to our advantage. A CPU profile is thus a recipe for adjusting CPUID (sub) leaves and MSRs to hide certain CPU features from guests. One can then vastly increase the number of nodes/hosts in a cluster a booted VM with an applied CPU profile may live migrate to at some point in the future!

FAQ

What about other ISAs

We focus on x86_64 for the time being. Other architectures such as ARM and RISC-V will only have the Host CPU profile for the foreseeable future, unless someone steps up and wants to tackle either of them already now.

Due to the vastly different nature of these architectures we expect there to be relatively little overlap with our work here in the context of x86_64 CPUs.

What about live migration between Intel and AMD CPUs?

It might work with an extremely limited minimal profile, but I don't think that would be suitable for workloads intended to be used in production.

The intended use-cases are thus Intel <-> Intel and AMD <-> AMD live migrations.