I’ve been facing a strange problem with my kata setup that manifests in my inability to use more than one vCPU.

The normal flow is that you use k8s’s limits.cpu to set the number of vCPUs; however, that doesn’t work for me:

Error: failed to create containerd task: failed to create shim task: failed to hot add vCPUs: only 0 vCPUs of 3 were added

This error comes from hotplugAddCPUs function, that talks to QEMU over the QMP socket and first queries the number of spare CPUs and then tries to add more (if needed). The QMP socket is a file descriptor, though, so you can’t access it directly. There is some code that allows you to expose a separate, unix socket, which is only enabled in debug mode, as you can easily break kata state by manipulating the guest directly.

The debug mode you need to enable in the config, so I went down the road figured if I can sideload a config with just that change, when I stumbled on a list of annotations you can apply to pod. Yes, it had io.katacontainers.config_path to set a config, but it also had io.katacontainers.config.hypervisor.default_vcpus to set the bootup vCPU count directly! The latter, though, is gated by an annotation check, so you can’t change it unless it’s in the config’s enable_annotations.

You can change the config with no questions asked, though.

And this is how I had an override config via the first annotation that uses the second annotation to set the vCPU count! Now, the only missing bit of the puzzle is to modify the Nix derivation for kata-runtime so that the config is always patched to allow setting CPU and memory count:

1
2
3
4
5
6
7

kata-runtime-pkg = pkgs.kata-runtime.overrideAttrs (oldAttrs: {
  postInstall = (oldAttrs.postInstall or "") + ''
    configFile="$out/share/defaults/kata-containers/configuration.toml"

    sed -i 's/^enable_annotations.*$/enable_annotations = ["enable_iommu", "virtio_fs_extra_args", "kernel_params", "default_vcpus", "default_memory"]/' "$configFile"
  '';
});