Proxmox Configuration for PCI Passthrough

For this project as mentioned earlier, I bought a ASRock Fatality x399 Motherboard.  To enable virtualization and iommu support you’ll have to hunt through the BIOS/UEFI settings tabs/folders.    With ASRock you’ll easily find it.  This should be under the processor submenu or in the Northbridge configuration page.  It was pretty obvious when I went looking for it.  From here simply following the following:

I’ve lifted the appropriate text, but I would read the original..The Basic steps are as follows

  • Enable Virtualization and IOMMU support in the bios/UEFI
  • Boot up and log back in with the root account using SSH
  • Update Grub

Activate IOMMU OS Support

vi /etc/default/grub
# Change the GRUB_CMDLINE_DEFAULT to
GRUB_CMDLINE_LINUX_DEFAULT="quiet" amd_iommu=on"
:wq!
update-grub
reboot
  • Confirm that the IOMMU is enabled by running:
dmesg | grep -e IOMMU 

If you see something like:

[0.621825] AMD-Vi: IOMMU performance counters supported
[0.621878] AMD-Vi: IOMMU performance counters supported
[0.623248] iommu: Adding device 0000:00:01.0 to group 0
[0.623902] iommu: Adding device 0000:00:01.1 to group 1
[0.624661] iommu: Adding device 0000:00:01.3 to group 2
[0.625369] iommu: Adding device 0000:00:02.0 to group 3
[0.626075] iommu: Adding device 0000:00:03.0 to group 4
[0.626768] iommu: Adding device 0000:00:03.1 to group 5
[0.627532] iommu: Adding device 0000:00:04.0 to group 6
[0.628252] iommu: Adding device 0000:00:07.0 to group 7
[0.629031] iommu: Adding device 0000:00:07.1 to group 8
[0.629740] iommu: Adding device 0000:00:08.0 to group 9
.
.
.
.
[0.647639] iommu: Adding device 0000:42:00.3 to group 35
[0.648339] iommu: Adding device 0000:43:00.0 to group 36.
[0.648956] iommu: Adding device 0000:43:00.2 to group 37.

Configure PCI Passthrough VFIO*

  • Add the following to /etc/modules
    • vfio
    • vfio_io_mmu_type1
    • vfio_pci
    • vfio_virqfd
update-initramfs -u
reboot
  • Ignore the section on IOMMU interrupt remapping.  I  wasted hours trying to confirm this..  Turns out it is outdated at least for my hardware
  • This is critical… 
    • Find the PCI numeric ID your card as been assigned,
lspci | grep -i vga
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev e7) (prog-if 00 [VGA controller])
    • My numeric ID is 09:00.0
lspci -v -s 09:00 | egrep '09:00'
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev e7) (prog-if 00 [VGA controller])
09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0
  • Both of the GPU/Video card and the built in Audio device are on the same PCI Bus
  • Now we just have to confirm that the two are isolated in their own IOMMU group.
dmesg | grep 09:00 | grep -i group
[0.635134] iommu: Adding device 0000:09:00.0 to group 16
[0.635330] iommu: Adding device 0000:09:00.1 to group 16
  • Now we just have to confirm that the two are isolated and there is nothing else in group 16.
root@max:~# dmesg | grep 'group 16'
[0.635134] iommu: Adding device 0000:09:00.0 to group 16
[0.635330] iommu: Adding device 0000:09:00.1 to group 16
  • At this point IOMMU is functional, and the GPU is isolated and not sharing a group.  If isn’t, just refer to online instructions to resolve

Blacklisting

We don’t want the Debian/Underlying OS to use the GPUs so we need to blacklist them.. This can be done by getting the Vendor ID or blacklisting the Vendor by alphanumeric chars.

root@max:~# cat /etc/modprobe.d/blacklist.conf 
blacklist radeon
blacklist Radeon
root@max:~#update-initramfs -u
root@max:-#reboot

Gathering Info to Build the VM

  • We need this information to build the VM.. Record it for later.
root@max:~# lspci -n -s 09:00
09:00.0 0300: 1002:67df (rev e7)
09:00.1 0403: 1002:aaf0
  • 09:00.0 is the Video GPU which has an id of 67df and 09:00.1 is the audio at aaf0 both have Vendor ID of 1002
  • Create the following file so that it looks like this, but with your info
root@max:/etc/modprobe.d# cat vfio.conf 
    options vfio-pci ids=1002:67df,1002:aaf0
root@max: reboot
  • After rebooting, confirm
lspci -v -s 09:00 | egrep '09:00|Kernel drive' 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev e7) (prog-if 00 [VGA controller])
  Kernel driver in use: vfio-pci 
09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0
  Kernel driver in use: vfio-pci
  • Please note that the Kernel Driver is vfio-pci
  • You should be ready to build the VM..
  • Note:  I had to add the following in grub video=efifb:off to GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"
  • Note.
    • Please refer to dmesg frequently.   It will quickly isolate and help solve any issue you might have..
    • Don’t ignore the bottom of Proxmox’s PCI Passthrough guide.  It is very helpful

Leave a Reply