With FreeBSD 11 comes a new version of Bhyve with a feature that makes installing Windows 10 a snap: a VNC accessible framebuffer driver! This lets any GUI OS, such as Windows, boot into graphics mode on the console.
Previously getting Windows working was possible but required a lot of effort to create a custom install ISO and use Windows’ archaic installer command line interface. No more! With the VNC framebuffer everything installs fine with any normal Windows 10 ISO image.
Getting Started / Lessons Learned
Getting things working is very easy — once you know what to do. Let’s start off by explaining the lessons I learned:
Use RealVNC or TigerVNC
TigerVNC has been tested under FreeBSD. If you’re on Windows do not try to use TightVNC, UltraVNC or anything else. It will not work. I wasted hours due to this 😛 Download RealVNC.
The LPC Device
We’ll be using the ‘LPC’ device as it maps the console “BIOS” (pre-framebuffer/UEFI) to the console.
The key here is that you must assign it to slot number 31 or the VM won’t work properly and you’ll see weird errors when you try to type on the console: atkbd data buffer full for every key pressed.
Use USB 3.0
USB 3.0 supports absolute X/Y coordinates for the mouse which makes the VNC use a lot nicer, so we’ll use the xhci (USB 3.0) device in tablet mode to achieve this.
You Can Resize Disks
Windows 7 and up let you resize volumes on the fly from the Disk Management applet (accessible in Windows 10 by right clicking on the Windows logo on the task bar). You can right click on the appropriate volume and shrink or expand it.
To expand it first you’ll want to resize the host disk image volume itself, then reboot the VM (as Bhyve doesn’t update this information on the fly), then go into Disk Management and you’ll see the extra free space at the end of the volume.
To expand a disk image file just pick a larger value and re-run truncate on it with the larger size:
truncate -s 64G os_disk.img
To expand a disk ZFS dataset:
# -- Adjust size: zfs set volsize=64G zroot/vms/win10/os_disk # -- Check size: zfs get volsize zroot/vms/win10/os_disk
These are examples which follow the naming conventions and sizes we use below, adjust as appropriate.
Cores vs. Sockets
Windows 10 64-bit supports up to 2 sockets (or physical CPUs), with up two 256 cores supported. This is an important distinction, since most workstations have a single physical CPU with 4-8 cores.
Bhyve, by default, however presents each core as a physical CPU to the guest OS, thus if you try to give Windows more than 2 CPUs it will only see the first two.
To fix this the following sysctl must be set in /boot/loader.conf (and a reboot to take effect):
hw.vmm.topology.cores_per_package=4
I was not able to find any documentation on this sysctl, but from my testing this is a “grouping” value. So if you specify 4, and give a VM 6 CPUs it will show 4 cores on the first, and 2 on the second. You can still give a VM an arbitrary number of CPUs, including less than this value (1, 2 or 3 will work) and will show to the guest as a single physical CPU with that many core(s).
I welcome any further information on this in the comments!
Setup
OK, so let’s get things going!
Firmware
First install the Bhyve UEFI firmware package:
pkg install -y bhyve-firmware
Networking & Kernel Modules
In /etc/rc.conf add these lines:
cloned_interfaces="bridge0 tap10" ifconfig_bridge0="addm em0 addm tap10"
Where em0 is the NIC you wish to share with your VM. This creates a bridge between the host’s real NIC (em0) and the VM’s virtual NIC (tap10) so it can get out.
Next add the following to /boot/loader.conf:
vmm_load="YES" if_tap_load="YES"
And this to /etc/sysctl.conf:
net.link.tap.up_on_open=1
Reboot to have these changes take effect.
Once you’ve booted back up continue on!
Create Virtual Disk
You have two options, you can create a virtual disk file directly, or use a ZFS dataset and have the VM access it directly (if you’re using ZFS). Each has its advantages, the image file is simplest and will work regardless of underlying filesystem.
Here is how to create a 32 GiB disk image file:
truncate -s 32G os_disk.img
Or, if you wish to create a ZFS dataset instead:
zfs create -V 32G -o refreservation=none zroot/vms/win10/os_disk
The above is a suggested location obviously, if you’re running ZFS I trust you know what you’re doing 🙂
Windows ISO
We’ll assume you already have a Windows 10 install ISO called windows10.iso.
VirtIO Driver ISO
Once we install the OS we’ll need to load VirtIO networking drivers to get the virtual NIC running, download the VirtIO driver ISO here.
First Launch – Install Windows
The command below will wait for you to connect via VNC (on port 5900) before booting, which you’ll need to do. I’ve found the VNC driver is wonky if you don’t use it with the wait parameter, or if you disconnect and reconnect, so be warned — do everything in one go.
Once we have the OS up and running we’ll switch to RDP / Remote Desktop instead.
So! Let’s bring up a 2 core (-c 2) 4 GiB of RAM (-m 4G) instance:
bhyve -c 2 -m 4G -H -w \ -s 0,hostbridge \ -s 3,ahci-cd,windows10.iso \ -s 4,ahci-hd,os_disk.img \ -s 5,virtio-net,tap10 \ -s 29,fbuf,tcp=0.0.0.0:5900,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ win10
Note: If you opted for using a ZFS dataset as a virtual disk you’ll specify the disk as the zvols dev node, for example: /dev/zvol/zroot/vms/win10/os_disk instead of os_disk.img.
Go ahead and connect via VNC, you should see a message Press any key to boot from CD ... be sure to press a key to enter the installer.
Proceed through the installer and install Windows! When the installer is done the VM will reboot and Bhyve will exit.
Second Launch – Install Networking
We’ll bring the instance back up, this time with the VirtIO driver ISO in the VM’s CD drive:
bhyve -c 2 -m 4G -H -w \ -s 0,hostbridge \ -s 3,ahci-cd,virtio-win-0.1.118.iso \ -s 4,ahci-hd,os_disk.img \ -s 5,virtio-net,tap10 \ -s 29,fbuf,tcp=0.0.0.0:5900,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ win10
Once again, connect via VNC to start booting.
Note: You may see a brief error that the CD is unbootable, then it should boot Windows off the virtual hard drive.
Once logged in open the CD drive and go into the NetKVM\w10\amd64 directory, right click on netkvm and select “install”. This should install the drivers and bring your networking up!
Last we’ll enable RDP since you’ll want to use that instead of VNC moving fowrard:
Right click on the Windows logo on the task bar, and pick “System”. From the System window click “Remote settings” from the left menu.
Select the “Allow remote connections to this computer” option under “Remote Desktop” and click “OK”. Attempt to RDP into the VM.
Once that works, shut the VM down one last time.
Normal Booting
Now that everything is set, we’ll boot the VM without a CD drive or the framebuffer. This is how you’ll normally start the VM from now on:
bhyve -c 2 -m 4G -H -w \ -s 0,hostbridge \ -s 4,ahci-hd,os_disk.img \ -s 5,virtio-net,tap10 \ -s 30,xhci,tablet \ -s 31,lpc \ -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ win10
Give it a few moments and you should then be able to RDP in. You’re all set!
I’ve found it to be very stable so far, but you can always re-add the framebuffer line if you have boot issues.
Automatic Startup & Shutdown
While the above command strings will start up your new Windows VM, you probably want it to come up on system startup and at the least be controllable via service start/stop commands.
A-Team Systems has developed Bhyve Management Tool (BMT) specifically for this purpose: see BMT on Github.
It supports FreeBSD, Linux, OpenBSD, NetBSD and Windows VMs as well. Take a look!
Thanks for the article. Have you tried to install Windows 7? In my case, the install ended with the error at the final stage and worked the mouse. I apologize for my English. )
Windows 7 doesn’t have the I/O drivers needed built in (at least the version I tried), I think the process is similar but there would be extra steps. I noticed that USB 3.0 mouse didn’t work, either, too.
This looks really helpful, but I must be missing something: I don’t see a FreeBSD version of realvnc on their website and the port net/vnc has been removed, so which vnc did you use?
I am not sure which would work on FreeBSD, I know nothing but RealVNC worked for me on Windows.
I have seen people have success with TSClient under FreeBSD and also TightVNC (which doesn’t work on Windows with Bhyve). Give it a try and report back?
TigerVNC worked fine.
bhyve dropped out during the second launch above but I just re-issued the command and Windows resumed its configuration options and completed normally.
Windows cannot connect to the DHCP server but I don’t think it is anything to do with Windows as I often have network problems with bhyve. Guess I need to get that sorted out.
Thanks for these instructions, they are the clearest I’ve seen.
No problem and glad to hear that worked. I’ve updated the article to note that TightVNC works for FreeBSD. Regarding networking sounds weird, we have lots of Bhyve VMs in production and everything is rock solid including networking with Windows. One thing we tend to forget: Make sure you’re letting PF ‘skip’ the tap and bridge interfaces if you’re using PF 🙂
You were right – it was PF all along. Doh!
By the way, I am using TigerVNC, not TightVNC.
Thanks again.
Doh! Adjusted to say TigerVNC for FreeBSD. Glad to help!
So, this all works. What about the VM being started at boot time ? Do you have any ideas on how to accomplish that ?
We use BMT which is our own lightweight tool for managing VMs. I just committed some basic documentation which should help you get going!
Thanks for your article. I have finally installed Windows 10 in FreeNAS 9.10 following your instructions. However, I find that no matter what value assigned to “-c” option (the VCPU number), the guest Windows always reports there is only one CPU. Have you ever encounter this?
Interesting, I really don’t know much about Bhyve under FreeBSD 9.1 as most of my experience is with 10.x and 11.x. Did you check hw.vmm.topology.cores_per_package ? Maybe it defaults to 1 in 9.1?
Thanks very much! My version is FreeNAS 9.10 based on FreeBSD 10.3R, not FreeBSD 9.1. I checked my sysctl, and hw.vmm.topology.cores_per_package=1. I consider that is the reason. But I cannot change the value with sysctl command, /boot/loader.conf and /etc/sysctl.conf. Do you have any idea to change the value, or I have to accept the default value anyway?
That value does need to go into loader.conf, however I think FreeNAS has a GUI for this stuff? https://forums.freenas.org/index.php?threads/loader-conf-not-applying.27287/
sysctl will let you check it but this is a boot-time value set in the loader and can’t be set once the system is running AFAIK.
I tuned the value both in the GUI and the shell, but both have no effect. I think it may be a bug when the vmm module recognizes my CPU. However, my acceptable, the guest VM runs quite stable but only slow with only one VCPU. Thanks for your help.
I keep receiving:
rdmsr to register 0xc0010114 on vcpu 0
VNC will start… show the Windows logo… then disconnect with the above being shown in the shell. Not sure what I’m doing wrong here.
Thanks for this great guide! I have managed to install Windows 10 under TrueOS following these instructions but now I need to get PCI passthrough to work for it to be actually usable for me. I need it to pass through some USB port(s) so I can use a USB headset or a similar sound IO device.
I have already checked my laptops ACPI and it does support DMAR / IOMMU. I have followed the instructions on https://wiki.freebsd.org/bhyve/pci_passthru but as it stands I’m getting kernel panics when I try using bhyve with a passthrough option such as:
-s 7,passthru,0/20/0
Have you been able to get PCI passthrough to work? Are USB controllers are fussy about which slot you use, similar to the LPC? It’d be great to see this guide updated to cover PCI passthrough too.
Finally, is there, or is somebody working on, a virtio block device driver for bhyve?
Thanks!
How is the graphics performance of Windows guests?
Good in our experience, faster than VMware ESXi on same hardware in my view. Especially when you take into account being able to ZFS back it.
I have now successfully installed both win7, win10, and windows server 2016.The latest known working version of virtio is 0.1.118. Later version does not work. I used the VNC to install the operating systems. TightVNC works and RealVNC does not work (drops connection frequently due to protocol errors) Problems remaining:
– running windows update will fail on some updates and render the installation unbootable (bhyve exits with code 139 sometimes during the reboot in the update process )
– windows 10 seems to run all available logical cores on 1-2 physical cores. A very laggy experience. I have tried a lot of different values for hw.vmm.topology.cores_per_package and hw.vmm.topology.threads_per_core.
Windows 7 has the best performance due to the CPU-problems in later versions.
For some reason i had to change “-s 30,xhci,tablet \” to “-s 20,xhci,tablet \” because if I don’t the system wouldn’t boot.
Dear, I installed windows server r2 2012 and I have problems when updating with windows update, install in another hypervisor (virtualbox) and I do not have that problem.
pd: can it be the host clock problem?
windows 7:
“windows cannot be installed on this disk”
Thanks a lot, all works just fine.
The last part, however, the one about RDP… I know it’s NOT the theme of this document, but it proves to be the trickiest.
I’ve spent a really enormous ammount of time trying to make xfreerdp connect to the host — only to run into the certificates problem and plane refusal to connect in the end! I really don’t know which side it is to blame now, but freerdp is upable to connect for unknown reasons, and the developers seem to be none the wiser. Windows host uses self-signed certificates (what a surprise, really!), and freerdp for some reasons seems unable to connect.
So maybe it was wise on your part not to go deep into this topic. After all, TigerVNC connection is not bad.
Make sure you have the ‘Allow Network Level Authentication connections only’ check box unchecked (it’s in the System Properties where you enable the Remote Desktop connections).
I’m not 100% sure but give it a try.
I’ve got a “Boot Failed. EFI DVD/CDROM”. tried again following every instruction, same result.
Are you sure this is for Windows 10 and not Windows 7? This error looks like it can happen with Windows 7 and with VMs with more than 4 partitions or drives in it. I would double check the installation ISO, too.
This is a great guide, easy to follow and works perfectly, thank you!
Great howto. Works smooth as a breeze for me on a Sandy Bridge.
Tricky: non us keyboard layout.
But a real good potential besides vmware.
I seem to get:
“Boot Failed. EFI DVD/CDROM
Boot Failed. EFI Hard Drive
.rfb unknown cli-code 97!”
And I have no idea why. Have you experienced this before? I can’t even get the ISO to load and I’ve tried loads of windows one but all linux vm isos work.
Using 11.2 freenas btw.
Hey Josh, sorry, I wonder if this is something different with FreeNAS (vs. FreeBSD)? I would double check that your commandline arguments are exactly correct, too.
Have you had success with this configuration on FreeBSD 12.2-RELEASE? It looks like the tun and tap kernel modules were combined as if_tuntap but on my system I don’t see it being listed when I do kldstat. I am able to get out to the web, but I can’t ping between the VMs on my bridge, and other hosts on my network cannot seem to communicate with the VMs.
I’m suspicious of the tun/tap change.
Seems to work for me under 12.2!
Even works with Windows 11 on FreeBSD 13!
Thanks Adam!
Hi, thank you for the great description, it helped me a lot for my project!
Have you also tried to write a rc.d script or, to be more precise, a script that can start the virtual machine in background, so that one can still go on working with the shell?
I’m trying that currently, but I haven’t been successful yet.
Thank you in advance!
Thank you, and yes take a look at BMT: https://github.com/ateamsystems/bmt/