I spent a year with CentOS on gigadelic, my web server, and found myself itching for FreeBSD. This installment of my unintentional “How to install FreeBSD on things” series examines the steps necessary to get FreeBSD running comfortably in my Xen environment, a paravirtualized instance at prgmr.
Note: This information also lives on the Prgmr wiki and may be more up-to-date there.
FreeBSD has fairly complete Xen domU support, though it’s not as well-tested as NetBSD’s. Nor does the FreeBSD project distribute an easy domU distribution, like NetBSD’s netbsd-INSTALL_XEN3_DOMU.gz that’s runnable right from pvgrub, so the installation procedure is a tad involved.
You may either build your own copy of FreeBSD/Xen from an existing FreeBSD installation or download my prebuilt copy.
Xen paravirtualization (PV), as used on prgmr, is supported only on FreeBSD i386, not amd64. FreeBSD amd64 supports only Xen HVM and is thus unusable in the prgmr environment.
Someone else’s binaries. Do you trust me?
Build Your Own Distribution
The operating system is built using the standard procedure, cross-compiling for i386 when necessary.
FreeBSD 9.0-RELEASE has a few outstanding issues on Xen, such as the inability to function with more than 768mb of provisioned memory and a panic when initializing SMP, even on vcpus=1 instances like mine. These two issues are fixed in 9-STABLE and will be rolled into 9.1-RELEASE later this year. I prefer to stick to releases when I can, though, so apply these two patches to your RELENG_9_0 source tree.
First, to fix the memory limitation. Taken from r228746.
And second, to bypass the SMP panic by limiting MAXCPU to 1. It may be worth building 9-STABLE instead if you require SMP support.
Optionally, patch xen/clock.c to silence annoying repeated console-spamming clock nudge messages unless the system is booted verbosely. Seen in kern/155353.
The stock XEN kernel config file should boot with no trouble. You’ll probably want to customize it a bit, though, to enable features like PF/ALTQ and to disable the system-slowing kernel debugging features, especially witness. Disable debugging with caution. You may want to build a XEN kernel first to ensure it boots and to diagnose why if it doesn’t.
You’re ready to build your i386 world and kernel. Use the TARGET and TARGET_ARCH environment variables to cross-compile for i386 if necessary, such as on my amd64 build host. Specify either the stock XEN kernel configuration or the name of any custom configuration you create.
Create a directory to serve as the temporary home of your OS distribution, set DESTDIR to its path, and install.
Set the mount options in fstab for what will be our root filesystem.
Remove the default virtual terminals from /etc/ttys. They don’t exist in the Xen environment and will throw errors into your console at boot if included. Then, add a line for the Xen console device so we can access the system via our out-of-band console.
Make any other changes you wish to include, create a tarball of the Xen world, and copy the kernel ELF out of DESTDIR for later use.
If you’re converting an existing Prgmr instance to FreeBSD, double-check your backups and make sure it holds nothing you’d be mad at yourself for losing.
Reboot your instance to the CentOS rescue environment via the out-of-band console, then blow away your disk with fdisk. Create a 512MiB Linux partition, then dedicate the rest of the disk to FreeBSD, hexadecimal partition type a5.
My properly-partitioned 25GB disk on gigadelic looks like this:
Format your Linux partition as ext2, a nice universally-mountable default, then mount it and create some needed directories.
Transfer your kernel ELF and world tarball to the root of the ext partition. These two files will consume approximately 5MiB and 150MiB, respectively. I hosted the two files using www/woof from my build machine.
Alternatively, download my pre-built world and kernel.
Move your kernel of choice to the /mnt/boot directory with filename kernel.
UFS via NetBSD
Reboot your instance once more, this time to the included NetBSD installer. If your dom0 doesn’t include NetBSD, check the directions for fetching the installer in the NetBSD how-to wiki article.
Drop to a shell when given the option. Double-check your partition layout with NetBSD’s fdisk. Your disk device will be /dev/xbd0.
The FreeBSD partition you created in Linux’s fdisk will show up as slice f and the ext partition as slice e of xbd0, as visualized here in the partitioning step of NetBSD’s guided installer.
Create a new FFSv2 filesystem on slice f.
Create mountpoints for both partitions, mount them, and extract your FreeBSD world tarball.
While still in NetBSD, create a text file at /mnt/boot/boot/grub/menu.lst. pvgrub will expect the user bootloader configuration in (hd0,0)/boot/grub/menu.lst. (hd0,0) is your ext2 partition in this case.
The kernel lines’ arguments instruct the kernel to root from /dev/xbd0s2, as seen earlier in fstab, to avoid a panic seen in kern/152228, and to tick the system clock at 100hz. kern.hz should be detected automatically, but specifying it here won’t hurt. The second pvgrub entry is identical but boots verbosely.
Once that’s in place, shut down the instance and make sure, through the admininstation console, that you’re using the i386 pvgrub. amd64 pvgrub cannot load our i386 kernel ELF. Option #6 in the console allows you to swap bootloaders.
Finally, start your instance and watch through the out-of-band console as pvgrub loads your user configuration and eventually the FreeBSD kernel!
First Boot Maintenance
Booted successfully? Awesome. There are just a few things to take care of before you can settle in to FreeBSD.
First, use mtree to repair directory and file permissions and flags mangled by NetBSD’s tar.
Enable blf hashing in login.conf as well, along with the UTF-8 character set.
Rebuild the login database with cap_mkdb /etc/login.conf, then set your new blf-hashed root password with passwd.
Set your time zone information with tzsetup, then create the mail alias database for Sendmail.
Set your hostname and configure networking in rc.conf. IPv4 can be configured with DHCP, but IPv6 must be configured manually. See Setting up IPv6 on the prgmr wiki. Gigadelic resides in the Fremont datacenter, so this example uses the Fremont IPv6 gateway address.
You can test your IPv6 setup by ping6-ing a known-v6 host.
Turn on OpenSSH if you desire remote access. Please realize your SSH daemon will be subject to near-constant automated break-in attempts. Use good passwords, leave AllowRootLogin disabled in /etc/ssh/sshd_config, and consider blocking bad hosts with PF or another firewall.
With that, you’re all done! Fetch a Ports tree with portsnap fetch extract, and install some software.
Spurious LORs under KERNCONF=XEN
The stock KERNCONF XEN has WITNESS enabled, a debugging feature for lock validation. If you’re running with options WITNESS you may see some spurious LORs related to the filesystem. They’re false positives, as noted in i386/153260, and your system will safely come right out the other end.
Updating the kernel
If you want to update your FreeBSD kernel ELF from within FreeBSD itself, install sysutils/e2fsprogs, create a mount point, and mount your ext2 boot partition.
FreeBSD has powered gigadelic and this blog for a month now with no panics or other show-stoppers.