How to netboot a Raspberry Pi 3 from a Linux server with TFTP and NFS

31 January 2018 · Adam Fontenot

Update for Raspberry Pi 4:

I originally wrote this guide in 2018, and then I got a Raspberry Pi 4 the same month as they were released. Unfortunately, The Pi 4 was released without initial support for netbooting. I did manage to work out how to get basic functionality, but I still had to keep the /boot partition on the SD card, which also meant remembering to sync that partition with the NFS /boot folder every time there was an OS update.

Even though the SD card wasn’t even mounted 99% of the time, apparently the heat of being connected to a Pi was enough to eventually fry my SD card. I managed to get it working just long enough to update the firmware to a new version, which just now, as of this month, arguably has full support for netbooting. (SELF_UPDATE could previously be enabled, but it’s finally the default. This means that the Raspberry Pi folks now support doing firmware updates via USB or NFS, not just the SD card.)

Several steps are needed: make sure you’re on the latest (>= 2020-09-03) firmware. Then run the following to enable boot from a TFTP server:

cd ~
cp /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2020-09-03.bin .
rpi-eeprom-config pieeprom-2020-09-03.bin > boot.config
# open boot.config in your editor of choice and change BOOT_ORDER to 0x412
rpi-eeprom-config --out new-pieeprom-2020-09-03.bin --config boot.config pieeprom-2020-09-03.bin
sudo rpi-eeprom-update -d -f new-pieeprom-2020-09-03.bin
# and reboot
sudo reboot

The 0x412 option will look for a TFTP server to boot from via DHCP, then fall back to the SD card, and then fall back to booting from USB. See here for documentation. The default option (as of the 2020-09-03 firmware) is 0x41, which unfortunately doesn’t even try to use netbooting.

Some other guides I found suggest that you disable the automatic update for the eeprom, in order to prevent it from disabling the netboot setting, but this should not be needed. The documentation says:

If you update your bootloader via apt, then any configuration changes made using the process described here will be migrated to the updated bootloader.

Once netbooting is enabled, the remainder of this guide should work just fine. As far as I can find, this is still the only guide to netbooting a Raspberry Pi that doesn’t assume you’re using another Raspberry Pi as a TFTP server.

One change I had to make is that I’ve switched from using a consumer grade router with custom Tomato-based firmware to using my own hardware with OPNsense. The same guidance I provide for how to get your DHCP server to send the PI to the TFTP server applies, but it seemed to be necessary to use the “Text” type for the server IP address field, not the “IP address or Host” option. Your milage may vary.

The original guide, for the Raspberry Pi 3:

You can find plenty of guides to booting a Raspberry Pi from the network, but the one’s I’ve seen make one or two assumptions. One is that the official guide assumes that you’re booting off of another Raspberry Pi, which you use to generate the boot folder. Another is that you’ll be providing DHCP and TFTP on the same server, e.g. using dnsmasq. This is annoying because you end up running two DHCP servers on the same network, and it has the potential to interfere with the network configuration on the server that’s running it.

So here’s my plan. I’ve already got a Tomato router (running Advanced Tomato), and it’s got dnsmasq as a DHCP server. So in theory I should be able to direct the Pi to my real server, which I’m going to run tftpd-hpa on. The same server also runs my NFS mounts, so I’ll use it to host the root directory for the Pi as well.

I expected this to be simple, but it turned out to be rather hard. There are a lot of moving pieces, and many of them don’t give you much help when they break. So I’m documenting the process here.

What you need to do this:

  1. A server to host NFS and TFTP
  2. A router with a configurable DHCP daemon, e.g. dnsmasq (Tomato-based routers are great for this.)
  3. A Raspberry Pi 3 (the first model to support netbooting without an SD card) with a working Raspbian installation on an SD card
  4. A working NFS setup (not hard to do, but not described here)
  5. Possibly a lot of patience. (This is very much not a step-by-step guide, I assume that you know how to find the tftpd config file yourself, know how to configure your network’s DHCP server, and so on and so forth. I’m simply documenting how to configure things.)

The first, and apparently most difficult part, is just getting a working TFTP server. I installed tftpd-hpa on Ubuntu 17.10, made sure port 69 was unblocked, did some basic configuration. No luck, even over localhost. After some mucking about, I could tell using -v -v -v with tftpd and journalctl that tftpd was receiving the request, but it never bothered to respond—and it didn’t say why. After hours of trying different suggestions online, I managed to get a working configuration with the following:

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/path/to/boot/"
TFTP_ADDRESS=":69"
RUN_DAEMON="yes"
TFTP_OPTIONS="--secure -p"

I’m not sure if the -p is needed, but honestly I’m too scared to change anything now. Some of these settings are almost designed to cause frustration. --secure isn’t even a security setting, it sets the “default” directory to TFTP_DIRECTORY and translates all addresses to subdirectories of that one. Yes, that’s right, without --secure, a request for a relative path won’t work. My /boot files are owned by root, umask=022, and that seems to work fine.

The main thing you need to do for TFTP is set the directory to the path you want to use for the Pi’s boot files. Since I have the NFS server and the TFTP server on the same hardware, I can conveniently point the TFTP server at /boot in the Pi’s root directory under NFS. (Note that you’re making these files available via the TFTP server, so you want to use the real path to where the boot files are located on your file system, not the place they’re made available over NFS.)

But first things first, we have to get the root directory onto the server first. The official guide is rather useless here. It assumes that we’re going to run the server off of another Pi, which we can conveniently copy the root from. I went the opposite direction. I set up the NFS server with the options

(rw,no_root_squash)

and, mounting the folder on the client Pi (with Raspbian installed to an SD card), copied the root with rsync to the NFS folder. Conveniently enough, we can use the exact same folder and settings to point the Pi to later. (Having virtually no security on the root folder of the Pi’s filesystem is pretty concerning, however. I haven’t figured out how to get around this given the limitations of PXE booting on the Pi. You could at least limit the IP addresses which the NFS server will answer to, but I do only recommend running this on a carefully secured subnet.)

Alright, so I now have a working Raspbian filesystem served over NFS, and I can point my TFTP server to the /boot directory as previously described. I do need to edit /boot/cmdline.txt as described in the official guide to point the boot code towards the NFS server. Also best to edit /etc/fstab and remove everything but /proc, otherwise Raspbian thinks the boot didn’t succeed.

# cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs 
    nfsroot=10.0.1.102:/path/to/root,vers=3 rw ip=dhcp rootwait elevator=deadline

The bit starting with “root” is the important part. We need to point the boot code to the directory with the Pi’s root on the NFS server, as shown.

The remaining task is to get the DHCP server on the network to respond to the PI’s PXE discovery request. I fought dnsmasq on Advanced Tomato for a long time, and finally settled on setting the appropriate DHCP fields by hand, as established in RFC 2132.

dhcp-option=43,Raspberry Pi Boot   
dhcp-option=66,10.0.1.102

Where the IP address belongs to the TFTP server. There are three spaces after “Boot” as suggested in the official guide, but I don’t know if they’re actually necessary. Surprisingly enough, these two options alone were enough to get it working—I just set them on the DHCP page of Advanced Tomato.

The last step is just to make sure booting over ethernet is enabled on the Pi, as the official guide says (or see above for the Pi 4). After that, removing the microSD card and rebooting, everything should just work!

©2024 Adam Fontenot. Licensed under CC BY-SA. About Me Projects RSS Feed