Blog

If you’re using Proxmox and you want to back up your VMs & containers there is no better way than Proxmox Backup Server (PBS). If you have a spare PC or motherboard and storage (SSD or spinning rust) you can use it to get regular scheduled backups, whatever retention you need, backup verification, deduplication and all that good stuff. The best bit? You can use if for free (paid support subscriptions available)!

So, you set up your backup server PC, set a daily backup schedule to run, then what? Your backup takes maybe half an hour each day to run … so for 23.5 hours that backup server is just spinning away burning electricity.

There is a better way!

Here’s how you can use Wake-on-LAN (WOL) to automatically wake your backup server and let it shut down gracefully after each backup, saving you electricity and cash.

Set up PBS

First we need to set up the backup server for Wake-on-LAN. To do this you will first need to change the BIOS settings. Where the setting is can vary depending on your motherboard’s BIOS, but you’re looking for something that says WOL, wake on LAN or wake on PCI. Turn that stuff on.

For some network cards, however, that’s not enough. You have to specifically enable wake-on-LAN via the driver before the machine shuts down, or it won’t wake.

Here’s how you do that. First, install the net-tools package:

apt install net-tools

Next we have to create a service that enables wake-on-LAN on the interface when the PC starts up. In this case the interface is called eno1, make sure you change it to whatever yours is called (find out from eg ip a command):

cat <<'EOF' >  /etc/systemd/system/wol.service
[Unit]
Description=Wake-on-LAN for eno1
Requires=network.target
After=network.target

[Service]
ExecStart=/usr/sbin/ethtool -s eno1 wol g
Type=oneshot

[Install]
WantedBy=multi-user.target
EOF

systemctl enable wol.service
systemctl start wol.service
systemctl status wol.service

So that covers the “waking up” bit … now we need to cover the shutdown when backup job is complete. Here’s a script that covers that. It will wait for awhile for a backup job to start, then wait for the job to complete, then once no jobs are running (and no-one logged in) it will do some cleanup tasks (verify all backups and upgrade packages) then shutdown the server:

cat <<'EOF' > /usr/local/bin/shutdown-nojobs.sh
#!/bin/bash

# First we wait for the first job to start. We don't want to shut down before the first backup has even started!

echo "Starting script!"

count=0

while [[ $count -lt 5 ]] do
        if [[ $(/sbin/proxmox-backup-manager task list | /bin/grep 'running') == *running* ]]; then
          echo "$(/bin/date +%F_%T) Found a backup job running!"
          count=100
        else
          count=$(( $count+1 ))
          echo "$(/bin/date +%F_%T) Waiting for first backup job to start - count: $count"
        fi
        sleep 10
done

# Next we wait until we have 20 count of no task running at 5 second intervals (hopefully all jobs completed)

echo "$(/bin/date +%F_%T) Waiting for all jobs to complete"

count=0

while [[ $count -lt 20 ]] do
        if [[ $(/sbin/proxmox-backup-manager task list | /bin/grep 'running') == *running* ]]; then
          echo "$(/bin/date +%F_%T) Found a backup job running."
          count=0
        elif [[ $(/bin/who | /bin/wc -l) != 0 ]]; then
          echo "$(/bin/date +%F_%T) Found a user logged in."
          count=0
        else
          count=$(( $count+1 ))
          echo "$(/bin/date +%F_%T) Idle. Countup (to 20): $count"
        fi
        sleep 5
done

echo "$(/bin/date +%F_%T) Done. Verifying store."

/sbin/proxmox-backup-manager verify store1 --outdated-after 30

echo "$(/bin/date +%F_%T) Done. Updating packages."

/usr/bin/apt update
/usr/bin/apt upgrade -y
echo "$(/bin/date +%F_%T) Done. Shutting down."

/sbin/shutdown now
EOF
chmod 770 /usr/local/bin/shutdown-nojobs.sh

Now we want to make that script run when the PC boots:

crontab -e

This will open your crontab file with your favourite editor. Add this line to the bottom:

@reboot /usr/local/bin/shutdown-nojobs.sh >> ~/shutdown-nojobs.txt

Lastly you need to find the MAC address of the network card in the backup server that will be used to wake the box.

root@prox-backup:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a8:a1:59:e1:55:f4 brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 192.168.1.220/24 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::aaa1:59ff:fee1:55f4/64 scope link
       valid_lft forever preferred_lft forever

In this example the MAC of our wake-on-lan interface is a8:a1:59:e1:55:f4.

Now reboot the backup server. It should come up, then shut down again after a short while.

Note that if you SSH in to the server while it’s up, it won’t shut down until you close the SSH connection, so if you need to do any maintenance on the server that’s an easy way to keep it up.

Now we need to add some script on the Proxmox server to wake the backup server when a backup job starts.

First let’s install the wakeonlan package

apt install wakeonlan

You can test waking the backup server like this (replace MAC address of the target server):

wakeonlan a8:a1:59:e1:55:f4

The backup server should boot:

Great!

Now let’s add some automation to your backup job to start the backup server when a job starts. Modify the script below with the MAC address of your server, hostname or IP of the PBS server and name of the backup server datastore as on the proxmox server:

cat <<'EOF' > /usr/local/bin/vzdump-hook-script
#!/bin/bash

if [ "$1" == "job-init" ]; then
  /usr/bin/wakeonlan a8:a1:59:e1:55:f4   # <-- edit with mac Address of PBS Server

  while [[ $count -lt 30 ]] do
        # if [[ $(/sbin/proxmox-backup-manager task list | /bin/grep 'running') == *running* ]]; then
        if [[ $(/bin/ping pbs.mydomain.com -c 1 | /bin/grep '1 received') == *received* ]]; then  # <-- edit with hostname or static IP of your server
          echo "$(/bin/date +%F_%T) Backup server up!"
          count=100
        else
          count=$(( $count+1 ))
          echo "$(/bin/date +%F_%T) Waiting for backup server up - count: $count"
        fi
        sleep 10
  done
  # Give backup server a chance to get store online
  sleep 10
  # Enable data store
  /usr/sbin/pvesm set pbs-store1 --disable false # <-- edit with name of the datastore on proxmox server
  sleep 10
  echo "Backup datastore status is: $(/usr/sbin/pvesm status 2> /dev/null |grep 'pbs-store1' | awk '{print $3}')" <-- edit with name of the datastore on proxmox server

fi

if [ "$1" == "job-end" ]; then
  sleep 10
  /usr/sbin/pvesm set pbs-store1 --disable true <-- edit with name of the datastore on proxmox server
fi

exit 0
EOF

For the script to run it must be set executable:

chmod 755 /usr/local/bin/vzdump-hook-script

Next we need to tell Proxmox to run this hook script each time a backup fires. There are a couple of ways to do this – either add the following line to the global /etc/vzdump.conf configuration where it will apply to all backup jobs, or you can add it in /etc/pve/jobs.cfg where it will apply to only the job[s] you specify. That’s useful for example if you also want to define jobs that back up to locally attached storage.

script /usr/local/bin/vzdump-hook-script

Finally, make sure the backup server is shut down then try to run a backup on the Proxmox server. You should see something like this in the backup job log:

INFO: Sending magic packet to 255.255.255.255:9 with a8:a1:59:e1:55:f4
INFO: 2024-08-21_10:07:37 Waiting for backup server up - count: 1
INFO: 2024-08-21_10:07:50 Waiting for backup server up - count: 2
INFO: 2024-08-21_10:08:00 Backup server up!
INFO: Backup datastore status is: active
INFO: starting new backup job: vzdump --storage pbs-store1 --all 1 --fleecing 0 --mailto [email protected] --mode snapshot --script /usr/local/bin/vzdump-hook-script --node prox --notes-template '{{guestname}}' --mailnotification always
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2024-08-21 10:08:32
INFO: status = running
INFO: VM Name: dt-11
INFO: include disk 'ide0' 'local-lvm:vm-100-disk-4' 128G
INFO: exclude disk 'ide2' 'local-lvm:vm-100-disk-6' (backup=no)
INFO: include disk 'efidisk0' 'local-lvm:vm-100-disk-2' 528K
INFO: include disk 'tpmstate0' 'local-lvm:vm-100-disk-5' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2024-08-21T09:08:32Z'
INFO: enabling encryption
.
.
.
.
.
INFO: Finished Backup of VM 430 (00:00:04)
INFO: Backup finished at 2024-08-21 01:49:03
INFO: Backup job finished successfully
INFO: notified via target `<[email protected]>`
TASK OK

Nice work! Enjoy your new automated, energy-saving backup solution!

TL;DR A default Proxmox 8 / Debian 12 install uses the wrong module for the RTL8111H – we need to build & install the correct module

I came across this problem using Proxmox on the Asrock J4125-ITX / J5040-ITX / N100 series boards. These are great boards to run a low-power mini server, they run Proxmox really very well, but a default install will leave you thinking the onboard NIC is faulty. Here’s why.

The Realtek RTL8111H NIC used on the Asrock boards is a cheap but capable Gigabit Ethernet (10/100/1000M) PCI Express device. Now I’ve used boards with on-board Realtek NICs for years without incident on Windows machines, but turns out Linux support for some of these chipsets is sub-optimal.

So you’ve done a default install of Proxmox. Your network is up, you have an IP, everything’s working, right? Wrong!

lspci -nnv

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
...
Kernel driver in use: r8169
Kernel modules: r8169

When we take a look at the RealTek website, we can see that the R8169 is part of a family of Realtek NICs sharing the same driver:
RTL8110S/RTL8169S/RTL8110SB/RTL8169SB/RTL8110SC/RTL8169SC

Now we know that our board is, in fact, sporting an RTL8111H, which isn’t in that list.

A little sleuthing and we can find a similar, but different family containing:
RTL8111B/RTL8111C/RTL8111D/RTL8111E/RTL8111F/RTL8111G(S)/RTL8111H(S)//RTL8118(A)(S)/RTL8119i/RTL8111L/RTL8111K
RTL8168B/RTL8168E/RTL8168H
RTL8111DP/RTL8111EP/RTL8111FP
RTL8411/RTL8411B

There’s another clue on that page in the driver name: GBE Ethernet LINUX driver r8168 … So Linux has chosen the wrong kernel module to support this hardware. It sort-of works, because the hardware is kinda similar, which is hella confusing, but you’ll run into all sorts of nasty behaviour if you don’t fix it.

RealTek supply a drivers for kernel up to v6.8, Proxmox 8 currently uses 6.8.8-4-pve and it’s working fine. Go ahead, download the file (behind a captcha, so you’ll need to use a browser):

GBE Ethernet LINUX driver r8168 for kernel up to 6.8, v8.053.00, 2024/04/18
https://www.realtek.com/Download/List?cate_id=584

You should get a file named r8168-8.053.00.tar.bz2. Copy it to /tmp on the Proxmox / Debian box (I used scp for this, if you’re coming from a Windows box WinSCP is a great option).

If you’ve just installed Proxmox and you don’t have a subscription, you’ll need to change to the free repositories. Don’t do this if you do have a subscription – instead enter your key in the Proxmox UI to enable the enterprise repositories.

echo "deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription" > /etc/apt/sources.list.d/ceph.list
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-enterprise.list

Do a quick update and install headers and build essentials

apt update
apt upgrade
apt install pve-headers build-essential

Next let’s extract the driver package

cd /tmp
tar xf r8168-8.053.00.tar.bz2
cd r8168-8.053.00/

Let’s check the module compiles OK:

make modules

If everything goes right you should get the following with no major errors:

make -C src/ modules
make[1]: Entering directory '/tmp/r8168-8.053.00/src'
make -C /lib/modules/6.8.8-4-pve/build M=/tmp/r8168-8.053.00/src modules
make[2]: Entering directory '/usr/src/linux-headers-6.8.8-4-pve'
CC [M] /tmp/r8168-8.053.00/src/r8168_n.o
CC [M] /tmp/r8168-8.053.00/src/r8168_asf.o
CC [M] /tmp/r8168-8.053.00/src/rtl_eeprom.o
CC [M] /tmp/r8168-8.053.00/src/rtltool.o
LD [M] /tmp/r8168-8.053.00/src/r8168.o
MODPOST /tmp/r8168-8.053.00/src/Module.symvers
CC [M] /tmp/r8168-8.053.00/src/r8168.mod.o
LD [M] /tmp/r8168-8.053.00/src/r8168.ko
BTF [M] /tmp/r8168-8.053.00/src/r8168.ko
Skipping BTF generation for /tmp/r8168-8.053.00/src/r8168.ko due to unavailability of vmlinux
make[2]: Leaving directory '/usr/src/linux-headers-6.8.8-4-pve'
make[1]: Leaving directory '/tmp/r8168-8.053.00/src'

If all looks good, we can run the convenience script from RealTek that will unload the r8169 module and install the newly compiled one in its place.

For the next step you will want to be on a physical console or IP KVM on the box … you’re going to be replacing the network card driver, which will drop connectivity, so don’t try to do this over an SSH connection (unless you’ve installed a second NIC for the purpose)!

chmod 755 autorun.sh
./autorun.sh

Check old driver and unload it.
rmmod r8169
Build the module and install
Skipping BTF generation for /tmp/r8168-8.053.00/src/r8168.ko due to unavailability of vmlinux
Warning: modules_install: missing 'System.map' file. Skipping depmod.
Backup r8169.ko
rename r8169.ko to r8169.bak
DEPMOD 6.8.8-4-pve
load module r8168
Updating initramfs. Please wait.
update-initramfs: Generating /boot/initrd.img-6.8.8-4-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Completed.

For good measure let’s blacklist the r8169 module to prevent future problems:

echo "blacklist r8169" >> /etc/modprobe.d/pve-blacklist.conf

Final check with lspci:

lspci -nnv

03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
...
Kernel driver in use: r8168
Kernel modules: r8168

Nice!

Now reboot your box and make sure everything works as it should.

If you can’t ping etc from the box, check the interface name of the RealTek NIC, it might have changed and you might need to update the bridge-port name for vmbr0

ip a

....
2: eno1: mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
....

nano /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
  address x.x.x.x/24
  gateway x.x.x.x
  bridge-ports eno1
  bridge-stp off
  bridge-fd 0

That’s it! Enjoy your swanky Proxmox box to the full!