Recent Network Changes

This section of this website follows the life-cycle of this home network project that I have been working on. And this section specifically covers a brief summary of these recent changes. But this entire page is dedicated to these recent modifications that were made in more detail. I was working on this projects website and decided to take a pause and work on some gaps in the home network that I felt could be staged for future changes or out-right fixed without much purchasing of new equipment.

What is Changing?

Below is a before and after of where I started before this section and where I ended up after a couple days of tinkering. You can find a brief summary of these networks here and here for an overview of each with larger diagrams.

The key changes are as follows:

1.) Migrate the hardware (mobo, CPU, RAM, spare NVMEs) from the dormant R3 system to my main server and maintain the boot drive and ZFS pool

• Clean the system internals
• Replace fans
• Cable manage/fix some of the internal clutter and case quirks
• Update BIOS + re-enable XMP
• NVME -> SSD boot drive cloning with dd and Ubuntu live boot

2.) Prepare the now dormant i7 (old server) system for donation or future projects

3.) Decommission the Raspi5 and AdGuard for backup server preparation

4.) Confirm the new i7 -> R3 server migration works and all processes come back up (Nextcloud/samba)

5.) VLAN network segregation (physical segregation due to 802.1Q limitations on equipment)

• TPLink Switch @ 192.168.10.0/24 (trusted VLAN)
• Spare Switch @ 192.168.20.0/24 (untrusted devices VLAN - staged)
• Rescue LAN @ 192.168.0.0/24 (left in place as a backup strategy)
• Configure ACLs for 192.168.20.0/24 through 192.168.10.0/24 to confirm DNS will still pass beyond this VLAN and all other connectivity is blocked
• Repoint fstab to new IPs on my desktop to mount samba drives

6.) Reinstall AdGuard on the R3 Ubuntu Server as its permanent home

7.) Build a wireless "remote" backup system with the Raspi5 and external Sabrent dock

8.) Install AdGuard on the Pi5 server as a secondary DNS

Hardware Migration

Initial Hardware Swap and Cleanup

I had two systems in my rack. A dormant Ryzen 3 3200G machine (left) and a recycled i7 2700 system that needed to be combined with one another (right). The i7 was my functioning server that needed to be taken off-line and functionally merged with the dormant Ryzen 3 machine while keeping the boot and storage drives intact. This was shaky territory for me as most of my hardware migrations to this point were done in Windows systems with mixed results. Below is a picture of both systems with the cover off. The chassis on the right is the final home for for my real server while the left side chassis will go dormant for now. Hardware decisions might look suspect, but as I mentioned elsewhere, these are all recycled parts and a hodge podge of components. I will admit these mini-ITX boards do look out of place in these chassis' especially next to an over sized AX1200 PSU. This was my first look into these machines for almost two years, and they are little bit messy internally.

I started on the final home machine first (the one on the top right in the above image), this chassis is interesting. It supports SFX PSUs, which I do not have, and ATX PSUs with a passthrough cable baked into the chassis. The passthrough cable has a 90 degree connector that is the wrong orientation. So I ran a standard cable from the front PSU power ports, through the case, and out of the SFX opening. You can see below the fitting it came with and the female passthrough port on the back. I ended up cutting this out completely, there was no way to unscrew it and slide it out, it is as if it was wired and terminated in place. In the end I 3D printed two pieces for the back to cover the old female power port and to cover the SFX opening with my less than perfect power solution hanging off the back of the machine.

There is a long central bracket in this build that serves as front to rear cable passthrough and a mount for the fans. Last time I was in here I decided to mount the fans towards the front, but with the mini-ITX motherboards there was plenty of space to swap fans to the other side of the bracket. You can see how tight that is (top left image) I could not even slide them out. I unscrewed the fan bracket, removed the old loud case fans that came with this case, and migrated the 4x Noctua fans to this build. I had to do a dry fit (bottom left image) to make sure the SATA cables could be ran through the right side channel cutout and still reach the motherboard. Last time I built this I only had two case fans, so I used the leftover fan openings to route SATA cables. I just wanted to confirm the cable channel and four fans would all work, and fortunately the SATA cables I had on hand were good to go. The bottom right image is everything in place. Fans fit, SATA cables reach, and my 3D printed covers for the rear are in. I also removed the empty front drive bay and put it aside for now, it is unused.

Hard Drive Install Limitations

So before I could continue, I ran into a drive problem. My original boot drive for the server in the old i7 machine that needed to be migrated over was NVME (240G). This A520 board I am utilizing has one NVME slot. I put it in, and everything just worked in a quick test that was performed, which was my first hurdle. And I have a four slot NVME PCI-E riser card that I was going to put in this machine and install the two spare 1TB NVMEs that were sitting in the dormant machine in addition to the two 1TB SATA SSDs that were in use (4TB of storage drives was the final goal). I went down a PCI-E bifurcation rabbit hole with this A520 board and the 3200G CPU. Everything I read said this board should have a BIOS setting to set that PCI-E slot in a 4x4x configuration to to split the lanes out for each drive on the riser. The plan to do 2x 1TB NVMEs on the riser, 1x 240G NVME in the motherboard (original boot drive), and 2x SATA SSDs (maintained) was not going to work. Turns out "G" SKU CPUs will disable this setting, despite that PCI-E slot having eight dedicated PCI-E lanes with a "G" SKU CPU, I think AMD dropped the ball on this one. So to get my final configuration to work, I would need to sit one 1TB NVME in the PCI-E riser card and the other in the motherboard, maintain the two 1TB SSDs as is and then clone the original 240G NVME boot drive to a spare 240G SATA SSD. I have two open SATA ports available and fortunately I had a spare 240G SSD in storage. So I was excited to boot up Clonezilla and try this out.

Clonezilla was not looking so hot for whatever reason, which is a shame because I've read a lot about it over the years and never got a chance to try it. Instead I went ahead and made an Ubuntu live boot drive, I have the 240G NVME internally and the 240G SATA SSD plugged into an external USB 3.0 adapter. We are going to use dd to clone these drives, last time I attempted this a few years back I broke something so I had an LLM check me on this one and the final clone trail looked something like this:

# identify disks
sudo lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE,LABEL,MOUNTPOINT
sudo pvs; sudo vgs; sudo lvs

# confirm capacities match - in my case this was perfect (if not I would have needed to adjust them)
sudo blockdev --getsize64 /dev/nvme0n1 /dev/sda
# 240057409536
# 240057409536

# resolve actual ids for each drive - in my case I am using SanDisk/Patriot drives
ls -l /dev/disk/by-id/ | grep -E "SanDisk|Patriot"

# unmount the target drive and confirm
sudo umount /dev/sda1 2>/dev/null
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT | grep -E "sda|nvme"

# the actual clone itself (source and destination)
sudo dd \
  if=/dev/disk/by-id/nvme-Patriot_M.2_P310_240GB_P310IDBB24081500474 \
  of=/dev/disk/by-id/ata-SanDisk_SDSSDA240G_174377804035 \
  bs=64M status=progress conv=fsync

# read the partition tables and verify they match
sudo partprobe /dev/sda
sudo sgdisk -v /dev/sda
lsblk -o NAME,SIZE,FSTYPE,LABEL /dev/sda

# power off and remove the original source (NVME) drive, boot into cloned drive

# verification after fresh boot
lsblk
findmnt /                      # expect /dev/mapper/ubuntu--vg-ubuntu--lv
sudo systemctl --failed
sudo zpool status

So this actually worked perfectly. I ran a quick boot test with no issue to the newly cloned SSD. Now I can install the newly cloned SSD as my new boot drive. The 240G NVME was marked as a "last good config" drive and put into storage in the case that something goes wrong during this entire process. The next problem was finding a place to mount this SSD. There are holes to mount this to the underside of the existing drive bay, but it is not a great design as the cables would be sandwiched up against the steel and the drive would be buried too deep into the bay (left image). I was going to make an adapter to slide this out to expose the connectors on the SSD, but in the end I made a side mount bracket to the outer wall of the existing drive bay (right image).

At this point I had the motherboard in, cables loosely in their correct location, but then it was really just getting the 1TB NVMEs installed. One in the riser and one in the motherboard.

Final Configuration

Below is the final configuration before putting the cover on and sliding it back into the rack. Night and day difference from before.

And for the other server, I am hoping to donate this to someone who needs it, for now it will just slide back into my 14U rack until I find a home for it. Might seem odd that I don't use this other i7 server for something, especially considering I build a backup server out of a Raspi5 later on in this section. The Raspi backup server by design should not be located in the same physical location as the source data. Additionally, another full server that is always on will bring some unwanted noise. Lastly, the chassis is in pretty rough shape and I don't have the tools to fix it. It was delivered to me with a considerable warp, I couldn't even get the I/O shield in. Couple pictures below of the state I left this in and the warp that is visible.

Quick BIOS Tweaks

Last thing I need to do is to get my R3 up, BIOS updated, and make some BIOS tweaks. Not many tweaks, just XMP enabled, and I forced the iGPU to only use a max of 64MB of RAM as the iGPU might hog some system memory, rather be safe than sorry.

Because the network adapter changed from motherboard to motherboard I did need to hop into my router and re-reserve the new MAC address and delete out the i7 MAC entry. Then I just made sure my SMB share was mounting and I had access. Beyond that, server hardware migration is completed.

Network Segregation (VLAN)

I was reading through a networking book recently and the author (Todd Lammle) had mentioned that most networks are just a slew of VLANs and the default was sometimes left in place like an access port/rescue port which gives access across all VLANs or something similar (reserved for administration). The plan is to add two VLANs 192.168.10.0/24 (trusted devices) and 192.168.20.0/24 (untrusted devices) to my network and leave 192.168.0.0/24 in place from the router port that I can patch a laptop into in the case I somehow lock the other VLANs from accessing network devices. In doing this I will be breaking two main network services which is my Ubuntu storage server and my AdGuard DNS. One other mention, my reserved IPs for servers and such should really fall into the .2-.99 range and non-static in the .100-.199 range, this is something I realized was configured incorrectly some time ago but these IPs are burned into my muscle memory and it would be a headache to change at this point.

Omada's Router Navigation

I've mentioned this before in this writeup, but my switch does not really support tagging/trunking/802.1Q. So this is all physically separated. I have a spare un-managed PoE switch that we are going to use for 192.168.20.0/24 VLAN and my TPLink switch for 192.168.10.0/24 fed with two separate un-tagged Cat6a cables coming directly off of the router ports. I set up the VLANs like such in my router:

VLAN10 and VLAN20 got the same treatment except VLAN20 I selected the "Network Isolation" check box. This check box did not exist prior to a recent firmware update. The issue is, even on this network with this selected, you can still login to the router through the web UI, which I do not want for this VLAN. Also, notice my primary and secondary DNS are set to my AdGuard Servers. Those are on a separate VLAN, so we will have some configuration on the ACL side to get this to work the way I'd like.

Then under the VLAN section of the router interface you can see all are shown on their designated port, and left un-tagged.

Then onto ACLs to block VLAN20 from accessing the gateway and allowing it to access the local DNS servers. This got a little weird, Omada's GUI is sort of all over the place. I had to start at Preferences -> IP Group -> IP Address and added individual IPs for both DNS servers and the full IP range for each VLAN. Technically VLAN1 and VLAN10 can be left off, but I added them for good measure.

Then selected the IP Group tab and I added each VLAN to its own individual group and and allocated both DNS servers to the group that I called G_ADGUARD_1.

Now finally we can go to Firewall -> Access Control to configure rules. Order matters and I found out the hard way that you can re-order them after you've already added a rule under the optional "ID" section and putting the number in that box that you want it to appear in the list. So ID 1 and ID 2 will allow DNS (TCP/UDP) requests through to VLAN10 so we can still get the benefit of AdGuard on this VLAN and ID 3 blocks access to the gateway so it cannot access router logins.

This process was not very intuitive, I think it is okay as it stands though. I had my Windows laptop plugged into the VLAN20 switch while I was playing with all of these settings. Below you can see in this configuration, my laptop is on the correct VLAN20 (192.168.20.100), both DNS servers are assigned on VLAN10, and I am able to resolve host names perfectly fine. Additionally I cannot ping any of the VLAN gateway IPs as I intended.

This portion is now completed. I am not sure what will happen on the hardware side in the future to my network, a full switch with tagging/trunking capabilities might cause me to reconfigure some of this, but the router side will be left mostly un-changed either way.

Some AdGuard Notes

I wrote this a little bit out of sequence, when I did this initial VLAN setup, I set my DNS to 1.1.1.1 temporarily while I was setting up VLANs knowing my DNS IP would be invalid for the time being. I reinstalled AdGuard on the R3 Ubuntu server right after, and then made a DNS fall back mirrored service on the Raspi5. I don't have much to say about AdGuard. There is a perfectly fine install guide here https://adguard-dns.io/kb/adguard-home/getting-started/. Only thing is, if you are configuring this on a headless machine with no DE/web browser then the initial setup wizard step needs to be done from a remote machine with a browser using the host machines IP and port 3000 just for the initial setup. My config file is located at /opt/AdGuardHome/AdGuardHome.yaml below are really the only changes I made:

 #DoH upstreams
 upstream_dns:
    - https://dns.quad9.net/dns-query
    - https://dns.digitale-gesellschaft.ch/dns-query
    - https://dns.mullvad.net/dns-query

#these are the actual filters that will catch telemetry/ad services
filters:
  - enabled: true
    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt
    name: AdGuard DNS filter
    id: 1
  - enabled: true
    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt
    name: AdAway Default Blocklist
    id: 2
  - enabled: true
    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_33.txt
    name: Steven Black's List
    id: 1788871889
  - enabled: true
    url: https://easylist.to/easylist/easylist.txt
    name: EasyList
    id: 1788871891
  - enabled: true
    url: https://easylist.to/easylist/easyprivacy.txt
    name: EasyPrivacy
    id: 1788871892

I have played with a couple different fallback and bootstrap DNS servers as well, but for the most part the above is the meat of it. quad9, mullvad, and digitale are the sort of "ultra private" type of companies that offer DoH so I decided to go down that road for the time being. If it was not obvious, the TVs are my main concern right now due to some recent security news. The TVs were priority number one to isolate using hardwired connections. I previously had these on the GuestSSID and AdGuard was filtering just fine, but a compromised TV could allow lateral movement through a network if it had been kept on my private VLAN.

Raspi5 Wireless Backup Server

So now that the Raspi5 was freed up from the network, I decided to bring it back in as a backup server. Currently it is a wireless on-site solution, but the ultimate goal is to take this off-site somehow, or at least have the option to do so with some tweaking. Up to this point I have had no remote/external backups. I have a Raspi5 connected to a 4-bay external dock that is populated with three 1TB HDDs in a raidz1 configuration. This device deserves an entire separate section on this projects website, so I will keep this section sort of brief. For transparency, I got stuck building and configuring this device. I looked for some guides online, the best one I found was clearly AI written and did not fulfill what I wanted. I also went back and forth on the solution. Borg backup vs a simple rsync vs sanoid/syncoid. I ended up with sanoid/syncoid and I am still not totally sold on it and this backup box could get a facelift in the near future.

Hardware Configuration and HDD Health

I purchased a 4-bay external drive dock over USB 3.0 from Sabrent, I got this on sale for $90. Then I had three spare 1TB HDDs sitting in storage. Absolutely no idea where these came from originally except for one. I ran smartctl against all three drives that I wanted to use and found the following:

===== /dev/sdb =====
Device Model:     WDC WD10EAVS-00M4B0
Serial Number:    WD-WCAV5A616467
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   057   057   000    Old_age   Always       -       31920
193 Load_Cycle_Count        0x0032   101   101   000    Old_age   Always       -       299784
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
===== /dev/sdc =====
Model Family:     Western Digital RE3 Serial ATA
Device Model:     WDC WD1002FBYS-02A6B0
Serial Number:    WD-WMATV2765462
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   031   031   000    Old_age   Always       -       50702
193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       8386
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
===== /dev/sdd =====
Device Model:     ST1000NM0053-1C1173
Serial Number:    Z1W1L7LS
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   045   045   000    Old_age   Always       -       48374
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   099   099   000    Old_age   Always       -       2078
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0

Interestingly IDs 5, 197, 198, and 187 should ideally all read 0, and they do which is good. ID 9, hours, look pretty alarming, these drives are old, but I would say old and healthy. Turns out drive sdd the Seagate drive is an enterprise spec. It sat somewhere for a very long time and had very little load cycles (ID 197) which indicates head parking. This is the healthiest drive of the bunch. And the same goes for drive sdc the Wester Digital HDD, this is an enterprise spec with very little load cycle count that racked up 50k hours on. sdb looks to be a different story, this drive has the least power on hours, but the most load cycle count. Turns out this drive specifically has a feature called IntelliPark that will park the head every eight seconds which caused it to rack up a ton of cycles despite it being the lowest power on drive. There is an interesting read about this here. But basically on Windows, this is a non-issue, this drive in particular came out of a pre-built NAS device that was being discarded and was probably running some form of Linux under the hood, I grabbed all drives from the device originally and found that two out of three of the drives were dead, but this one survived. This is the shakiest drive of the bunch. In the end I used a raidz1 configuration which is similar to RAID5. If one drive dies, I am still good and we can rebuild through the zpool replace command if a failure occurs.

A quick note about the Sabrent dock. This dock has a cloning feature built in. There are safeguards on the dock so you don't accidentally clone the drives in place and destroy a storage pool. There is a clone toggle switch that needs to be turned on, a dock reboot needs to be done, and then the clone button needs to be pressed twice. Just worth mentioning this dock feature could be a danger area.

Software Configuration and Initial Setup

My main server is running Ubuntu Server 26.04 LTS with a ZFS file system on the storage drives and the Pi ended up getting a copy of Ubuntu Server 26.04 LTS ARM. I believe that my server came with ZFS packaged with the system but the Raspi needed it installed along side the kernel module with

sudo apt install --yes zfsutils-linux zfs-dkms

then running zfs --version on both machines is wise because a version mismatch can cause issues. In my case I am running ZFS 2.4.1 on both machines.

The Backup/Snapshot Holes at the Ryzen Server

My Ryzen Ubuntu Server had a simple zfs storage tank already configured from previous version of my setup and was as follows

NAME             USED  QUOTA  MOUNTPOINT
tank             259G   none  /tank
tank/mike        251G   450G  /tank/mike
tank/misc        128K   200G  /tank/misc
tank/nextcloud  3.94G   200G  /tank/nextcloud

mike and misc are reserved for SMB local shares and Nextcloud for the Nextcloud storage side.

But I did not have the snapshot scheduler (sanoid) configured up to this point which would simply allow me to recover a deleted file or something similar from the same storage tank. ZFS out of the box supports snapshots natively but the sanoid side is the actual scheduler that decides what to keep/pin on the drives and what to prune and when to do it. So sanoid needs to be installed then a pointer directory with a symlink to follow

sudo apt install sanoid
sudo mkdir -p /etc/sanoid
sudo ln -s /usr/share/sanoid/sanoid.defaults.conf /etc/sanoid/sanoid.defaults.conf

that last command is a symlink because we installed from apt whereas the github README expects an install from a gitclone and simply uses the cp copy command at this stage. I got a little confused around here, I think its really a quirk when installing from apt is all.

Then onto the actual config file for snap frequency and rules around the existing datasets which will be created be created with the following

sudo nano /etc/sanoid/sanoid.conf

And the config I used is

[tank/mike]
        use_template = production
[tank/misc]
        use_template = production
[tank/nextcloud]
        use_template = production

[template_production]
        frequently = 0
        hourly = 24
        daily = 14
        weekly = 4
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

This will change slightly overtime due nextcloud having a built in borg backup system in place I needed to fit this around. But other than that it is just a matter of enabling sanoid's timer that will wake up every 15m to check the above ini file, read through it, and decide what to do with snapshots (pin, prune, etc.)

sudo systemctl enable --now sanoid.timer #only required if a reboot did not happen

and you can see timer status with systemctl list-timers sanoid.timer which should return something like

NEXT                        LEFT LAST                          PASSED UNIT         ACTIVATES
Sat 2026-09-12 16:15:00 EDT 6min Sat 2026-09-12 16:00:10 EDT 8min ago sanoid.timer sanoid.service

So great, we have snapshots rolling on the server. Now what? How do I browse them? These snapshots don't really "live" anywhere in a traditional sense, they aren't copies of data. ZFS never overwrites data in place, so when you change a file, the old blocks are still sitting on the disk waiting to be reused. A snapshot just tells ZFS not to reuse them yet. Nothing gets copied, nothing gets moved, and a snapshot of 250GB can technically be 0B until that data starts getting modified. In any case, I wanted to be able to see these on my desktop, just in case I needed to recover something or "go back in time" in a sense. So we can see these sort of ethereal directories with

sudo ls /tank/mike/.zfs/snapshot | tail -5
autosnap_2026-09-12_16:00:20_hourly
autosnap_2026-09-12_17:00:20_hourly
autosnap_2026-09-12_18:00:20_hourly
autosnap_2026-09-12_19:00:20_hourly
autosnap_2026-09-12_20:00:10_hourly

#and can even go deeper with
sudo ls /tank/mike/.zfs/snapshot/autosnap_2026-09-12_20:00:10_hourly/MOVIES
#a bunch of my movies listed...

Really cool. So this also means I can add this to my samba config on the server and mount it directly to my desktop using sudo nano /etc/samba/smb.conf and adding this

[mike_snap]
   path = /tank/mike/.zfs/snapshot
   valid users = mike
   read only = yes
   browseable = yes

then a tweak to my fstab entry on my desktop for mounting and I can now see these listed in a browseable read-only fashion.

So onto the Nextcloud problem. I thought setting up something like the above for my Nextcloud tank would be adequate, but Nextcloud has its own built in backup system using borg backup under the hood. Turns out Nextcloud has other database items elsewhere on the boot drive outside of the basic storage pool that need to be carried over as part of their backup system which is sort of explained in their README as a PostgresQL database which probably consists of user info, directory info, and whatever else Nextcloud needs to function. Essentially what I did here was create a ZFS dataset for Nextcloud AIOs backup with a 50G quota, pointed Nextcloud to this location from the admin panel, and let NC AIO/borg do its thing. Then layered sanoid on top of this baked in system. The following will create 50G dataset in my existing storage tank, I handed ownership of that directory to www-data which is the service account AIO's mastercontainer runs as. The directories created are owned by root, and a container running as a non-root user can't write into one, so without this the backup fails on permissions. Lastly the chmod 750 gives that account full access and its group read-only access, and everyone else gets nothing.

sudo zfs create -o quota=50G tank/ncbackup
sudo chown www-data:www-data /tank/ncbackup
sudo chmod 750 /tank/ncbackup

I learned a lot from that above command, especially the www-data portion. This is a default group/account that web servers will run on Linux systems, at least Debian based ones under UID 33 and can be confirmed with getent passwd www-data. It is essentially a service level account NC AIO uses it but something like a straight up Apache web server would use the same.

So NC AIO was pointed at this location through the admin panel, probably worth a mention that borg's encryption password for NC cant be reset so writing it down is a smart choice at this phase which I have done. It should spit this out after enabling backups through their admin panel. After all of that we need to re-configure the sanoid.conf for this new directory so I added the following to it using sudo nano /etc/sanoid/sanoid.conf and adding

[tank/ncbackup]
        use_template = borgrepo

[template_borgrepo]
        hourly = 0
        daily = 5
        weekly = 2
        monthly = 0
        autosnap = yes
        autoprune = yes

This got a little weird because I am layering two things on top of one another. Borg backup (NC AIO's built in repo) + sanoid snaps. I was able to confirm its backup cadence with

sudo docker logs nextcloud-aio-borgbackup
...
BORG_PRUNE_OPTS are --keep-within=7d --keep-weekly=4 --keep-monthly=6 #<---- this is what i was interested in

Borg keeps versions of NC data and ZFS keeps versions of borg. Different jobs, different timescales. Two different things dancing around one another. Little bit confusing.

So localized snapshots for both Nextcloud and my SMB shares are up and running. At this point we had the Pi up and running with it's HDDs installed via an external dock, ZFS versions confirmed matching the Ryzen 3 Ubuntu server to the Pi and that is all.

The Pi Backup Side

As mentioned before, I have never had a full backup solution to my data, at least not an automated one. I have done the 6-month cold stored off-site storage protocol on my own which is a manual procedure, but nothing like what I am doing here. The goal was to make a system that grabs the snapshots I just finished configuring and pulls them over the network on a daily cadence. That's it. Physically this server is on the same local network but could be deployed off-site with some configuration changes. When I was researching this, every guide was a sort of "push" procedure. The source server sends to an external backup server. I wanted to flip that on its head and do this the other way around. The backup server reaches into the source and pulls, with no ability to modify or change anything on the source server. This is a simple security measure. An attacker encrypts my source drives? The snapshots on the backup are still clean and I revert from those. An attacker gets root and destroys the snapshots too? The backup box has its own copies and there is no credential on the source that can reach it and pruning snaps on both systems are separately configured. Someone breaks in and steals the backup box? The drives are encrypted at rest and require a key to bring the pool back up after a reboot. This setup is borderline inconvenient by design. There are permission layers in place so the Pi can't modify anything on R3 either. Those layers start with a dedicated syncoid account creation on the R3 source server.

Creating Syncoids Account on the Source Server

Worth mentioning first syncoid ships with sanoid by default. sanoid handles snapshots and syncoid handles sending these over a network. syncoid does not contain a configuration file like sanoid and all configuration is done via CLAs. We are going to start by adding a syncoid service level account to the source Ryzen 3 Ubuntu machine with

sudo useradd -m -s /bin/bash syncoid

-m is used to give this account a home directory which we will need for the public SSH keys you will see later on and -s /bin/bash will set bash as the default login shell which this service account will need to execute bash commands on the system. The following locks the account out from password auth but the SSH key setup later will still work

sudo passwd -l syncoid

Now it is just a matter of applying the ZFS permissions to this account with

sudo zfs allow -u syncoid send,hold,release tank

this will allow user (-u) syncoid to read and send a snapshot (send) and then hold and release are a bit of a pair which can pin a snapshot so it cant be destroyed mid transfer from sanoids snapshot pruning. Nothing else is granted to this account. Create, mount, receive, rollback, snapshot, destroy were all left out. This account can only read what already exists and cannot modify the existing pool in any way. This can be confirmed with

sudo zfs allow tank
---- Permissions on tank ---------------------------------------------
Local+Descendent permissions:
    user syncoid hold,release,send

SSH Keys on the Backup Server

So because we locked the syncoid accounts password entirely we can create an SSH key on the backup server side (Raspi5 side) with

sudo ssh-keygen -t ed25519 -f /root/.ssh/syncoid_ed25519

this will allow us to auth this service level account with SSH keys in lieu of a password, which essentially allows it to run passively without user intervention as well. The above command produces two keys. A public and private key. The private key syncoid_ed25519 will only ever live on the backup server while the other key syncoid_ed25519.pub will get copied to the authorized_keys in our source server. ssh-keygen is essentially baked into every flavor of Linux as part of OpenSSH and -t ed25519 denotes the algorithm used. Why two keys instead of one? This is classic asymmetric cryptography. The public key is not a secret, this is what will allow the source server to be compromised and have no access to the destination backup server. The private key never crosses the network. It's used locally to sign is all.

Then we can copy this over to the source servers tmp directory with

sudo cat /root/.ssh/syncoid_ed25519.pub | ssh <admin-user>@192.168.10.110 'cat > /tmp/syncoid.pub'

I had to do an admin login in this case because we are in the process of setting up the keys then on the Ryzen 3 source machine.

Then on the source Ryzen 3 machine I did the following

sudo mkdir -p /home/syncoid/.ssh #useradd -m made the home dir, not this line directly
sudo mv /tmp/syncoid.pub /home/syncoid/.ssh/authorized_keys #move the key file and rename to the filename sshd looks for
sudo chown -R syncoid:syncoid /home/syncoid/.ssh #hand it to the account
sudo chmod 700 /home/syncoid/.ssh #owner only: read, write, enter
sudo chmod 600 /home/syncoid/.ssh/authorized_keys #owner only: read, write

Then to confirm this landed on the source R3 machine I ran

sudo ssh-keygen -l -f /home/syncoid/.ssh/authorized_keys

this will print the finger print hash. OR if something went wrong it will print is not a public key file (which of course happened to me on my first attempt). Ideally this will return a SHA256 string. This can also be performed on the destination backup server with

sudo ssh-keygen -l -f /root/.ssh/syncoid_ed25519.pub

the output should match across both machines.

Then for an end-to-end test which confirms we can see into the source machine we can run the following from the backup server and we should see a list of snapshots on the host machine using our newly created SSH key

sudo ssh -i /root/.ssh/syncoid_ed25519 syncoid@192.168.10.110 'zfs list -t snapshot -r tank | tail -5'
tank/nextcloud@autosnap_2026-09-13_17:00:20_hourly    168K      -  4.42G  -
tank/nextcloud@autosnap_2026-09-13_18:00:20_hourly    128K      -  4.42G  -
tank/nextcloud@autosnap_2026-09-13_19:00:20_hourly    136K      -  4.42G  -
tank/nextcloud@autosnap_2026-09-13_20:00:20_hourly    120K      -  4.42G  -
tank/nextcloud@autosnap_2026-09-13_21:00:11_hourly    120K      -  4.42G  -

Capping ARC

ZFS uses system memory as a cache for frequently accessed data. This is a feature of ZFS. This is called ARC, or Adaptive Replacement Cache, there is a good read of this here. ARC reminds me of something like Intel's dead Optane platform. In any case, we need to give ARC a cap so it doesn't eat all of the system memory. And this backup system only gets a daily write, this is not a critical configuration. On an 8GiB Pi I decided to cap this at 3GiB. We can run

sudo nano /etc/modprobe.d/zfs.conf

and adding the following to this empty file

options zfs zfs_arc_max=3221225472 #3G cap

Ctrl+O, Enter, Ctrl+X to save and exit. And that gives me the 3GiB ARC cap. To confirm reboot and run

cat /sys/module/zfs/parameters/zfs_arc_max
3221225472 #should return this in the terminal

Pool Creation on the Backup Server

Time to find my three HDD device IDs that will be used in this raidz1 setup. I wanted IDs by wwn and not their generic IDs. Something like sdb , sdc, etc. is too volatile and could change over time. I saw this happen once with this external dock. To find the IDs I used the following

ls -l /dev/disk/by-id/ | grep wwn
# you should see something like 'wwn-0x5000c500665b8b41 -> ../../sdc' and i just took note of the wwn for each HDD that will be used in the next command

Then I created the actual raidz1 pool, calling it backup and using keyformat=passphrase. raidz1 should give me around 1.75T of usable storage with one disk being able to fail with no data lost. After running this command you'll get a passphrase prompt twice to set the password to unlock the drives

sudo zpool create -o ashift=12 -O compression=lz4 -O atime=off -O xattr=sa -O encryption=aes-256-gcm -O keyformat=passphrase -O keylocation=prompt -O mountpoint=/backup -O canmount=noauto backup raidz1 /dev/disk/by-id/wwn-0xYOURDRIVE1 /dev/disk/by-id/wwn-0xYOURDRIVE2 /dev/disk/by-id/wwn-0xYOURDRIVE3

To confirm this landed I ran zpool status backup and it returned the following

zpool status backup
config:

    NAME                        STATE     READ WRITE CKSUM
    backup                      ONLINE       0     0     0
      raidz1-0                  ONLINE       0     0     0
        wwn-0xDRIVE1            ONLINE       0     0     0
        wwn-0xDRIVE2            ONLINE       0     0     0
        wwn-0xDRIVE3            ONLINE       0     0     0

One consequence of this setup, which is by design as part of my encryption at rest goal, is that the zpool will import after a reboot or an initial power-up but the drives stay locked behind the passphrase. To bring them back up after a reboot the following needs to be performed every single reboot

sudo zfs load-key backup
#put the passphrase in
sudo zfs mount backup

Dataset Creation on the Backup Server and Initial Transfer Seed

The only dataset we are going to create in this new pool is backup/nas1 during the initial data seed all children will follow as part of the seed. So I ran the following to create the dataset

sudo zfs create -o canmount=noauto -o readonly=on backup/nas1

Then for the initial seed I ended up running this inside of tmux so an SSH drop would not disrupt the process. I also took the Pi off of WiFi and did this over a wire, so the local IP is different for this initial seed. I ran one command at the Pi for each dataset on the source machine

sudo syncoid --no-sync-snap --no-privilege-elevation --sshkey=/root/.ssh/syncoid_ed25519 syncoid@192.168.10.110:tank/mike backup/nas1/mike

sudo syncoid --no-sync-snap --no-privilege-elevation --sshkey=/root/.ssh/syncoid_ed25519 syncoid@192.168.10.110:tank/mike backup/nas1/misc

sudo syncoid --no-sync-snap --no-privilege-elevation --sshkey=/root/.ssh/syncoid_ed25519 syncoid@192.168.10.110:tank/mike backup/nas1/ncbackup

The two flags worth mentioning:

--no-sync-snap tells syncoid to replicate snapshots that already exist rather than creating its own temporary one. By default it makes a syncoid_* snapshot on the source and deletes it afterward, which needs destroy. I did not grant destroy for this account on purpose when we set our ZFS permissions on this service account. It also means snapshot lifecycle stays entirely with sanoid on the source. This flag is what allows the tight permissions on this account to continue to work.

--no-privilege-elevation stops syncoid prefixing its remote commands with sudo. Its baseline assumption is root at both ends, which is what most guides do. The syncoid account isn't in sudoers by design, so without this flag every remote command fails.

Both flags exist because I chose something other than the default. That's the whole least-privilege design in action.

After the seeding is completed I confirmed they landed with

zfs list -r backup
zfs list -t snapshot -r backup | tail

Automating the Pull Process

So the manual commands work. But I obviously wanted this backup pull to occur daily at 5am. This is a two part proces. An actual script that does the pull, which consists of some "pre-flight" checks: is the pool imported? Is it unlocked? And does it actually exist as a file system? That last one is important, because if the pool is not imported or the key is not loaded /backup still technically exists as an ordinary directory on the Pi's boot drive. It will ask the kernel if this is a real mount location or just a folder. The second part of this is a systemd timer and a service. The timer fires at 5am daily and the service runs the script.

For the pull script itself I did

sudo nano /usr/local/sbin/zfs-backup-pull.sh

and pasted in the following. Ctrl+O, Enter, Ctrl+X to save and exit.

#!/bin/bash
# Pull ZFS snapshots from R3 (tank) to the local backup pool.
# Pre-flight gates the pull: an unimported or locked pool leaves
# /backup as a plain directory on the boot media, and a sync into
# it would fill the boot drive while appearing to succeed.

set -u

POOL="backup"
MOUNT="/backup"
SRC_HOST="192.168.10.110"
SRC_USER="syncoid"
SSHKEY="/root/.ssh/syncoid_ed25519"
DATASETS=(mike misc ncbackup)

fail() { echo "PRE-FLIGHT FAILED: $1" >&2; exit 1; }

zfs mount "$POOL" 2>/dev/null
zfs mount -a 2>/dev/null

zpool list "$POOL" >/dev/null 2>&1 \
    || fail "pool '$POOL' not imported"

[ "$(zfs get -H -o value keystatus "$POOL")" = "available" ] \
    || fail "pool '$POOL' key not loaded (zfs load-key $POOL)"

mountpoint -q "$MOUNT" \
    || fail "'$MOUNT' is not a mountpoint"

echo "Pre-flight OK. Starting pull from ${SRC_HOST}."

rc=0
for ds in "${DATASETS[@]}"; do
    echo "--- tank/${ds} -> ${POOL}/nas1/${ds}"
    if syncoid --no-sync-snap --no-privilege-elevation \
        --sshkey="$SSHKEY" \
        "${SRC_USER}@${SRC_HOST}:tank/${ds}" \
        "${POOL}/nas1/${ds}"
    then
        echo "--- tank/${ds}: OK"
    else
        echo "--- tank/${ds}: FAILED" >&2
        rc=1
    fi
done

[ "$rc" -eq 0 ] && echo "All datasets pulled successfully." \
                || echo "One or more datasets failed." >&2
exit "$rc"

Then made it executable with

sudo chmod 700 /usr/local/sbin/zfs-backup-pull.sh

And for the pull service

sudo nano /etc/systemd/system/zfs-backup-pull.service

#and pasted in the following:
[Unit]
Description=Pull ZFS snapshots from R3
After=network-online.target zfs.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/zfs-backup-pull.sh
TimeoutStartSec=30min

And lastly the timer

sudo nano /etc/systemd/system/zfs-backup-pull.timer

#and pasted in the following:
[Unit]
Description=Daily ZFS backup pull from R3

[Timer]
OnCalendar=*-*-* 05:00:00
Persistent=true
AccuracySec=1min

[Install]
WantedBy=timers.target

The automation portion is done. Onto auto pruning at the backup Pi server.

Backup Server Auto-prune

The goal here is to allow the backup server to hold and retain roughly double the amount of snapshots that our source server holds. We will need to use sanoids pruning on the Pi side to handle this. Without this policy in place the Pi would retain all snapshots ever made despite the source server deleting them. This is a similar step to what we did on the source server side with a slightly different retention policy. The one key difference is the disabling of the sanoid.timer (which handles snapshots and pruning) because the Pi itself does not actually create snapshots, it simply is taking them in from a source. So I ran the following

sudo apt install sanoid
sudo mkdir -p /etc/sanoid
sudo ln -s /usr/share/sanoid/sanoid.defaults.conf /etc/sanoid/sanoid.defaults.conf
sudo systemctl disable --now sanoid.timer

Then for the config setup run

/etc/sanoid/sanoid.conf

I pasted in the following

[backup/nas1/mike]
        use_template = archive
[backup/nas1/misc]
        use_template = archive
[backup/nas1/ncbackup]
        use_template = borgarchive

[template_archive]
        frequently = 0
        hourly = 48
        daily = 30
        weekly = 8
        monthly = 12
        yearly = 0
        autosnap = no
        autoprune = yes

[template_borgarchive]
        frequently = 0
        hourly = 0
        daily = 14
        weekly = 4
        monthly = 0
        yearly = 0
        autosnap = no
        autoprune = yes

Retention policy is almost double compared to the source server's templates. And one key thing here is autosnap = no. As stated before, we are not creating snaps here, just grabbing and pruning.

Because we disabled the default sanoid.timer I went ahead and built my own service and timer to handle pruning with the following

sudo nano /etc/systemd/system/zfs-backup-prune.service #to create the service

#pasted in the following
[Unit]
Description=Prune replicated ZFS snapshots (prune only, never snapshot)
After=zfs.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/sanoid --prune-snapshots --verbose
Environment=HOME=/root

sudo nano /etc/systemd/system/zfs-backup-prune.timer #to create the timer

#pasted in the following
[Unit]
Description=Daily prune of replicated snapshots

[Timer]
OnCalendar=*-*-* 06:00:00 #this fires off 1h after the pull occurs daily
Persistent=true
AccuracySec=1min

[Install]
WantedBy=timers.target

Then Ctrl+O, Enter, Ctrl+X to save and exit. With this in place this is all setup. There are a couple more things I wanted to setup and a few pain points with this in general.

Checking Backup's Health

As of right now there is no monitoring set in place for this backup server. For now I will check in on it from time to time with a list of health check commands. This is adequate for a home server, but anything beyond my scale would obviously need a real monitoring system to alert if problems have occurred. I actually had this system up and running for a couple days as a test and have been having some issues with the pull service process. It is something I will need to dig into a bit more. I did alias the bcheck command.

sudo nano /usr/local/sbin/backup-check.sh #for our health check script

and pasted in the following

#!/bin/bash
# Weekly health check for the ZFS replication target.
# Reports state AND history: a oneshot that failed yesterday and
# succeeded today shows clean in systemctl --failed.

echo "=== FAILED UNITS ==="
systemctl --failed --no-pager

echo -e "\n=== TIMERS ==="
systemctl list-timers 'zfs-backup-*' --no-pager

echo -e "\n=== POOL ==="
zpool status -x backup
zpool list backup

echo -e "\n=== NEWEST SNAPSHOTS ==="
now=$(date +%s)
for ds in mike misc ncbackup; do
    # Thresholds key to the SOURCE's snapshot cadence per dataset.
    # mike/misc get hourlies, so a 05:00 pull lands one minutes old.
    # ncbackup only gets a daily (hourly = 0), taken ~20:00 on the
    # source, so it is legitimately ~33h old before the next arrives.
    case "$ds" in
        ncbackup) limit=40 ;;
        *)        limit=26 ;;
    esac

    line=$(zfs list -t snapshot -H -p -o name,creation -s creation \
             -r "backup/nas1/$ds" 2>/dev/null | tail -1)
    if [ -z "$line" ]; then
        printf '%-10s %s\n' "$ds" "** NO SNAPSHOTS FOUND **"
        continue
    fi

    name=$(echo "$line" | cut -f1)
    when=$(echo "$line" | cut -f2)
    age=$(( (now - when) / 3600 ))

    if [ "$age" -gt "$limit" ]; then
        flag="** STALE **"
    else
        flag=""
    fi

    printf '%-10s %3sh  %s %s\n' "$ds" "$age" "${name#*@}" "$flag"
done

echo -e "\n=== POOL I/O EVENTS (last 3 days) ==="
journalctl -k --since "3 days ago" --no-pager \
    | grep -E 'uncorrectable I/O failure|has been suspended|zio pool=' \
    | tail -5 || echo "none"

echo -e "\n=== PULL FAILURES (last 3 days) ==="
journalctl -u zfs-backup-pull.service --since "3 days ago" --no-pager \
    | grep -E 'PRE-FLIGHT FAILED|Failed with result' \
    | tail -5 || echo "none"

Ctrl+O, Enter, Ctrl-X to save then

sudo chmod 700 /usr/local/sbin/backup-check.sh 

for full read, write, and execute permissions. Then adding bcheck to ~/.bashrc with nano ~/.bashrc and adding the following to the bottom

alias bcheck='sudo /usr/local/sbin/backup-check.sh'

Ctrl+O, Enter, Ctrl-X to save and running source ~/.bashrc to reload if a reboot did not take place. Now when I run bcheck I get a quick summary like the below

=== FAILED UNITS ===
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.

=== TIMERS ===
NEXT                        LEFT LAST                        PASSED UNIT                   ACTIVATES
Tue 2026-09-15 05:00:00 EDT  12h Mon 2026-09-14 05:00:23 EDT      - zfs-backup-pull.timer  zfs-backup-pull.service
Tue 2026-09-15 06:00:00 EDT  13h Mon 2026-09-14 06:00:00 EDT      - zfs-backup-prune.timer zfs-backup-prune.service

2 timers listed.
Pass --all to see loaded but inactive timers, too.

=== POOL ===
pool 'backup' is healthy
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup  2.72T   384G  2.34T        -         -     0%    13%  1.00x    ONLINE  -

=== NEWEST SNAPSHOTS ===
mike         backup/nas1/mike@autosnap_2026-09-13_09:00:20_hourly   Sun Sep 13  5:00 2026
misc         backup/nas1/misc@autosnap_2026-09-13_09:00:20_hourly   Sun Sep 13  5:00 2026
ncbackup     backup/nas1/ncbackup@autosnap_2026-09-13_00:00:20_daily   Sat Sep 12 20:00 2026

=== POOL I/O EVENTS (last 3 days) ===

=== PULL FAILURES (last 3 days) ===

I will manually run this over SSH from time to time just to keep tabs on the system.

Pictures of the Device

Here is a picture of the device on the bench next to some conecpt sketches of an enclosure and cooling shroud.

And then a final picture of the device. The Pi's carrier was printed in PETG, becuase it can get warm. The rest is PLA. It contains an exhaust fan on the top wired directly to the Pi's USB 2.0 port.

Final Thoughts on the Backup System and Some Headaches

I do not think I would build one of these again. I feel like there should have been an easier way to achieve what I wanted but in a simpler fashion. I looked into borg backup I looked into a simple rsync but I ended up on sanoid/syncoid because it is native to ZFS and this is the file system I am using for storage at the moment. I might even revisit TrueNAS Scale again in the future with something like Nextcloud's app for TrueNAS or experiment with Unraid.

Unfortunately many guides I came across were for a root -> root sync from the source to the destination. My early goal was to make it so if either side is compromised then it would have no consequence to the entire system. I built a one way street here. On top of everything else, these Pi devices are notorious for systemd/startup timing issues and external USB docks. This is something that wouldn't poke it's head out if I had been using internal components to a standard motherboard. So I spent considerable time just getting the hardware to play ball with me alongside sleep/wait timers so the zfspool would show upon boot. Then two other issues with the no pool available. One was a stale cache file issue and scan-based import not working with a bad cache file. Even with startup timing solved and a few days of testing I wound up with some I/O failures that I am still fiddling with.

Sep 14 05:00:25 nas1-backup kernel: WARNING: Pool 'backup' has encountered an uncorrectable I/O failure and has been suspended.

And for another Pi quirk, the WiFi power save measures for this device were disabled with a separate service so the WiFi was not going into an idled state.

Then there is the Nextcloud issue. I previously had snapshots enabled for my /tank/nextcloud directory which turns out did nothing for me as that directory relied on other components on the system to restore itself. So I dug into Nextclouds native borg backup and had to create a directory for their repo snapshots and then wrap sanoid/syncoid around this.

Then my snapshots timestamps are in UTC, which is not what I want because all of these machines are set to Eastern time. So timestamps read a little confusing and this is an easy one to forget when browsing snapshots.

What about actual recovery? This has yet to be rehearsed or tested as I am still working this one out a little bit. If the source machine disappears, how do I recover? Its a matter or reconfiguring the boot drive and basically everything in this section and then some. Then copying snapshots over the network and restoring from there. My plan in the short term, because I have spare drives, will be to clone both the server boot drive and the backup server boot drive then cold store them as a last good configuration. Ideally though some form of configuration repo/image process might be best in the future if I decide to leave this system in place.

Overall this feels a little flimsy from the management perspective. Once everything is in place its not too bad, but if I let this sit for 6-months and need to revisit and fix this it is going to be tough to unravel. When I configured TrueNAS years ago it felt very intuitive considering it just has a web based front end. I figured because my experience with ZFS stopped there, why not build something so I could see what was happening under the hood through a CLI. I did need to lean on a Claude project space for this one more than I would have liked to. It became apparent early on I was going to hit some walls configuring this. So it did help quite a bit with a lot of the troubleshooting and of course the large scripts. It was also able to spit out a lengthy document to re-build if need be. Due to me leaning on Claude for some of the config heavy components, I do not 100% trust this just yet and I'd like to let this device run for a month or so while I keep an eye on it. In the end this might have been a waste of time, but a good learning experience either way.