I’m a fan of pfSense, myself. But other suggestions here for OSes have been reasonable. I have a netgate router feeding an eero wap with a second wap creating a bridged wifi network. Future-proofing with 10GB on a wired switch if a good idea. I got a pair of Unifi 2.5GB switches with 10GB uplink for that. The difference in performance moving large data around is massive. I have 10GB between my primary machine, the one that I run as my always-on server, and my NAS. It’s awesome. Everything else is 2.5GB.

Edit: made one bit plural

First things first, make sure your ISP’s device can be put into a bridge mode, or that you can get a modem instead, otherwise you’re going to be dealing with double NAT issues.

@Ek-Hou-Van-Braai depends what you’re labbing. any sort of virtualisation etc then a mini PC from Beelink or Bosgame etc. networking have a look at Ubiquiti stuff.
yes, you can get “refurbished” stuff pike Cisco switches or whatever. but consider if big, old, slow, power hungry, umsupported kit is what you are ok with (it may well be)

Do you want the router to also be 10" rack-mountable? That seems like it’d be a big input into the hardware you get.

Opnsense on basically anything. That’s what I’d recommend as a platform, so see if they have recommended hardware for cell network support.

Or if you’re okay with commercial products, cradlepoint makes good cell network hardware. But you should still have a separate firewall/router and just use the cradlepoint as a modem.

The short answer: For a router, either find an off-the-shelf Wifi router that is supported by OpenWRT (very nice and very easy), or (and this is my personal preference) build your own firewall mini PC which will be much more complex and powerful to the point of complete overkill but also fully controllable right down to the network stack (and what’s the point of a homelab if not fiddling around with such things?).

You can run OpenWRT directly on full AMD64 PC if you want, or even just a Raspberry Pi (some people appear to have had good luck with the 4B and 5, though I don’t know the specifics of that approach) The famous PfSense would be another option, based on BSD. I used to use that, but I really wanted something directly Linux-based.

Which brings us to the fact that you can also even use a standard Linux distro like Debian and install all the tools you want on top of that and set up all the firewall yourself from scratch. That is actually what I do, using Linux kernel’s nftables for NAT Masquerading/IP forwarding and managing it currently with foomuuri which is essentially just a very lightweight nftables configuration manager. It doesn’t do anything you can’t do directly with nftables, but even though it’s perfect for me but I’m not sure I would recommend it in general. They have some very simple examples, but the documentation is pretty sparse, you need to either understand nftables under the hood or infer what you can by reading between the lines of the few examples you can find. A more mature and traditional Linux firewall like firewalld might be preferable if you want. Either way, this is definitely a much more complex route though, and fighting with firewall rules to get things to work is not everybody’s idea of “fun”. It is powerful though, and infinitely flexible. If you want it to “just work” without hassle, stick to the single-purpose devices and use OpenWRT as the OS designed to do this. It’s way simpler.

If you do decide do go the DIY firewall route though, all you really need for a firewall PC is at least a second NIC (some motherboards have two wired NIC onboard already, you can use one for WAN and the other + WiFi for LAN) or you can a PCIe network card that has multiple ports. I wouldn’t really recommend using one of your existing Mini PCs for this, as it’s really not a good idea to share the firewall/network appliance functionality shared with other services, both for security and for configuration complexity reasons. The firewall really works best and is easiest to configure when it is truly just a gateway for the network, putting traffic from one side out the other side, plus whatever fundamental network/firewall services you need to accomplish that. When you start also trying to selectively route some of that traffic to actual services on the firewall itself, it gets really complex and ugly really fast, and even if you can get it working which is often very nontrivial, it’s also very fragile and it’s easy to blow open holes in your security this way.

I’ve actually now got a pair of mini-PC firewalls, both set up using foomuuri, uCARP and Kea to do failover with each other so if one goes offline the other takes over its IP and starts routing traffic until it comes back. It’s not perfect or completely bulletproof but it’s pretty good for an amateur! In a pinch (when my previous, non-redundant firewall died) I’ve also used an GL.iNet travel router as my network’s primary router temporarily and their routers support an expansion board with 5G/SIM support so that could be an option too. I have to say it worked perfectly and was actually pretty nice, my only hesitation is that the travel router (at least the one I have, Beryl AX) seems to run a bit hot and I’m not sure it’s really intended for 24/7/365 operation (plus I need it for when I travel). They do make home routers too though, so maybe worth looking into, they’re really nice hardware running their own fork of OpenWRT out of the box.

Assuming you want to replace it all, not just home lab use…

1. Drop their router/modem combo if you can, get your own modem and router. Options are pretty wide here, but what I prefer is a wired router and separate WAPs. I’d lean toward opnsense for the router OS, and I’d use something with as little as two to four ports - one for the modem, one to hit a switch, two more gives you a second modem option (cellular as mentioned) and a second switch to hit if needed. Ideally with 10gbps for future proofing. Dont make your router/FW do lift of a switch, IMO.
2. Get a switch sized to your network. Since you’re going with a 10" rack, a small 8 port with a couple 10gbit uplinks would fit the bill. Managed only here. You dont need the latest and greatest - I have a stack of Aruba 2920s, 48+4 PoE+ (stacking cables) that I got for free that were being replaced. They came out in 2013 and went end of sale in 2017, and have been in my home lab since. So - any thing managed that handles what you have and a bit more.
3. In terms of WAP, TP-Link, ASUS, and Zyxel all have decent hardware that works well.

They are not open source but I just started dipping my toes into the Mikrotik ecosystem and the hardware has been pretty nice from what I have seen. I am not a network guy, just a homegamer coming from normal asus routers though. They have a couple of options for adding cell service via sim cards but I have not looked too far into it.

Edit: it looks like there is an openwrt release for the rb5009ug I am using. I may need to check that out.