Are you looking for something with a nice UI or just hardware you can install your OS on and roll everything yourself? What's your budget?
Ubiquiti makes a pretty user-friendly rack-mounted router: https://store.ui.com/us/en/products/udm-pro
You can also check out pfSense, which is an OS that does routing and more. You can install it on your own hardware or buy a machine through them: https://www.pfsense.org/products/
If you're just looking for a compact server that you can install your own OS on, then SuperMicro has a sale going on right now: https://store.supermicro.com/us_en/catalogsearch/result/?q=SuperServer%201u&categories=Deals
Just know that whatever you pick to replace the router-portion of your current Netgear device, you'll also have to invest in a separate WiFi access point to provide your wireless network.