Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
linux_router:hardware [2023-04-16 Sun wk15 09:34] – [VM / Docker on Router] baumkp | linux_router:hardware [2024-12-15 Sun wk50 10:07] (current) – [Old Router Hardware] baumkp | ||
---|---|---|---|
Line 1: | Line 1: | ||
{{tag> | {{tag> | ||
- | =====Router Hardware===== | + | ======Router Hardware====== |
- | (Jan 2023) For my router, including DNS (BIND9) and DHCP (ISC DHCP) I am using a Supermicro SYS-E200-9B that comes with a Supermicro motherboard X11SBA-LN4F. I purchased this in 2016 and got functional in 2017, whilst waiting for NFTables to run all required features on Ubuntu. | + | |
+ | =====ikoolcore-r2-max===== | ||
+ | (Dec 2024) The Supermicro SYS-E200-9B has stopped working. | ||
+ | ++++ikoolcore-r2-max specifications| | ||
+ | *Processor: Intel Alder Lake-N i3-N305 (Also N100 option, standard without system fans) | ||
+ | *Memory: 1 x SO-DIMM DDR5 4800MHz, 32GB(SAMSUNG). | ||
+ | *Ethernet Ports: 2 x Marvell AQC113C-B1-C 10Gbps Network cards(via PCIe 3.0 x 2), 2 x Intel i226-v 2.5G network cards (via PCIe 3.0 x 1) | ||
+ | | ||
+ | *Storage: 2 x M.2 2242/2280 NVMe SSD, PCIe 2.0 x 1 | ||
+ | *USB Ports: 2 x USB-A 3.0(5Gbps), 1 x USB-C 3.2 Gen2(10Gbps) | ||
+ | *Display: HDMI 2.0 and Type-C display output with 4K 60fps support | ||
+ | *Cooling System: Full aluminum body passive cooling, dual 4010 fans active cooling for aluminum fins | ||
+ | *System Compatibility: | ||
+ | *BIOS: AMI EFI BIOS with Auto Power-on, WOL, and PXE support | ||
+ | *Power Supply: DC IN 12-19V | ||
+ | *Dimensions: | ||
+ | *Weight: Main unit 1050g (1110g for Fanless Unit), packed about 1600g | ||
+ | *More: | ||
+ | | ||
+ | | ||
+ | |||
+ | More Information AND FAQs, please visit [[https:// | ||
+ | |||
+ | =====Old Router Hardware===== | ||
+ | ++++old hardware tldr;| | ||
+ | With the X11SBA-LN4F finally failing about 8 years after purchase (2016) and 7 years after be placed in to operating I am honestly disappointed in its reliability. | ||
+ | |||
+ | ====X11SBA-LN4F==== | ||
+ | For my router, including DNS (BIND9) and DHCP (ISC DHCP) I am using a Supermicro SYS-E200-9B that comes with a Supermicro motherboard X11SBA-LN4F. I purchased this in 2016 and got functional in 2017, whilst waiting for NFTables to run all required features on Ubuntu. | ||
* NFtables for firewall and routing | * NFtables for firewall and routing | ||
* Bind9 for DNS | * Bind9 for DNS | ||
Line 9: | Line 37: | ||
I looked at the various options for the router hardware, written in 2016. | I looked at the various options for the router hardware, written in 2016. | ||
- | ++++tldr;| | + | |
*A small ARM based machine, e.g. Raspberry Pi 3. (The current RPi looks much more capable.) However these machines are generally limited in a number of way, including by definition not x86 based. Many do not have more than one NIC and the NIC are often not full Gigabit. (To be fair this hardware may be sufficient in most cases, as most homes do not have better than 100Mb/s internet connections, | *A small ARM based machine, e.g. Raspberry Pi 3. (The current RPi looks much more capable.) However these machines are generally limited in a number of way, including by definition not x86 based. Many do not have more than one NIC and the NIC are often not full Gigabit. (To be fair this hardware may be sufficient in most cases, as most homes do not have better than 100Mb/s internet connections, | ||
- | *The Raspberry Pi 4 looks like a much better option than earlier versions for a home router. Still has the complexity of only native 1 NIC, but that is full 1Gbe and there are 2 USB 3 port to allow another full 1Gbe NIC off USB. | + | *The Raspberry Pi 4 & 5 looks like a much better option than earlier versions for a home router. Still has the complexity of only native 1 NIC, but that is full 1Gbe and there are 2 USB 3 port to allow another full 1Gbe NIC off USB. |
*An older x86 based machine. The main downside to these is poor power consumption and large size, even an old server tends to use more than 30W at the wall, or greater than $60/year power. Also the board I had only had one built in NIC, so I would need a PCIe NIC card. There is also the issue of reliability and performance for the older hardware, although it is probably good enough in this respect. That all being said if one is strapped for cash this may be a good way to start as the upfront cost would be smallest, if not zero. | *An older x86 based machine. The main downside to these is poor power consumption and large size, even an old server tends to use more than 30W at the wall, or greater than $60/year power. Also the board I had only had one built in NIC, so I would need a PCIe NIC card. There is also the issue of reliability and performance for the older hardware, although it is probably good enough in this respect. That all being said if one is strapped for cash this may be a good way to start as the upfront cost would be smallest, if not zero. | ||
*At the moment, 2016, there are a lot of Intel Celeron J1900 based units with 4 NICs around. The J1900 is an older CPU, 4 cores, 2.0-2.42 GHz. Also in many cases the NIC hardware is older, particularly on the cheaper units, so care must be taken if you want to ensure more up to date hardware. These machines are a good option, low power (~8 - 10W), small size. They come with 2 SATA ports and mini PCI-E slots. By the time you fit them out they cost out USD250 - 350, with 4-8GB RAM and 120GB mSata drive. The cheaper options are as noted above usually with older NIC hardware and lower memory and HD size and can be had at even lower prices. | *At the moment, 2016, there are a lot of Intel Celeron J1900 based units with 4 NICs around. The J1900 is an older CPU, 4 cores, 2.0-2.42 GHz. Also in many cases the NIC hardware is older, particularly on the cheaper units, so care must be taken if you want to ensure more up to date hardware. These machines are a good option, low power (~8 - 10W), small size. They come with 2 SATA ports and mini PCI-E slots. By the time you fit them out they cost out USD250 - 350, with 4-8GB RAM and 120GB mSata drive. The cheaper options are as noted above usually with older NIC hardware and lower memory and HD size and can be had at even lower prices. | ||
- | *I decided to get a Supermicro [[https:// | + | *I decided to get a Supermicro [[https:// |
- | <fs smaller> I don't see the point installing a 64bit OS on systems with less than 4GB of RAM. A 32bit OS can only natively access up 4 GB RAM, but should give better compromise with such limited RAM.</ | + | <fs smaller> I don't see the point installing a 64bit OS on systems with less than 4GB of RAM. A 32bit OS can only natively access up 4 GB RAM, but should give better compromise with such limited RAM.</ |
- | ++++ | + | |
+ | ====Specific issues with use of headless X11SBA-LN4F hardware==== | ||
+ | |||
+ | ====IPMI KVM Display Problems==== | ||
+ | Acronyms can be painful. IPMI = Intelligent Platform Management Interface, KVM = Keyboard video and mouse, BMC = Baseboard management controller. | ||
+ | |||
+ | The remote KVM and IPMI, BMC are not used often, however they negate the need for the use of separate keyboards and monitors to set up and maintain these machines and allow true convenient headless set up, maintenance and operation. Normally an SSH terminal is all that is required, however a BMC with KVM allows full on/ | ||
+ | |||
+ | The Pentium N3700 comes with a built-in graphics adaptor. On the headless BMC system the built-in graphics adapter is not required and can interfere with the BMC graphic adapter. The best solution is to turn off the Intel integrated graphics device (IGD), which is enabled by default. The graphics then defaults to the BMC adaptor. The IGD can be turned of from the BIOS motherboard options (In this case under Advanced-Chipset Configuration-North Bridge-Intel IGD Configuration). The terminal also seems to default to 1024x768 resolution, so no additional work is required for this. The 18.04 Server loader also had a problem with existing drive partitions, so I needed to manually remove all existing partitions using fdisk, from 18.04 install terminal. | ||
+ | |||
+ | My home server already in service over 5 years (as of 2017) has a Supermicro motherboard with Intel Atom C2750 CPU [[https:// | ||
+ | |||
+ | I now have a new server with the newer Supermicro motherboard with Intel Atom C3000 series CPU, also the 8 core version. (It was hard to justify the extra cost for the 12 or 16 core versions and I had no other hardware for the 10GB/s Ethernet option). The link to 8 core Supermicro motherboard with embedded 4 x 1GBe LAN [[https:// | ||
+ | ++++Forcing Display option at boot in Ubuntu| | ||
+ | ====Forcing Display option at boot in Ubuntu==== | ||
+ | **Note this method did not work in Ubuntu 18.04 amd64 server edition** | ||
+ | |||
+ | Basically after setting up Ubuntu 16.04 amd64 server edition on the router hardware I noticed a problem with the IPMI KVM terminal display. During the Ubuntu start-up the KVM screen would just go blank. However login into a SSH session on the main board NIC was working normally. After a bit of head scratching and investigation I worked out the problem to be related to the design of Intel N3700 with the built graphics processor that was conflicting with the BMC graphics processor built into the motherboard a Supermicro [[https:// | ||
+ | |||
+ | So the solution is to ensure that Ubuntu does not load any " | ||
+ | ++++Controlling BMC Terminal Resolution in Ubuntu| | ||
+ | ====Controlling BMC Terminal Resolution in Ubuntu==== | ||
+ | **Note this method was not tested in Ubuntu 18.04 amd64 server edition** | ||
+ | |||
+ | The BMC terminal screen seems to default to 640x480 resolution. To improve consider the following. Add ' | ||
+ | ++++Router Ethernet Hardware Consideration| | ||
+ | ====Router Ethernet Hardware Consideration==== | ||
+ | The X11SBA-LN4F hardware comes with 4 dedicated NIC controllers. NIC0 is on a dedicated PCIe lane, whereas NIC1 to 3 use a multiplexer to share another PCIe lane. The PCIe lane with the 3 shared NIC controllers have enough bandwidth to handle maximum combined throughput of the 3 NICs, however the multiplex does add a minor processing delay, although better than an additional external switch. | ||
+ | |||
+ | I plan to dedicate NIC0 to the WAN and bridge NICs 1-3 to the LAN. Also the bridged LAN network will used for the main server and its VMs with dedicated IP addresses on the LAN. The main NFTables based router will run on bare metal and a number of VMs used for DNS, DHCP, VPN and logger.++++ | ||
+ | |||
+ | =====VM / Docker on Router===== | ||
+ | ===Progress=== | ||
+ | As of 2023/01 I setup a VM manager (Libvirt/ | ||
+ | Next: | ||
+ | *ISC Kea DHCP in Docker (currently ISC DHCP in bare metal) | ||
+ | *ISC Bind 9 in Docker (currently ISC Bind 9 in bare metal) | ||
+ | *Wireguard VPN in Docker (currently Wireguard VPN in bare metal) | ||
- | Comments on Proxmox; | + | ===Router |
- | ++++tldr| | + | |
- | ====VM / Docker on Router==== | + | |
- | Router key features: | + | |
- Operate reliably 24 hours per day, 7 days a week | - Operate reliably 24 hours per day, 7 days a week | ||
- Low power operation, power cost money | - Low power operation, power cost money | ||
Line 28: | Line 90: | ||
- Hardware suitable for purpose: | - Hardware suitable for purpose: | ||
- At least 2 NICs (1 WAN plus 1 or more LAN, quality native type NICs, not USB based), 4+ NICs preferable. | - At least 2 NICs (1 WAN plus 1 or more LAN, quality native type NICs, not USB based), 4+ NICs preferable. | ||
- | - NICs to be 1 GB/s type minimum, although as of 2023 2.5GB/ | + | - NICs to be 1 GB/s type minimum, although as of 2023, 2.5GB/ |
- Sufficient CPU power not to limit primary performance | - Sufficient CPU power not to limit primary performance | ||
- Correct CPU options, e.g. AES, [[https:// | - Correct CPU options, e.g. AES, [[https:// | ||
Line 41: | Line 103: | ||
*ISP Internet connectivity (bare metal) | *ISP Internet connectivity (bare metal) | ||
*main firewall (bare metal) | *main firewall (bare metal) | ||
- | ++++ | ||
*DNS | *DNS | ||
*DHCP | *DHCP | ||
*VPN (for secure public access to LAN) | *VPN (for secure public access to LAN) | ||
- | Assumptions and Limitations | + | ===Assumptions and Limitations=== |
- | *Low power means low CPU resources, hence care with applications that require significant or otherwise unnecessary resources. | + | *Low power means lower CPU resources, hence care with applications that require significant or otherwise unnecessary resources. |
*Some services on bare metal to ensure reliable performance | *Some services on bare metal to ensure reliable performance | ||
*This machine is much slower than usual hardware, and this is noticeable on interface usage, even no graphical. | *This machine is much slower than usual hardware, and this is noticeable on interface usage, even no graphical. | ||
- | *The network and related services performance must NOT limit performance on upstream IP connectivity to greater than 100Mb/s and preferably only limit as speed get close to NIC's 1 Gb/s hardware speed. | + | *The network and related services performance must NOT limit performance on upstream IP connectivity to greater than 100Mb/s and preferably only limit as speed get close to NIC's 1 Gb/s hardware speed. |
Docker really does some work on the firewall using iptables. | Docker really does some work on the firewall using iptables. | ||
===Why not Proxmox=== | ===Why not Proxmox=== | ||
+ | ++++tldr;| | ||
*I have not used to date, this is I have no experience with Proxmox | *I have not used to date, this is I have no experience with Proxmox | ||
*I already have a lot of experience on run Debian, libvirt/ | *I already have a lot of experience on run Debian, libvirt/ | ||
*Proxmox seems to need to be installed on bare metal. | *Proxmox seems to need to be installed on bare metal. | ||
- | ====Specific issues with use of headless X11SBA-LN4F hardware==== | + | ++++ |
- | ++++IPMI KVM Display Problems| | + | |
- | ====IPMI KVM Display Problems==== | + | |
- | Acronyms can be painful. IPMI = Intelligent Platform Management Interface, KVM = Keyboard video and mouse, BMC = Baseboard management controller. | + | |
- | + | ||
- | The remote KVM and IPMI, BMC are not used often, however they negate the need for the use of separate keyboards and monitors to set up and maintain these machines and allow true convenient headless set up, maintenance and operation. Normally an SSH terminal is all that is required, however a BMC with KVM allows full on/ | + | |
- | + | ||
- | The Pentium N3700 comes with a built-in graphics adaptor. On the headless BMC system the built-in graphics adapter is not required and can interfere with the BMC graphic adapter. The best solution is to turn off the Intel integrated graphics device (IGD), which is enabled by default. The graphics then defaults to the BMC adaptor. The IGD can be turned of from the BIOS motherboard options (In this case under Advanced-Chipset Configuration-North Bridge-Intel IGD Configuration). The terminal also seems to default to 1024x768 resolution, so no additional work is required for this. The 18.04 Server loader also had a problem with existing drive partitions, so I needed to manually remove all existing partitions using fdisk, from 18.04 install terminal. | + | |
- | + | ||
- | My home server already in service over 5 years (as of 2017) has a Supermicro motherboard with Intel Atom C2750 CPU [[https:// | + | |
- | I now have a new server with the newer Supermicro motherboard with Intel Atom C3000 series CPU, also the 8 core version. (It was hard to justify the extra cost for the 12 or 16 core versions and I had no other hardware for the 10GB/s Ethernet option). The link to 8 core Supermicro motherboard with embedded 4 x 1GBe LAN [[https:// | ||
- | ++++Forcing Display option at boot in Ubuntu| | ||
- | ====Forcing Display option at boot in Ubuntu==== | ||
- | **Note this method did not work in Ubuntu 18.04 amd64 server edition** | ||
- | |||
- | Basically after setting up Ubuntu 16.04 amd64 server edition on the router hardware I noticed a problem with the IPMI KVM terminal display. During the Ubuntu start-up the KVM screen would just go blank. However login into a SSH session on the main board NIC was working normally. After a bit of head scratching and investigation I worked out the problem to be related to the design of Intel N3700 with the built graphics processor that was conflicting with the BMC graphics processor built into the motherboard a Supermicro [[https:// | ||
- | |||
- | So the solution is to ensure that Ubuntu does not load any " | ||
- | ++++Controlling BMC Terminal Resolution in Ubuntu| | ||
- | ====Controlling BMC Terminal Resolution in Ubuntu==== | ||
- | **Note this method was not tested in Ubuntu 18.04 amd64 server edition** | ||
- | |||
- | The BMC terminal screen seems to default to 640x480 resolution. To improve consider the following. Add ' | ||
- | ++++Router Ethernet Hardware Consideration| | ||
- | ====Router Ethernet Hardware Consideration==== | ||
- | The X11SBA-LN4F hardware comes with 4 dedicated NIC controllers. NIC0 is on a dedicated PCIe lane, whereas NIC1 to 3 use a multiplexer to share another PCIe lane. The PCIe lane with the 3 shared NIC controllers have enough bandwidth to handle maximum combined throughput of the 3 NICs, however the multiplex does add a minor processing delay, although better than an additional external switch. | ||
- | |||
- | I plan to dedicate NIC0 to the WAN and bridge NICs 1-3 to the LAN. Also the bridged LAN network will used for the main server and its VMs with dedicated IP addresses on the LAN. The main NFTables based router will run on bare metal and a number of VMs used for DNS, DHCP, VPN and logger.++++ | ||
<- linux_router: | <- linux_router: | ||