KPTree Router Setup

Work in progress

Router Background

Well here it goes. I have had the router hardware for a few months now. I am proposing to setup up primarily as a home router on baremetal, and with DNS, DHCP and VPN probably running in VMs or containers. Further to this I will attempt to use NFTables instead of IPTables to setup the router. It is a bit to bite off, but I have the time now. It has taken a lot of time to read, research, and test the configuration. I have yet to fully implement. Further to this separate configuration files are required, as the exact specific configuration would be a security risk to make public.

A number of online references were used to assist with planning and configuring this router. These can be found in the Reference (Related Links) section and were particularly relevant are also repeated in the section text. No individual online reference was exclusivly used.

Router Hardware

I looked at the various options for the router hardware:

  • A small ARM based machine, e.g. Raspberry Pi. However these machines are generally limited in a number of way, including by definition not x86 based. Many do not have more than one NIC and the NIC are often not full Gigabit. The main upside is that they are small, low power and relatively cheap. Those with only one NIC need to be setup with USB NIC adaptors, that further complicates setup, performance and reliability. Although better spec'ed machines, e.g. with multiple gigabit NICs, start getting more pricey too. I suppose you get what you pay for....
  • An older x86 based machine. The main downside to these is poor power consumption and large size, even an old server tends to use more than 30W at the wall. Also the board I had only had one built in NIC, so I would need a PCIe NIC card. There is also the issue of reliability and performance for the older hardware, although it is probably good enough in this respect.
  • At the moment there are a lot of Intel Celeron J1900 based units with 4 NICs around. The J1900 is an older CPU, 4 cores, 2.0-2.42 GHz. Also in many cases the NIC hardware is older, particularly on the cheaper units, so care must be taken if you want to ensure more up to date hardware. These machines are a good option, low power (~8 - 10W), small size. They come with 2 SATA ports and mini PCI-E slots. By the time you fit them out they cost out USD250 - 350, with 4-8GB RAM and 120GB mSata drive. The cheaper options are as noted above usually with older NIC hardware and lower memory and HD size and can be had at even lower prices.
  • I decided to get a Supermicro SYS-E200-9B Intel Pentium N3700 system with 4 x Intel i211-AT GbE LAN. I got with maximum 8GB RAM and 120GB mSata HD. The N3700 CPU is more modern than the J1900 and includes AES instruction that the J1900 does not have. The AES CPU instruction help improve encryption performance significantly, hand for SSL / VPN. Otherwise the overall performance is slightly better (1.6-2.4GHz) and power lower than the J1900. This unit also comes with a dedicated IPMI LAN Port, allowing full remote KVM operation on the network. A downside of the IPMI is that it uses another 3.5W of power. My home server is also a Supermicro based unit with dedicated IPMI LAN Port and has given me a good 2 years of service to date. Downside is mainly the price, USD490 + delivery, as these units are not sold locally I purchase in USA and had it mailed at USD75. In any case this hardware should allow for a router with great performance for some years to come. Again you get what you paid for.....

I dont see the point installing a 64bit OS on systems with less than 4GB of RAM. A 32bit OS can only natively access up 4 GB RAM, but should give better compromise with such limited RAM.

IPMI KVM Problems

Acronyms can be painful. IPMI = Intelligent Platform Management Interface, KVM = Keyboard video and mouse, BMC = Baseboard management controller.

The remote KVM and IPMI, BMC are not used often, however they negate the need for the use of separate keyboards and monitors to setup and maintain these machines and allow true convenient headless setup, maintenance and use.

Basically after setting up Ubuntu 16.04 amd64 server edition on the router hardware I noticed a problem with the IPMI KVM terminal. During the Ubuntu startup the KVM screen would just go blank. However login into a SSH session on the main board NIC was working normally. After a bit of head scratching and investigation I worked out the problem to be related to the design of Intel N3700 with the built graphics processor that was conflicting with the BMC built into the motherboard a Supermicro X11SBA-LN4F in the also Supermicro SYS-E200-9B. So the solution is to ensure that Ubuntu does not load any "special" main board (Celeron N3700) CPU graphic drivers. For Debian and Ubuntu this is done by setting the "noomodest" option into the grub bootloader. this can be done by editing the grub bootloader during boot up, a one off solution and by making permanent by editing the grub configuration file. The reliablesite.net give a good explanation in their article How to set 'Nomodeset' into the grub bootloader. For the permanent solution basically edit etc/default/grub, adding nomodeset such that GRUB_CMDLINE_LINUX_DEFAULT= "nomodeset" and then execute sudo update-grub.

My home server already in service over 2 years has a Supermicro motherboard with Intel Atom C2750 CPU A1SAi-2750F also with IPMI, BMC & KVM and did not display this problem. This makes sense as the Atom C2750 CPU does not have a internal graphic capacity, so the only graphics capacity was on the BMC video controller. The Ubuntu drivers defaulted to this basic BMC graphics display system.

Router Linux Network Setup

The hardware comes with 4 dedicated NIC controllers. NIC0 is on a dedicated PCIe lane, whereas NIC1 to 3 use a multiplexer to share another PCIe lane. The PCIe lane with the 3 shared NIC controllers have enough bandwidth to handle maximum combined throughput of the 3 NICs, however the multiplex does add a minor processing delay, although better than an additional external switch.

I plan to dedicate NIC0 to the WAN and bridge NICs 1-3 to the LAN. Also the bridged LAN network will used for the VMs with dedicated IP addresses on the LAN. The main NFTables based router will run on baremetal and a number of VMs used for DNS, DHCP, VPN and logger.

Router Basic Ubuntu Server Setup

Download the latest Ubuntu Server amd64 ISO file from the Ubuntu website. I setup the IPMI KVM to provide virtual storage to the Ubuntu ISO file and started up the Router. The Ubuntu software loads up of the ISO across the KVM virtual storage setup and can be then setup as normal. When setting up Ubuntu below are some the of keypoints:

  • I setup Ubuntu to use LAN port 2, as I want to use LAN port 1 as the Router WAN port.
  • I do not encrypt the home directory. (See How to install Ubuntu Server - Xenial Xerus 16.04LTS for an explanation.)
  • I just use standard setup for one main partition, which basically gives one large data partition using all the disk space, save that allowed for the SWAP partition. The SWAP partition is automatically sized based upon detected memory. (I have never been one for multiple partitions.)

Ubuntu Network Setup

The units 4 main port will be setup to look like a router with 1 WAN port and 3 LAN port. The WAN port will be setup on NIC0 with the LAN ports 0-2 will be on bridged NIC1-3.

To check available interfaces and names: "ip link"

Ensure the bridge utilites are loaded: "sudo apt install bridge-utils""

Edit the network configuration file: "/etc/network/interfaces" as follows:

# The loopback network interface

auto lo

iface lo inet loopback


# WAN interface

auto eno1

iface eno1 inet dhcp


#LAN are eno2 eno3 eno4 on bridged mode on the 192.168.a.x subnet

auto br0

iface br0 inet static

bridge_ports eno2 eno3 eno4

address 192.168.a.1

network 192.168.a.0

netmask 255.255.255.0

broadcast 192.168.a.255

dns-nameservers 192.168.a.b

bridge_stp off # disable Spanning Tree Protocol

bridge_fd 9 # no forwarding delay

bridge_hello 5

bridge_maxage 60

The following is a list of some stanza not used and why:

# The gateway directive is not required as any traffic to 192.168.a.1 not on subnet /24 will be Netfiltered and if accepted passed to WAN.

# gateway 192.168.a.1


# auto eth1 and iface eth1 inet manual are not required as as iface inet br0 will bring up the components assigned to it.

# iface eno2 inet manual

# iface eno3 inet manual

# iface eno4 inet manual


# The use of allow-hotplug eth1 is not used as normally these interfaces should be running. The br0 interface will be used for virtual machine access and must be running to allow coorect VM startup. During inital boot this will cause delays as attempts are made to find network devices, particularly any not used. These delays are necessary for reliable start-up and operation.

# allow-hotplug eno2

# iface eno2 inet manual

# pre-up ifconfig $IFACE up

# pre-down ifconfig $IFACE down

System Forwarding Enable

To allow the router to forward packet the Linux kernal must be setup to allow this. This is not necessarily a standard option.

NFTables Configuration

Under construction

If is difficult to find good simple comprehensive information for NTFilter at this time, perhaps it is too new. There is alot of information on iptables. One of the best references I found Wolfhechel github nftables router. The reference at stosb is good, but not for a router Explaining My Configs: nftables. Create the following file called: "router.nft".

Sample NFTables configuration

define external = eno1

define internal = br0

define dhcp_range = 192.168.0

#Clean out the current ruleset

flush ruleset

table inet firewall {

set blacklist {

type ipv4_addr

}

set tcp_open_ports {

type inet_service; flags interval;

elements = {

http, https,

smtp, imaps, pop3s,

ssh,

}

}

set udp_open_ports {

type inet_service; flags interval;

elements = {

openvpn,

}

}

chain base_checks {

# allow established/related connections

ct state {established, related} accept

# early drop of invalid connections

ct state invalid drop

}

chain tcp_cek {

# bad tcp -> avoid network scanning:

tcp flags & (fin|syn) == (fin|syn) drop

tcp flags & (syn|rst) == (syn|rst) drop

tcp flags & (fin|syn|rst|psh|ack|urg) < (fin) drop

tcp flags & (fin|syn|rst|psh|ack|urg) == (fin|psh|urg) drop

}

chain input {

type filter hook input priority 0; policy drop;

jump base_checks

jump tcp_cek

# no ping floods:

ip protocol icmp limit rate 10/second accept

ip protocol icmp drop

# drop connections from blacklisted addresses

ip saddr @blacklist drop

# accept input from loopback and internal interfaces

iif { lo, $internal } accept

# avoid brute force on ssh: (if using external ssh)

tcp dport ssh limit rate 15/minute accept

# allow icmp

ip protocol icmp icmp type { echo-request, echo-reply, time-exceeded, parameter-problem, destination-unreachable } accept

ip6 nexthdr icmpv6 icmpv6 type { echo-request, echo-reply, time-exceeded, parameter-problem, destination-unreachable, packet-too-big, nd-router-advert, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld-listener-query } accept

# allow tcp/udp open ports

tcp dport @tcp_open_ports accept

udp dport @udp_open_ports accept

# everything else reject port unreachable

reject with icmpx type port-unreachable

}

chain forward {

type filter hook forward priority 0; policy drop;

iif $external oif $internal jump base_checks

iif $internal oif $external accept

# Allow coming out of the vpn

# ip saddr 192.168.87.0/24 iifname tun0 accept

# Allow connecting to home_srv.

ip daddr home_srv ct status dnat accept

}

chain output {

# We allow everything out

type filter hook output priority 0; policy accept;

}

}

table ip nat {

map tcp_nat_map {

type inet_service : ipv4_addr

# elements = { http-alt : 192.168.1.101, 81 : 192.168.1.100} #see note below

}

map udp_nat_map {

type inet_service : ipv4_addr

}

chain wan_in {

tcp dport { http, https} dnat 192.168.1.15

tcp dport { pop3s, imaps, smtp} dnat 192.168.1.18

dnat tcp dport map @tcp_nat_map

dnat tcp dport map @udp_nat_map

}

chain prerouting {

type nat hook prerouting priority 0; policy accept;

iif $external jump wan_in

}

chain postrouting {

type nat hook postrouting priority 0; policy accept;

oif $external masquerade

# oifname {ens3, tun0} masquerade

}

}

Some key related commands:

  • To load the nft configuration file: "sudo nft -f router.nft"
  • The nftables configuration file can be made into an executable script as follows.
    • Add the following at the top of the file:
    • #!/user/sbin/nft -f
    • chmod +x router.nft
    • Use "whereis nft" to find where the nft executable is located
    • Remember that if located in same directory use "sudo ./router.nft" to run.
  • To list currently loaded nft tables: "sudo nft list tables"
  • To list a loaded table use: "sudo nft list table inet firewall", the "inet firewall as listed from "sudo nft list table"
  • To list a loaded table use: "sudo nft list table ip nat". (The -a option lists handles.)
  • To add elements to nat tcp_nat_map: "sudo nft add element nat tcp_nat_map { 81 : 192.168.1.100, 8080 : 192.168.1.101 }". (For some reason the version of nft I have will not read in the elements via a nft -f command.)

Some other handy pointers and commands related to the NFtables setup:

How to make NFTables configuration permanent / restore on boot

DHCP / DNS Setup

A couple of interesting resources are BigDinosaur Blog Running BIND9 and ISC-DHCP and Kill-9 Ubuntu 16.04 based Router Part 2 - DHCP and Part 5 - DNS. Another reference is Lani's Weblog - Make your DHCP server dynamically update your DNS records on Ubuntu 12.04 (Precise Pangolin).

DNS Setup

First install or ensure already installed the DNS server software: "sudo apt install bind9"

Next check the named.conf configuration file, "less /etc/bind/named.conf". This can remain as default as below. However the configuration files noted there in will need to be set up. We will copy the existing files to default:

  • "sudo cp /etc/bind/named.conf.options /etc/bind/default.named.conf.options"
  • "sudo cp /etc/bind/named.conf.local /etc/bind/default.named.conf.local"
  • "sudo cp /etc/bind/named.conf.default-zones /etc/bind/default.named.conf.default-zones"

The /etc/bind/named.conf is not changed, and should look as below.

// Default contents of /etc/bind/named.conf

// This is the primary configuration file for the BIND DNS server named.

//

// Please read /usr/share/doc/bind9/README.Debian.gz for information on the

// structure of BIND configuration files in Debian, *BEFORE* you customize

// this configuration file.

//

// If you are just adding zones, please do that in /etc/bind/named.conf.local

include "/etc/bind/named.conf.options";

include "/etc/bind/named.conf.local";

include "/etc/bind/named.conf.default-zones";

Next modify the named.conf.options configuration file, "sudo vim /etc/bind/named.conf.options", as noted below.

options {

directory "/var/cache/bind";

auth-nxdomain no; # conform to RFC1035

forwarders {

208.67.222.222;

208.67.220.220;

};

allow-query {

192.168.0/24;

127.0.0.1;

};

allow-transfer {

192.168.0/24;

127.0.0.1;

};

};

The forwarders section contains the DNS servers to be checked if this DNS does not have the record. I have been using OpenDNS to allow some free security screening, IP 208.67.222.222 and 208.67.220.220. Another common option is to use Google DNS at 8.8.8.8 and 8.8.4.4

Next create a cryptographic key file using "Sudo /usr/sbin/rndc-confgen -a", note that this command can take quite some time to complete, a number of minutes. The command produces a key file "/etc/bind/rndc.key".

key "rndc-key" {

algorithm hmac-md5;

secret "ABCabc12345678/abc1234==";

};

Configure the DNS zones "sudo vim /etc/bind/named.conf.local"

key "rndc-key" {

algorithm hmac-md5;

secret "ABCabc12345678/abc1234==";

};

zone "kptree.net" {

type master;

file "/var/lib/bind/kptree.net.hosts";

allow-update { key rndc-key; };

};

zone "0.168.192.in-addr.arpa" {

type master;

file "/var/lib/bind/192.168.0.rev";

allow-update { key rndc-key; };

};

Modify the forward lookup zone definition file "sudo vim /var/lib/bind/kptree.net.hosts"

# This line indicates that the object we're configuring below (in this case,

# kptree.net) has its origin at the "." domain. "." is the root domain

# from which all the TLDs branch.

$ORIGIN .

# Next line defines the DNS time-to-live setting

$TTL 907200 ; 1 week 3 days 12 hours

# The next set of lines are the "Start of Authority" record and define important

# info about the domain. In my case, we're defining kptree.net and saying

# that router.kptree.net is its source host, and webmaster@kptree.net

# is the domain maintainer. For the e-mail address, we use a dot instead of an @.

# The lines after that define the zone serial number, which is used to keep track

# of when the zone file was modified, and then some interval definitions which

# you can leave as default.

kptree.net IN SOA router.kptree.net. admin.kptree.net. (

234284 ; serial

10800 ; refresh (3 hours)

3600 ; retry (1 hour)

604800 ; expire (1 week)

38400 ; minimum (10 hours 40 minutes)

)

# Next, we define the hosts necessary to make the domain function. First, we add

NS router.kptree.net.

# ...then an "A Record" for the domain server's IP address...

A 192.168.0.1

# ...and finally "MX Records" so that e-mail for the domain's e-mail addresses

# goes to the right place.

MX 10 mail.kptree.net.

#

# NOTE THE TRAILING PERIODS. THEY ARE EXTREMELY IMPORTANT.

#

#

# Now we're ready to begin adding hosts, but first we need another origin

# statement to indicate that the hosts added below originate not from ".", like

# the domain itself; rather, they originate from "kptree.net".

#

$ORIGIN kptree.net.

#

# Again, NOTE THE TRAILING PERIOD.

# Now we add A records for the non-DHCP hosts in the domain:

kptreeserver A 192.168.0.2

switch A 192.168.0.3

wwwserver A 192.168.0.4

mailserver A 192.168.0.5

Define the reverse zone, "sudo vim /var/lib/bind/192.168.1.rev"

# Again, we have an origin record and a TTL entry...

$ORIGIN .

$TTL 907200 ; 1 week 3 days 12 hours


# note the name of the reverse domain: "0.168.192.in-addr-arpa". This

# is a special name format used only by reverse lookup domains.

0.168.192.in-addr.arpa IN SOA router.kptree.net. admin.kptree.net. (

12 ; serial

10800 ; refresh (3 hours)

3600 ; retry (1 hour)

604800 ; expire (1 week)

38400 ; minimum (10 hours 40 minutes)

)

NS router.kptree.net.

#

# Just like above, we now set our origin away from "." to the actual domain

# name, which is "0.168.192.in-addr-arpa", and then we add records. However,

# this time, we're adding "PTR records", or pointer records.

$ORIGIN 1.168.192.in-addr.arpa.

2 PTR switch.kptree.net.

3 PTR kptreeserver.kptree.net.

4 PTR wwwserver.kptree.net.

5 PTR mailserver.kptree.net.

If and of the above files are changed the serial number should be incremented up before updating the the DNS service, "sudo systemctl restart bind9". A common technique is to use the date followed by a small single or double digit number, e.g. 2017072101.

DHCP Setup

First install or ensure already installed the DHCP server software: "sudo apt install isc-dhcp-server"

Next edit the dhcp configuration file: "sudo vim /etc/dhcp/dhcpd.conf"

ddns-updates on;  
ddns-update-style interim;  
update-static-leases on;
authoritative;
key rndc-key { algorithm hmac-md5; secret LBLC2Dg8v6hYNE/ecnd6Ag==;}
allow unknown-clients;
use-host-decl-names on; 
default-lease-time 1814400; #21 days
max-lease-time 1814400; #21 days
log-facility local7;
# kptree DNS zones
zone kptree.net. {  
    primary localhost; # This server is the primary DNS server for the zone
    key rndc-key; # Use the key we defined earlier for dynamic updates
    }
zone 0.168.192.in-addr.arpa. {  
    primary localhost; # This server is the primary DNS server for the zone
    key rndc-key; # Use the key we defined earlier for dynamic updates
    }
# kptree LAN scope
subnet 192.168.0.0 netmask 255.255.255.0 {  
    range 192.168.0.50 192.168.0.100;
    option subnet-mask 255.255.255.0;
    option routers 192.168.0.1;
    option domain-name-servers 192.168.0.1;
    option domain-name "kptree.net";
    ddns-domainname "kptree.net.";
    ddns-rev-domainname "in-addr.arpa.";
}
# kptree.net groups
group {  
   		
    # APS Solar Energy Management Unit
    host aps-ema.kptree.net {
        hardware ethernet 10:87:1A:01:37:AB;
        fixed-address 192.168.0.250;
        ddns-hostname "aps-ema";
        }
	
    # OpenSprinkler
    host sprinkler.kptree.net {
        hardware ethernet 00:09:09:1D:81:10;
        fixed-address 192.168.0.251;
        ddns-hostname "sprinkler";
        }
		
    # WiFi Access Point #1
    host wifi-ap1.kptree.net {
        hardware ethernet 90:47:13:AC:A3:FE;
        fixed-address 192.168.0.240;
        ddns-hostname "wifi-ap1";
        }
		
    # Printer #1 Canon MP690
    host printer1.kptree.net {
        hardware ethernet 00:2E:9F:AD:F4:B1;
        fixed-address 192.168.0.224;
        ddns-hostname "printer1";
        }
    }

Restart the DHCP and DNS servers to update for latest configurations changes. DNS: "sudo systemctl restart bind9" and DHCP: "sudo systemctl restart isc-dhcp-server".

Reserved Ports and IPv4 Reserved Addresses

It took me some time to track down this authoritive information, but it was relatively simple as this information is authoritivily define in RFC (Request of Comments) defacto standards as noted below.


IANA Service Name and Transport Protocol Port online Number Registry

RFC 3232 replaced RFC 1700. RFC 3232 states that RFC 1700 has been replaced by an online database, see link given above. RFC 6335 also has information on Port Number Registry and the associated database.

Most Unix like operating systems have a service name database file: "/etc/services". It is assumed that nft uses the /etc/services database for named ports definition.

Some ports of interest:

Name      Port Protocols        Description
ftp-data  20   {tcp, udp, sctp} File Transfer [Default Data]
ftp       21   {tcp, udp, sctp} File Transfer Protocol [Control]
ssh       22   {tcp, udp, sctp} The Secure Shell (SSH) Protocol
smtp      25   {tcp, udp}       Simple Mail Transfer
domain    53   {tcp, udp}       Domain Name Server (DNS)
bootps    67   {tcp, udp}       Bootstrap Protocol Server (DHCP)
bootpc    68   {tcp, udp}       Bootstrap Protocol Client (DHCP)
http      80   {tcp, udp, sctp} World Wide Web HTTP
pop3      110  {tcp, udp}       Post Office Protocol Version 3
ntp       123  {tcp, udp}       Network Time Protocol
imap2     143  {tcp, udp}       Internet Message Access Protocol
ldap      389  {tcp, udp}       Lightweight Directory Access Protocol
https     443  {tcp, udp, sctp} http protocol over TLS/SSL
urd       465  {tcp}            ssmtp smtps URL Rendesvous Directory for SSM
ldaps     636  {tcp, udp}       ldap protocol over TLS/SSL (was sldap)
rsync     873  {tcp, udp}       rsync
ftps-data 989  {tcp, udp}       File Transfer [Default Data]
ftps      990  {tcp, udp}       File Transfer Protocol [Control]
imaps     993  {tcp, udp}       imap4 protocol over TLS/SSL
pop3s     995  {tcp, udp}       pop3 protocol over TLS/SSL (was spop3)
openvpn   1194 {tcp, udp}       OpenVPN

Notes:

  1. Items in bold as per /etc/services in Ubuntu 16.04, where different from IANA
  2. TCP = Transmission Control Protocol
  3. UDP = User Datagram Protocol
  4. SCTP = Stream Control Transmission Protocol
  5. The Bootstrap Protocol was a precursor to DHCP (Dynamic Host Configuration Protocol)

RFC 3330 Special-Use IPv4 Addresses, September 2002

IPv4 uses some of these special addresses for private LANs (Local Area Network)s with NAT (Network Address Translation) used to connect the LANs to the WAN (Wide/World Area Network) via a router. This was required to compensate for the limited address space in IPv4. IPv4 NAT also provide some security benefits by obscuring the private LAN addresses from the public WAN.

IPv6 does not use NAT as its native address space is suffiently large never to require in the forseeable future.


Another interesting link, IANA https://www.iana.org/protocols

The following is a list of related commonly used commands and scripts:

  • Get external IP address "wget http://ipinfo.io/ip -qO -"
  • To check the current network hardware configuration "ip a"
  • Systemd common commands (start / stop / restart / status) (enable / disable for boot control)
  • To start (/stop /enable) the bind9 daemon "sudo systemctl start bind9"
  • To check networking status "sudo systemctl status networking"
  • List current Systemd operating units: "sudo systemctl list-units | grep '*'". Change or remove the grep statement as required.

Some related links

Linux Router Setup links

Some basic research on Linux Router led me to the following:


NFTables links

Some basic research on NFTables led me to the following


Ubuntu Network Setup Links

Links relating to bridged and bonded Networking

A bridged network allows different networks to be connected, both physical, like NICs or Wifi and virtual, allowing virtual machine to connect to a physical network and even be assigned a LAN IP address. Bonding allows physical networking devices such as NICs or Wifi to be bonded to allow increased bandwidth or redundancy. Sadly there seems to be alot of information out there that isceither for older version of software or other purposing.


DNS/DHCP Related links


Other Related links