A CentOS v5 Linux IPv6/IPv4 router appliance

An as built for a 1U router appliance, using CentOS v5.3

[Note: Written in 2009]

The aim is to provide IPv4 and IPv6 routing to a small network, where servers (virtual and physical) are each on their own fire-walled network. For IPv4 use a /16 net from the 10.0.0.0/8 range, with a /24 for each network. For IPv6 use an ISP (or tunnel provider) assigned /48 (or /56) network.

The router is configured for static (connected) routing only. Each network has a single gateway. No OSPF/BGP routing is required (or used).

Most of the networks are run with 9000byte (jumbo) frames. Those with physical devices that are unable to support larger frames sizes are run with a MTU of 1500bytes (standard frame sizes).

The hardware used is a Supermicro X7SBL motherboard in a 1 U chassis, with an additional PRO/1000 PT PCIe network card, and a 8GB compact flash in a SATA adapter. Multiple physical ethernet connections are used with IEEE 802.3ad link aggregation, with tagged VLAN frames.

Basic Configuration

Install a base CentOS x86_64 distribution, with the minimum set of packages (169 packages), and then apply all current updates. The machine has the name orange.lucidsolutions.co.nz.

Add a user account (with a public key in ~/.ssh/authorized_keys2), and lock down the openssh-server using the following simple configuration.

Note: RHEL/CentOS doesn't support full IPv6 connection tracking (i.e no nf_conntrack_ipv6.ko module). It is described in bug 232933.

Less is more

Disabled all unused services (Note: For simplicity, SELinux is disabled by setting 'SELINUX=disabled' in /etc/sysconfig/selinux).

# for S in kudzu lvm2-monitor mcstrans messagebus netfs restorecond ; do chkconfig $S off ; done

Networking

The networking needs to support bonding, VLAN's, with IPv4 and IPv6. Configure the following:

/etc/modprobe.conf
/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth1
/etc/sysconfig/network-scripts/ifcfg-eth2
/etc/sysconfig/network-scripts/ifcfg-eth3
/etc/sysconfig/network-scripts/ifcfg-bond0
/etc/sysconfig/network-scripts/ifcfg-bond0.xxx

Given that the IPv6 network addresses are not owned by me, and it likely that I would get a new range if I change ISP (or tunnel provider), I want to be able to renumber without having to change too many configuration files, and without typing in the same IPv6 prefix in a lot of places. So I have defined the IPv6/IPv4 prefix in the '/etc/sysconfig/network' configuration file.

Install VLAN tools.

# yum install vconfig

Use a script to create 32 VLAN based subnets. This gives a base set of configuration files, with a lot less typo's than manually typing them all in. Note: The lines after the cat statement are indented using tab characters (not spaces). Start with a configuration file defining the IPv4/IPv6 prefixes ('/etc/sysconfig/network-prefix'):

IPV6_UNICAST_PREFIX=2001:4428:225
IPV6_ULA_PREFIX=fd0c:898b:471c
LOCAL_IPV4_PREFIX=10.20

A small script

#!/bin/bash

source /etc/sysconfig/network-prefix

for (( i = 0 ; i < 32 ; i++ )) ; do

  IFCFG=/etc/sysconfig/network-scripts/ifcfg-bond0.$(( $i + 100 ))
  if [ ! -e $IFCFG ] ; then
    echo "Creating VLAN $(( $i + 100 )), IPv4 ${LOCAL_IPV4_PREFIX}.$i.0, IPv6 ${LOCAL_IPV6_PREFIX}:$(printf %.4x $i)::/64"
    cat > $IFCFG <<-EOF
        DEVICE=bond0.$(( $i + 100 ))
        TYPE=Ethernet
        ONBOOT=yes
        BOOTPROTO=none

        # IPv4
        IPADDR=${LOCAL_IPV4_PREFIX}.$i.1
        NETMASK=255.255.255.0

        # IPv6
        IPV6ADDR=${IPV6_UNICAST_PREFIX}:$(printf %.4x $i)::1/64
        IPV6ADDR_SECONDARIES=${IPV6_ULA_PREFIX}:$(printf %.4x $i)::1/64
EOF
  else
    echo "$IFCFG already present"
  fi

done

Enable IPv4 forward of packets in '/etc/sysctl.conf':

net.ipv4.ip_forward = 1

IPv4 DHCP Server

For the legacy IPv4 hosts, wireless clients, and some appliances it is useful to be able to use DHCP. Install the ISC DHCP server.

# yum install dhcp

Edit the DHCP server configuration '/etc/dhcpd.conf' to support IPv4 DHCP on the required networks.

authoritative;
ddns-update-style none;

option domain-name "lucidsolutions.co.nz";
option domain-name-servers 192.168.0.2, 192.168.0.18;
option ntp-servers         192.168.0.2, 192.168.0.20;
option time-offset         43200;

default-lease-time 86400;
max-lease-time 604800;

option subnet-mask              255.255.255.0;

subnet 10.20.1.0 netmask 255.255.255.0 {
        option routers                  10.20.1.1;
        range 10.20.1.100 10.20.1.199;
}


subnet 10.20.4.0 netmask 255.255.255.0 {
        option routers                  10.20.4.1;
        range 10.20.4.100 10.20.4.199;
}

subnet 10.20.12.0 netmask 255.255.255.0 {
        option routers                  10.20.12.1;
        range 10.20.12.100 10.20.12.199;
}

subnet 10.20.13.0 netmask 255.255.255.0 {
        option routers                  10.20.13.1;
        range 10.20.13.100 10.20.13.199;
}

Edit the DHCP service configuration '/etc/sysconfig/dhcpd' to configure which interfaces to listen on.

DHCPDARGS="bond0.101 bond0.104 bond0.112 bond0.113"

Start the service

# chkconfig dhcpd on
# service dhcpd start

Radvd

Given the machine is acting as an IPv6 router, install the radv daemon.

radvd is the router advertisement daemon for IPv6. It listens to router solicitations and sends router advertisements as described in "Neighbor Discovery for IP Version 6 (IPv6)" (RFC 2461). With these advertisements hosts can automatically configure their addresses and some other parameters. They also can choose a default router based on these advertisements.

# yum install radvd

Generate a base configuration file for the 32 networks created on the router.

#!/bin/bash

source /etc/sysconfig/network-prefix

for (( i = 0 ; i < 32 ; i++ )) ; do
  cat <<EOF
interface bond0.$(( $i + 100 ))
{
        AdvSendAdvert on;
        prefix ${IPV6_UNICAST_PREFIX}:$(printf %.4x $i)::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
        };
        prefix ${IPV6_ULA_PREFIX}:$(printf %.4x $i)::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
        };
};

EOF
done

Take the output and put it into the '/etc/radvd.conf' configuration. Start the daemon:

# chkconfig radvd on
# service radvd start

IPv4 Firewall

IPv6 Firewall

Take the same approach as the IPv4 firewall script.

Ethernet Device Numbering

The standard way to ensure devices are numbered in a consistent order after boot is to add the MAC address to the ifcfg-ethX file, and/or to specify the driver in the modprobe.conf file. This will be performed automagically during installation.

Alternatively, instead of matching the device names to the MAC addresses, use udev and match based on the PCI bus location/id. This should mean that a new NIC can be put in the machine (in the same slot), or a new motherboard installed, and the configuration shouldn't need changing.

Looking at the tree view of the PCI devices, the PCI id's can be determined. The onboard NICs have higher (larger numbers) than the PCIe NIC's, and thus enumerate second. This caused the '82573E' based NIC with the IPMI card to be have an unexpected device name.

# lspci -vvt
-[0000:00]-+-00.0  Intel Corporation 3200/3210 Chipset DRAM Controller
           +-01.0-[0000:01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
           |                 \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
           +-1a.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
           +-1a.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5
           +-1a.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6
           +-1a.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
           +-1c.0-[0000:05]--
           +-1c.4-[0000:0d]----00.0  Intel Corporation 82573E Gigabit Ethernet Controller (Copper)
           +-1c.5-[0000:0f]----00.0  Intel Corporation 82573L Gigabit Ethernet Controller
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
           +-1e.0-[0000:11]----04.0  XGI Technology Inc. (eXtreme Graphics Innovation) Unknown device 0021
           +-1f.0  Intel Corporation 82801IR (ICH9R) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA AHCI Controller
           +-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
           \-1f.6  Intel Corporation 82801I (ICH9 Family) Thermal Subsystem

The device allocation can be seen here (prior to renumbering the devices with the udev rules).

$ find /sys -name 'net:eth*'
/sys/devices/pci0000:00/0000:00:1c.5/0000:0f:00.0/net:eth3
/sys/devices/pci0000:00/0000:00:1c.4/0000:0d:00.0/net:eth2
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/net:eth1
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net:eth0

Create a udev rules file to control device numbering based on PCI bus id, using the ids above. This is the udev rules configuration file '/etc/udev/rules.d/99-ethernet.rules':

KERNEL=="eth*", ID=="0000:0d:00.0", NAME="eth0"
KERNEL=="eth*", ID=="0000:0f:00.0", NAME="eth1"
KERNEL=="eth*", ID=="0000:01:00.0", NAME="eth2"
KERNEL=="eth*", ID=="0000:01:00.1", NAME="eth3"

With this in place the 'alias ethX e1000e' rules can be removed from '/etc/modprobe.conf' file, and the MAC address configurations can be removed from the ifcfg-ethX files.

Logging

Given the stateless Linux support, the log files will be stored in RAM. Send messages to the local syslog server. Add the following line to the '/etc/syslog.conf' configuration file:

*.info;mail.none;authpriv.none;cron.none                @syslog.lucidsolutions.co.nz

Sensors

Install lm-sensors and support for the W83793 sensor.

# yum install lm_sensors
# rpm -ivh w83793-kmdl-2.6.18-128.7.1.el5-1.0-1.RHL5.x86_64.rpm

The sensors configuration for this motherboard (/etc/sysconfig/lm_sensors) should read:

MODULE_0=w83793
MODULE_1=w83627hf

The w83793 is wired such that all fans are controlled by the PWM1 output. Add an '/etc/fancontrol' configuration file:

# Configuration file generated by pwmconfig, changes will be lost
INTERVAL=10
FCTEMPS=hwmon0/device/pwm1=hwmon2/device/temp1_input
FCFANS= hwmon0/device/pwm1=hwmon0/device/fan7_input hwmon0/device/fan2_input hwmon0/device/fan1_input
MINTEMP=hwmon0/device/pwm1=50
MAXTEMP=hwmon0/device/pwm1=60
MINSTART=hwmon0/device/pwm1=10
MINSTOP=hwmon0/device/pwm1=10
MINPWM=hwmon0/device/pwm1=10
MAXPWM=hwmon0/device/pwm1=255

Use the sample fancontrol init script. The script assumes the fancontrol program is located in /usr/local/sbin, and the program doesn't appear to daemonise correctly, so I patches the script after copying.

# cp /usr/share/doc/lm_sensors-2.10.7/fancontrol.init /etc/init.d/fancontrol

The changes to the script.

diff /usr/share/doc/lm_sensors-2.10.7/fancontrol.init /etc/rc.d/init.d/fancontrol
18c18
< FANCONTROL="/usr/local/sbin/fancontrol"
---
> FANCONTROL="/usr/sbin/fancontrol"
26c26
<    daemon $FANCONTROL -d
---
>    daemon $FANCONTROL -d &

Start the service

# chkconfig fancontrol on
# service fancontrol start

Time synchronisation

Use ntp to synchronise the hardware clock.

# yum install ntp

Configure ntp (/etc/ntp.conf) to use time servers that are close. Before running ntp for the first time, step the clock (so that the time is reasonably accurate), using one of the servers listed in the ntp.conf.

# ntpdate 3.nz.pool.ntp.org
 7 Sep 20:15:28 ntpdate[3306]: step time server 218.185.224.7 offset 25262.161492 sec

Configure to start, and run the daemon:

# chkconfig ntpd on
# service ntpd start

Other tools

Install other tools that are useful/essential.

# yum install bind-utils logwatch screen which vixie-cron postfix iptstate yum-utils

Add postfix for as a mailer for daily info. Use a default configuration, except to configure the 'myorigin' and 'relayhost'.

# diff /etc/postfix/main.cf~ /etc/postfix/main.cf
93c93
< #myorigin = $mydomain
---
> myorigin = $mydomain
304c304
< #relayhost = $mydomain
---
> relayhost = $mydomain

Remove the ecryptfs-utils package, as it appears to pull in a large number of X related libraries

# rpm -e ecryptfs-utils.i386 ecryptfs-utils.x86_64

Stateless Linux support for a readonly filesystem

Given the root filesystem is on a compact flash card, the number of writes performed should be reduced. By enabling the stateless Linux support, the temporary state (log files, lock files, tmp etc) is written to a memory backed filesystem. Because the machine has no swap, limit the size of the temporary filesystem.

Edit the '/etc/fstab' file to change the root filesystem to have no atime, and add an explicitly sized tmpfs for temporary state.

LABEL=/                 /                       ext3    noatime        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

# rw access to etc without tmpfs files in the way
/etc                    /mnt/etc                none    noauto,bind     0 0

tmpfs               /var/lib/stateless/writable tmpfs   noauto,size=512M 0 0

Create a directory for the root filesystem to be mounted (as referenced in the fstab above)

# mkdir /mnt/etc

Edit the '/etc/readonly-root' configuration file to enable stateless Linux support. The only change we are making here is to enable the 'TEMPORARY_STATE' option. Note that the 'READONLY' option is not enabled.

# Set to 'yes' to mount the system filesystems read-only.
READONLY=no
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
TEMPORARY_STATE=yes
# Place to put a tmpfs for temporary scratch writable space
RW_MOUNT=/var/lib/stateless/writable
# Label on local filesystem which can be used for temporary scratch space
RW_LABEL=stateless-rw
# Label for partition with persistent data
STATE_LABEL=stateless-state
# Where to mount to the persistent data
STATE_MOUNT=/.snapshot

Configuration Management

The configuration of the router is worthy of full change management. Just about everything in the filesystem has very little change, with the exception of a few configuration files. The following files are managed using subversion:

/etc/modprobe.conf
/etc/fancontrol
/etc/sysconfig/network

/etc/sysconfig/iptables
/etc/sysconfig/iptables-config
/etc/sysconfig/ip6tables
/etc/sysconfig/ip6tables-config
/etc/sysconfig/network-scripts/ifcfg-eth*
/etc/sysconfig/network-scripts/ifcfg-bond0*

Subversion Server

On the subversion server, add a user for the router machine. Allow that account access to the portion of the repository used for the machines configuration.

# htpasswd -m /etc/httpd/conf/svnpasswd orange
New password:
Re-type new password:
Adding password for user orange

Allow the 'orange' user access to the part of the repository, by adding the following to '/etc/httpd/conf/svnauthz'

[config:/trunk/router/orange]
orange = rw

Router

On the router box, install the subversion client. The version from the CentOS repository is v1.4.x (c.f. 1.6.x which is current) - but this is good enough.

# yum install subversion

Checkout the repository, and add files as required. Commit the changes.

# svn co --username orange  https://svn.lucidsolutions.co.nz/config/trunk/router/orange/etc /etc
# svn add -N /etc/sysconfig
A         sysconfig
# svn add -N /etc/sysconfig/network-scripts
A         /etc/sysconfig/network-scripts
# svn add sysconfig/network-scripts/ifcfg-eth*
A         /etc/sysconfig/network-scripts/ifcfg-eth0
A         /etc/sysconfig/network-scripts/ifcfg-eth1
A         /etc/sysconfig/network-scripts/ifcfg-eth2
A         /etc/sysconfig/network-scripts/ifcfg-eth3
# svn add /etc/sysconfig/network-scripts/ifcfg-bond*
A         /etc/sysconfig/network-scripts/ifcfg-bond0
A         /etc/sysconfig/network-scripts/ifcfg-bond0.100
A         /etc/sysconfig/network-scripts/ifcfg-bond0.101
# svn add /etc/sysconfig/network
A         /etc/sysconfig/network
# svn add /etc/sysconfig/iptables
A         /etc/sysconfig/iptables
# svn add /etc/sysconfig/iptables-config
A         sysconfig/iptables-config
# svn add /etc/sysconfig/ip6tables
A         /etc/sysconfig/ip6tables
# svn add /etc/sysconfig/ip6tables-config
A         /etc/sysconfig/ip6tables-config
# svn add /etc/modprobe.conf
A         /etc/modprobe.conf
# svn add /etc/fancontrol
A         /etc/fancontrol
# svn ci -m "Initial configuration files" /etc
Adding         /etc/fancontrol
Adding         /etc/modprobe.conf
Adding         /etc/sysconfig
Adding         /etc/sysconfig/ip6tables
Adding         /etc/sysconfig/ip6tables-config
Adding         /etc/sysconfig/iptables
Adding         /etc/sysconfig/iptables-config
Adding         /etc/sysconfig/network
Adding         /etc/sysconfig/network-scripts
Adding         /etc/sysconfig/network-scripts/ifcfg-bond0
Adding         /etc/sysconfig/network-scripts/ifcfg-bond0.100
Adding         /etc/sysconfig/network-scripts/ifcfg-bond0.101
Adding         /etc/sysconfig/network-scripts/ifcfg-eth0
Adding         /etc/sysconfig/network-scripts/ifcfg-eth1
Adding         /etc/sysconfig/network-scripts/ifcfg-eth2
Adding         /etc/sysconfig/network-scripts/ifcfg-eth3
Transmitting file data ..............
Committed revision 142.

If the configuration is changed externally, checkout the changes using:

# svn up /etc

If the configuration is changed on the machines, then commit those changes using

# svn ci -m "Description of the change" /etc

References

Network script configuration

/etc/modprobe.conf

alias eth0 e1000e
alias eth1 e1000e
alias eth2 e1000e
alias eth3 e1000e

alias bond0 bonding
options bond0 mode=802.3ad lacp_rate=fast miimon=100

/etc/sysconfig/network

HOSTNAME=orange.lucidsolutions.co.nz
NETWORKING=yes
VLAN=yes

NETWORKING_IPV6=yes
IPV6INIT=yes
IPV6FORWARDING=yes

LOCAL_IPV6_PREFIX=2001:4428:225
LOCAL_IPV4_PREFIX=10.20

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
HWADDR=00:15:17:54:7A:8C
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.0.250
NETMASK=255.255.255.0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
HWADDR=00:15:17:54:7A:8D
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
NOZEROCONF=yes
ARP=no
MTU=9000
IPV6_MTU=9000

/etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2
HWADDR=00:30:48:D6:71:16
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
NOZEROCONF=yes
ARP=no
MTU=9000
IPV6_MTU=9000

/etc/sysconfig/network-scripts/ifcfg-eth3

DEVICE=eth3
HWADDR=00:30:48:D6:71:17
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
NOZEROCONF=yes
ARP=no
MTU=9000
IPV6_MTU=9000

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
TYPE=Bonding
NOZEROCONF=yes
MTU=9000
ARP=no

/etc/sysconfig/network-scripts/ifcfg-bond0.100

DEVICE=bond0.100
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=${LOCAL_IPV4_PREFIX}.0.1
NETMASK=255.255.255.0

IPV6_AUTOCONF=no
IPV6ADDR=${LOCAL_IPV6_PREFIX}:0::1

lm_sensors

Dependencies Resolved

============================================================================
 Package          Arch         Version                  Repository     Size
============================================================================
Installing:
 lm_sensors       x86_64       2.10.7-4.el5             base          527 k
Installing for dependencies:
 perl             x86_64       4:5.8.8-18.el5_3.1       updates        12 M

Transaction Summary
============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

w83793 module

# rpm -ivh w83793-kmdl-2.6.18-128.7.1.el5-1.0-1.RHL5.x86_64.rpm
Preparing...                ########################################### [100%]
   1:w83793-kmdl-2.6.18-128.########################################### [100%]

Sections

Personal tools

A CentOS v5 Linux IPv6/IPv4 router appliance

Basic Configuration

Less is more

Networking

IPv4 DHCP Server

Radvd

IPv4 Firewall

IPv6 Firewall

Ethernet Device Numbering

Logging

Sensors

Time synchronisation

Other tools

Stateless Linux support for a readonly filesystem

Configuration Management

Subversion Server

Router

Links

References

Network script configuration

lm_sensors

Document Actions