Month: September 2017

Run a GNU/Linux Router with Ubiquiti ERL

danny Leave a comment

Background

The Ubiquiti EdgeRouter Lite (ERL) is a bit different than your typical SOHO “router”, which is a combination of a router, a switch, and an access point. I have had some pretty bad experiences with these all-in-one products. The firmware provided with all-in-one routers tends to be very buggy. The idea behind the ERL is that it focuses specifically on the router portion of the gateway. If you have more than one wired client, you will also need a switch. If you don’t have the need for VLANs, you can pick up an inexpensive unmanaged switch. You may also need to purchase a wireless access point if you want wifi. However, if you are replacing an all-in-one router, you can reconfigure it to run in access point mode.

Port Usage

The ERL has 3 ports which can be used in many different configurations. One possibility is to support balancing between two WAN connections. In my use case, I used the additional port for a DMZ. The DMZ could alternatively be implemented logically with a single port using VLANs, but this requires a switch with VLAN support. It is possible to configure it such that two ports belong to the same network, but I don’t recommend this configuration because it will hurt network performance. If not implementing a DMZ or using multiple WANs,  I recommend using one port for a wired LAN and one port for a wireless LAN.

Initial Configuration

Modify the gui to use non-standard ports. This allows us to port forward the default 80 and 443 ports to a webserver hosted in the DMZ.

set service gui http-port 7080
set service gui https-port 7443

Adjust the hostname and time zone to meet your needs.

set system host-name myrouter
set system time-zone America/Chicago

One issue is that by default the DNS server will resolve the router’s hostname as 127.0.1.1. This may be undesirable if trying to ssh into the router from within the LAN or DMZ. To resolve this issue, you can override the IP returned when resolving the hostname of the router itself.

set system ip override-hostname-ip 192.168.102.1
Initial Firewall Setup

Next, perform the initial firewall setup.

edit firewall
set all-ping enable
set broadcast-ping disable
set ipv6-receive-redirects disable
set ipv6-src-route disable
set ip-src-route disable
set log-martians enable
set receive-redirects disable
set send-redirects enable
set source-validation disable
set syn-cookies enable
top
Allow-all Firewall

Create allow-all firewall, which allows through all connections.

edit firewall name allow-all
set default-action accept
set rule 1 action drop
set rule 1 description 'Drop invalid state'
set rule 1 log enable
set rule 1 state invalid enable
top
Allow-Established Firewall

Create the allow-est-drop-inv firewall, which allows through only established connections.

edit firewall name allow-est-drop-inv
set default-action drop
set enable-default-log
set rule 1 action accept
set rule 1 description 'Allow established connections'
set rule 1 state established enable
set rule 1 state related enable
set rule 2 action drop
set rule 2 description 'Drop invalid state'
set rule 2 log enable
set rule 2 state invalid enable
top

Configure LAN Interface

eth2 is configured as my LAN interface with the static IP 192.168.101.1 and acts as the gateway for the 192.168.101.0/24 subnet

edit interfaces ethernet eth2
set address 192.168.101.1/24
set description LAN
set duplex auto
set speed auto
top
Configure LAN DHCP

Next we setup the authoritative DHCP server for the LAN. We want it to make the LAN port on the ERL the default route and DNS server for the LAN subnet. You specify the range of dynamic IP pools. The range of valid hosts is .1 to .254 but we are already using .1 for the router. Since my home network is small, I reserved the .2 through .99 IPs for use as static mappings. The lease 86400 setting indicates a 24 hour DHCP lease duration.

edit service dhcp-server shared-network-name LAN
set authoritative enable
set subnet 192.168.1.0/24 default-router 192.168.101.1
set subnet 192.168.1.0/24 dns-server 192.168.101.1
set subnet 192.168.1.0/24 lease 86400
set subnet 192.168.1.0/24 start 192.168.101.100 stop 192.168.101.254
top

Next we configure a static mapping for each host in the DMZ using their MAC address.

edit service dhcp-server shared-network-name LAN subnet 192.168.1.0/24
set static-mapping myhost1 ip-address 192.168.101.2
set static-mapping myhost1 mac-address 'ab:cd:ef:34:56:78'
set static-mapping myhost2 ip-address 192.168.101.3
set static-mapping myhost2 mac-address 'ab:cd:ef:56:78:90'
top
Configure LAN DNS

Next we specify the desired alias and IP address for each host in the LAN.

edit system static-host-mapping
set host-name myhost1.lan alias myhost1
set host-name myhost1.lan inet 192.168.101.2
set host-name myhost2.lan alias myhost2
set host-name myhost2.lan inet 192.168.101.3
top

Configure LAN firewall

First we create the zone policy for LAN to drop all traffic by default.

edit zone-policy zone LAN
set interface eth2
set default-action drop
top

Then we create the zone policy for local such that all traffic from local is allowed, but limited traffic to local is allowed.

edit zone-policy zone local
set local-zone
set default-action drop
set from LAN firewall name lan-local
set from local firewall name allow-all
top
lan-local firewall

Create the lan-local firewall such that only established connections, ICMP/DNS/DHCP traffic, and SSH/HTTPS for router maintenance are allowed.

edit firewall name lan-local
set default-action drop
set enable-default-log
set rule 1 action accept
set rule 1 description 'Allow established connections'
set rule 1 state established enable
set rule 1 state related enable
set rule 2 action drop
set rule 2 description 'Drop invalid state'
set rule 2 log enable
set rule 2 state invalid enable
set rule 3 action accept
set rule 3 description 'Allow ICMP'
set rule 3 protocol icmp
set rule 4 action accept
set rule 4 description 'Allow SSH/HTTPS'
set rule 4 destination port 22,7443
set rule 4 protocol tcp
set rule 5 action accept
set rule 5 description 'Allow DNS'
set rule 5 destination port 53
set rule 5 protocol tcp_udp
set rule 6 action accept
set rule 6 description 'Allow DHCP'
set rule 6 destination port 67,68
set rule 6 protocol udp
top

Configure WAN Interface

eth0 is configured as my WAN interface and uses DHCP to acquire a dynamic ISP from my ISP via the cable modem

edit interfaces ethernet eth0
set address dhcp
set description WAN
set duplex auto
set speed auto
top

To make this work properly, we need to use NAT to share the single WAN IP address with all hosts in our DMZ and LAN.

edit service nat rule 5010
set description 'Masquerade for WAN'
set outbound-interface eth0
set type masquerade
top

Next, I configure the ERL as a forwarding DNS server. I prefer to use the google public DNS servers. Alternatively, you can use your ISP’s DNS servers, but at least my ISP, I have found them to be less reliable.

edit service dns forwarding
set cache-size 150
set listen-on eth2
set listen-on eth1
set name-server 8.8.8.8
set name-server 8.8.4.4
top

Configure LAN Firewall

Setup the zone policy for the DMZ, allowing all traffic from local and LAN, but only limited traffic back from WAN and DMZ.

edit zone-policy zone WAN
set interface eth0
set default-action drop
set from local firewall name allow-all
set from LAN firewall name allow-all
top

Set firewall for WAN to LAN to only allow established connections.

edit zone-policy zone LAN
set from WAN firewall name allow-est-drop-inv
top

Set firewall for WAN to local to wan-lan, which only allows established connections and SSH/HTTPS for remote router maintenance.

edit zone-policy zone local
set from WAN firewall name wan-local
top
wan-local firewall

Create the wan-local firewall, such that only established connections and SSH/HTTPS (for remote router maintenance) are allowed.

edit firewall name wan-local
set default-action drop
set enable-default-log
set rule 1 action accept
set rule 1 description 'Allow established connections'
set rule 1 state established enable
set rule 1 state related enable
set rule 2 action drop
set rule 2 description 'Drop invalid state'
set rule 2 log enable
set rule 2 state invalid enable
set rule 3 action accept
set rule 3 description 'Allow SSH/HTTPS'
set rule 3 destination port 22,7443
set rule 3 protocol tcp
top

Configure DMZ Interface

The configuration for the DMZ is almost identical to that for the LAN.

eth1 is configured as my DMZ interface with the static IP 192.168.102.1 and acts as the gateway for the 192.168.102.0/24 subnet

edit interfaces ethernet eth1
set address 192.168.102.1/24
set description DMZ
set duplex auto
set speed auto
top
Configure DMZ DHCP

Next we setup the authoritative DHCP server for the DMZ. We want it to make the DMZ port on the ERL the default route and DNS server for the DMZ subnet. You specify the range of dynamic IP pools. The range of valid hosts is .1 to .254 but we are already using .1 for the router. Since my home network is small, I reserved the .2 through .99 IPs for use as static mappings. The lease 86400 setting indicates a 24 hour DHCP lease duration.

edit service dhcp-server shared-network-name DMZ
set authoritative enable
set subnet 192.168.102.0/24 default-router 192.168.102.1
set subnet 192.168.102.0/24 dns-server 192.168.102.1
set subnet 192.168.102.0/24 lease 86400
set subnet 192.168.102.0/24 start 192.168.2.100 stop 192.168.102.254
top

Next we configure a static mapping for each host in the DMZ using their MAC address.

edit service dhcp-server shared-network-name DMZ subnet 192.168.102.0/24
set static-mapping myhost3 ip-address 192.168.102.2
set static-mapping myhost3 mac-address 'ab:cd:ef:12:34:56'
set static-mapping myhost4 ip-address 192.168.102.3
set static-mapping myhost4 mac-address 'ab:cd:ef:23:45:67'
top
Configure DMZ DNS

Next we specify the desired alias and IP address for each host in the LAN and DMZ.

edit system static-host-mapping
set host-name myhost3.dmz alias myhost3
set host-name myhost3.dmz inet 192.168.102.2
set host-name myhost4.dmz alias myhost4
set host-name myhost4.dmz inet 192.168.102.3
top
Port Forwarding

Depending on what services you are hosting within your DMZ, you will need to setup port forwarding to allow external clients to reach your server(s).

For example, if your server with IP 192.168.102.2 is listening for connections on the standard SSH port of 22, you can forward connections to port 2222 to the server.

edit service nat rule 3
set description 'Port forward 2222 to 22'
set destination port 2222
set inbound-interface eth+
set inside-address address 192.168.102.2
set inside-address port 22
set log disable
set protocol tcp
set type destination
top

Configure DMZ firewall

Setup the zone policy for the DMZ, allowing all traffic from local, but allowing only limited traffic from the LAN and WAN.

edit zone-policy zone DMZ
set interface eth1
set default-action drop
set from LAN firewall name lan-dmz
set from WAN firewall name wan-dmz
set from local firewall name allow-all
top

Set firewall for DMZ to WAN to only allow all.

edit zone-policy zone WAN
set from DMZ firewall name allow-all
top

Set firewall for DMZ to LAN to only allow established connections.

edit zone-policy zone LAN
set from DMZ firewall name allow-est-drop-inv
top

Set firewall for DMZ to local to dmz-local to only allow established connections.

edit zone-policy zone local
set from DMZ firewall name dmz-local
top
dmz-local firewall

Create dmz-local firewall such that only ICMP/DNS/DHCP traffic, established connections, and SSH/HTTPS (for router maintenance) are allowed.

edit firewall name dmz-local
set default-action drop
set enable-default-log
set rule 1 action accept
set rule 1 description 'Allow established connections'
set rule 1 state established enable
set rule 1 state related enable
set rule 2 action drop
set rule 2 description 'Drop invalid state'
set rule 2 log enable
set rule 2 state invalid enable
set rule 3 action accept
set rule 3 description 'Allow ICMP'
set rule 3 protocol icmp
set rule 4 action accept
set rule 4 description 'Allow SSH/HTTP/HTTPS'
set rule 4 destination port 22,7443
set rule 4 protocol tcp
set rule 5 action accept
set rule 5 description 'Allow DNS'
set rule 5 destination port 53
set rule 5 protocol tcp_udp
set rule 6 action accept
set rule 6 description 'Allow DHCP'
set rule 6 destination port 67,68
set rule 6 protocol udp
top
lan-dmz Firewall

Create lan-dmz firewall such that only ICMP traffic, NetBIOS/SSH/SMB/HTTPS to a specific host, and established connections are allowed.

edit firewall name lan-dmz
set default-action drop
set enable-default-log
set rule 1 action accept
set rule 1 description 'Allow established connections'
set rule 1 state established enable
set rule 1 state related enable
set rule 2 action drop
set rule 2 description 'Drop invalid state'
set rule 2 log enable
set rule 2 state invalid enable
set rule 3 action accept
set rule 3 description 'Allow ICMP'
set rule 3 protocol icmp
set rule 4 action accept
set rule 4 description 'Allow SSH/SMB/HTTPS'
set rule 4 destination address 192.168.102.2
set rule 4 destination port 22,445,443
set rule 4 protocol tcp
set rule 5 action accept
set rule 5 description 'Allow NetBIOS'
set rule 5 destination address 192.168.102.2
set rule 5 destination port 137,138,139
set rule 5 protocol tcp_udp
top
wan-dmz Firewall

Create wan-dmz firewall such that only established connections and SSH/HTTPS traffic to a specific host are allowed.

edit firewall name wan-dmz
set default-action drop
set enable-default-log
set rule 1 action accept
set rule 1 description 'Allow established connections'
set rule 1 state established enable
set rule 1 state related enable
set rule 2 action drop
set rule 2 description 'Drop invalid state'
set rule 2 log enable
set rule 2 state invalid enable
set rule 3 action accept
set rule 3 description 'Allow SSH/HTTPS'
set rule 3 destination address 192.168.102.2
set rule 3 destination port 22,443
set rule 3 log disable
set rule 3 protocol tcp
top

Configure Hairpinning

A common issue you may encounter is if a host within your LAN uses the public IP address of your router to reach a service hosted within your DMZ, then you need to setup what is known as hairpinning (or loopback NAT). The following commands may be used for accessing a webserver via HTTPS within your DMZ from your LAN using your public IP address. Note that the destination address and port reflect the values after port forwarding rules have been applied.

edit service nat rule 5011
set description 'MASQ for hairpin'
set destination address 192.168.102.0/24
set destination port 443
set log disable
set outbound-interface eth1
set protocol tcp
set source address 192.168.102.0/24
set type masquerade
top

Setup Dynamic DNS

If you are hosting any services from your home network, it is important to know your public IP address. However, one common problem is that many residential ISPs don’t offer static public IP addresses (or only offer them with an expensive monthly charge). Thus, the ISP instead assigns you a dynamic public IP address that may change over time. One way to overcome this is to use a dynamic DNS service which periodically updates a DNS record with your new IP address.

If you do not have a static public IP address, I highly recommend using dynamic DNS to ensure you always know the public IP address of your router. The easiest and cheapest way to accomplish this is to  sign up for an account on duckdns.org for free. The ERL can then automatically keep your IP address up-to-date using the dyndns2 protocol. Once signed up, you just need to replace the values for host-name and password listed in bold below.

edit service dns dynamic interface eth0 service custom-duckdns
set host-name myhostname
set login nouser
set password 12345678-1234-1234-1234-123456789012
set protocol dyndns2
set server www.duckdns.org
top

Monitor Your GNU/Linux Server

danny Leave a comment
Motivation

One of the challenging of hosting services on your own GNU/Linux server is ensuring that the services remain running properly. There are free software monitoring packages that can be used to automatically detect various problems and notify you via email. There are some limitations to running the monitoring software on the same host that you want to monitor, because if the host becomes unreachable, you won’t be notified. However, it can still be useful for verifying that ports are open, processes are running, and performance is normal. If your server uses RAID, it is also critical to get notifications when a drive fails to avoid outages and most importantly prevent data loss. To accomplish these goals, my preference is to use icinga2.

Prerequisites

On Debian 9 GNU/Linux, install the icinga2 package.

sudo apt-get install -y icinga2
Setup IcingaWeb2

Install the icingaweb2 package.

sudo apt-get install -y icingaweb2

Enable the icingaweb2 apache configuration and reload apache.

sudo a2enconf icingaweb2
sudo systemctl reload apache2

Finally open a browser and navigate to /icingaweb2 and follow the installation wizard.

Modify http check to check for https

Add the following bolded line to /etc/icinga2/conf.d/hosts.conf:

object Host NodeName {
 ...
 vars.http_vhosts["http"] = {
 http_uri = "/"
 http_ssl = "true"
 }
}

 

Host Your Own WordPress Blog

danny Leave a comment
Prerequisites

On a Debian 9 GNU/Linux, install the wordpress package and adjust file ownership and permissions for www-data user.

sudo apt-get install -y wordpress
sudo chown -R www-data:www-data /usr/share/wordpress/
sudo chmod -R g+w /usr/share/wordpress/
Configure Apache

Copy the following into /etc/apache2/conf-available/wordpress.conf.

Alias /blog /usr/share/wordpress
<Directory /usr/share/wordpress>
 Options FollowSymLinks
 AllowOverride Limit Options FileInfo
 DirectoryIndex index.php
 Require all granted
</Directory>

Enable the configuration and reload apache.

sudo a2enconf wordpress
sudo systemctl reload apache2

 

Create a Free Software Baby Monitor with Zoneminder

danny Leave a comment
Motivation

When trying to figure out what baby monitor to buy, I found that there are three types of baby monitors: “cloud”-based monitors, IP cameras, and short-range RF monitors without network connectivity. The first two are capable of being accessed remotely, but the last one can only be accessed from within a few hundred feet. The “cloud”-based monitors are convenient but are a huge invasion of privacy because they give a third party access to your baby monitor. If you want to be able to view your baby monitor remotely without sacrificing your privacy, I have found the best choice is using an IP camera and using a free software package called zoneminder to access it.

Prerequisites

On Debian 9 GNU/Linux, install the zoneminder package.

sudo apt-get install -y zoneminder
Enable Web Interface

To enable the web interface, simply enable the zoneminder configuration in apache2 and tell it to reload.

sudo a2enconf zoneminder
sudo systemctl reload apache2
Hardening Zoneminder

Click the Options link in the upper right hand corner, navigate to the Users tab, and click the “Add New User” button. Give the user full permissions and delete the default user. Then navigate to the System tab and enable OPT_USE_AUTH to enable authentication.

Adding a Monitor

Note that the specific settings you want to choose may vary depending on the model of your IP camera. To add a monitor, click the “Add New Monitor” button, choose a unique name for the monitor, and select the Ffmpeg source type. Navigate to the source path and set the source path to “rtsp://user:password@hostname”. Navigate back to the general tab, enable the monitor, and click save.

Host Your Own Music Streaming Service with Subsonic

danny 2 Comments
Motivation

Many people use music streaming services for the convenience and portability they can offer. However, nearly all of the ones I could find require the use of non-free software, use Digital Restrictions Management (DRM), and prevent you from sharing music with friends and family. All of these problems can be avoided by hosting your own music streaming service on your own server.

Prerequisites

On Debian 9 GNU/Linux, install subsonic from the repository.

sudo apt-get install -y subsonic
Configure Subsonic

The configuration file for subsonic is located at /etc/default/subsonic. Modify the SUBSONIC_USER setting to be subsonic. This improves security because if the daemon is compromised, the intruder doesn’t get root access. Modify the SUBSONIC_ARGS to specify the context of /subsonic, the port as 8080 and the max-memory as 150.

SUBSONIC_ARGS="--context-path=/subsonic --port=8080 --max-memory=150"
SUBSONIC_USER=subsonic
Issues with the Default Configuration

By default subsonic runs on a non-standard port and does not use TLS encryption. Since you are entering credentials to subsonic, it a good idea to use TLS encryption to prevent eavesdroppers from discovering your password. While you can configure subsonic to run TLS encryption with self-signed certificate, I found it difficult to get it to use a trusted certificate. Running on a non-standard port may also require you to add an additional port-forwarding rule on your router for you to access subsonic remotely. If you already have apache setup with port-forwarding and a trusted TLS certificate, then this these problems can be solved by simply having apache act as a proxy to the subsonic service.

Configure Apache as a Web Proxy

First ensure that apache is installed and the proxy and proxy_http modules are enabled.

sudo apt-get install -y apache2
sudo a2enmod proxy proxy_http

Then create the text file /etc/apache2/conf-available/subsonic.conf and copy the following content into the file:

SSLProxyEngine On
ProxyPreserveHost Off
ProxyRequests Off
<Location /subsonic>
 ProxyPass http://localhost:8080/subsonic
 ProxyPassReverse http://localhost:8080/subsonic
</Location>

Finally, enable the proxy and reload the apache2 service.

sudo a2enconf subsonic
sudo systemctl reload apache2

Host Your Own Samba File Server

danny Leave a comment
Motivation

A common problem with running a mixed-environment is sharing files between GNU/Linux and Windows computers. The best way to provide Windows computers access to your GNU/Linux file server is using Samba.

Prerequisites

Install samba server and client packages.

$ sudo apt-get install -y samba smbclient
Configure Samba Server

I configure two shares: one read-only (called data) and one read-write (called data-rw). Append the following to the end of /etc/samba/smb.conf:

[data]
 comment = Data Share (read-only)
 path = /mnt/md0/
 available = yes
 valid users = myuser
 read only = yes
 browseable = yes
 public = yes
 writeable = no
[data-rw]
 comment = Data Share
 path = /mnt/md0/
 available = yes
 valid users = myuser
 read only = no
 browseable = yes
 public = yes
 writeable = yes
Create Samba User

Next, create a new samba user using the following command and it will prompt you to specify a password for the user and then again to confirm the password.

sudo smbpasswd -a myuser
New SMB password:
Retype new SMB password:
Verify the Configuration with testparm

The testparm utility checks the /etc/samba/smb.conf configuration file for internal correctness.

$ testparm
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
WARNING: The "syslog" option is deprecated
Processing section "[homes]"
Processing section "[printers]"
Processing section "[print$]"
Processing section "[data]"
Processing section "[data-rw]"
Loaded services file OK.
Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions
Test the Samba Server with smbclient and smbget

smbclient can be used to verify that all the shares are available on the server.

$ smbclient -L euclid
Domain=[WORKGROUP] OS=[Windows 6.1] Server=[Samba 4.5.8-Debian]

 Sharename   Type    Comment
 ---------   ----    -------
 print$      Disk    Printer Drivers
 data        Disk    Data Share (read-only)
 data-rw     Disk    Data Share
 IPC$        IPC     IPC Service (Samba 4.5.8-Debian)
 myuser      Disk    Home Directories
Domain=[WORKGROUP] OS=[Windows 6.1] Server=[Samba 4.5.8-Debian]

 Server         Comment
 ---------      -------
 EUCLID         Samba 4.5.8-Debian

 Workgroup      Master
 ---------      -------
 WORKGROUP      EUCLID

smbget is a wget-like utility for downloading files from a samba share

$ smbget smb://euclid/data/test.txt
Using workgroup WORKGROUP, user myuser
smb://euclid/data/test.txt
Downloaded 830b in 1 seconds

Customize Your Debian GNU/Linux Server

danny Leave a comment
Background

There are many GNU/Linux distributions available but my favorite is Debian. The latest version at the time of this post is Debian 9 “Strecth”. Debian does not come optimally prepared for my specific needs right out of the box. Other distributions come with some of these things pre-installed, but Debian does not want to force them on those who don’t want them. However, with a few quick configuration changes and additional packages, the system can be greatly improved. This post describes the initial changes I prefer to make after installing the stock images.

Ensure the Packages are Updated

The first step is to ensure that all packages are up to date.

# apt-get update
# apt-get upgrade
Install etckeeper

The first thing I like to install is etckeeper, which keeps track of any changes made to etc configuration files with a version control system.  But before I install etckeeper, I first install and configure git. I prefer to do this first so that all my other configuration changes in /etc are tracked in git.

# apt-get install -y git
# git config --global user.name "Jon Doe"
# git config --global user.email "jondoe@example.com"
# apt-get install -y etckeeper
Install vim

Next I like to install vim and make sure it is selected as the default editor.

# apt-get install -y vim
# echo 'SELECTED_EDITOR="/usr/bin/vim.basic"' > ~/.selected_editor

I also run update-alternatives to set vim.basic as my editor of choice. You can optionally use readlink to confirm that the change worked.

# update-alternatives --set editor /usr/bin/vim.basic
# readlink /etc/alternatives/editor
/usr/bin/vim.basic
Modify bashrc

Define PS1 in root’s .bashrc to add color to terminal prompt.

# sed -i \
 -e "s/^# export LS_OPTIONS='--color=auto'$/export LS_OPTIONS='--color=auto'/" \
 -e "s/^# eval \"\`dircolors\`\"$/eval \"\`dircolors\`\"/" \
 -e "s/^# alias ls='ls \$LS_OPTIONS'$/alias ls='ls \$LS_OPTIONS'/" \
 -e "s/^# alias ll='ls \$LS_OPTIONS -l'$/alias ll='ls \$LS_OPTIONS -l'/" \
 -e "s/^# alias l='ls \$LS_OPTIONS -lA'$/alias l='ls \$LS_OPTIONS -lA'/" \
 -e "s/^# alias rm='rm -i'$/alias rm='rm -i'/" \
 -e "s/^# alias cp='cp -i'$/alias cp='cp -i'/" \
 -e "s/^# alias mv='mv -i'$/alias mv='mv -i'/" \
 /root/.bashrc
# echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;31m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]$ '" >> /root/.bashrc
Setup User Account

Next, I create a user account, add it to the sudo group, and update it’s .bashrc file.

# apt-get install -y sudo
# adduser danny
# adduser danny sudo
# sed -i \
 -e "s/^#force_color_prompt=yes$/force_color_prompt=yes/" \
 -e "s/^ #alias ls='ls --color=auto'$/ alias ls='ls --color=auto'/" \
 -e "s/^ #alias grep='grep --color=auto'$/ alias grep='grep --color=auto'/" \
 -e "s/^ #alias fgrep='fgrep --color=auto'$/ alias fgrep='fgrep --color=auto'/" \
 -e "s/^ #alias egrep='egrep --color=auto'$/ alias egrep='egrep --color=auto'/" \
 -e "s/^#alias ll='ls -l'/alias ll='ls -l'/" \
 -e "s/^#alias la='ls -A'/alias la='ls -A'/" \
 -e "s/^#alias l='ls -CF'/alias l='ls -CF'/" \
 -e "$a\\nexport PATH=$PATH:/usr/sbin:/sbin" \
 /home/danny/.bashrc
Keep Clock Synchronized with NTP

To keep the clock synchronized with internet standard time servers, install the ntp daemon.

$ sudo apt-get install -y ntp
Prevent Catastrophic User Error

Next, I like to install several packages that help prevent catastrophic user error. Firstly, I like to install safe-rm, which is a wrapper around the rm command that prevents accidental deletions of files. Secondly, I like to install molly-guard, which guards against accidental shutdowns or reboots by prompting the user for the hostname before allowing the instructions to execute.

$ sudo apt-get install -y safe-rm molly-guard
Ensure Latest Version of Libraries Are In Use

needrestart checks which running daemons need to be restarted after library upgrades. This may help catch issues sooner and ensure that the latest version of the library is being used by all running daemons. It also informs you when a restart is required to use a newer version of the kernel.

$ sudo apt-get install -y needrestart
Make iptables Changes Persistent

By default changes to iptables will not be preserved on reboot. To fix this, I install iptables-persistent and have it save the current running configuration to /etc/iptables/rules.v4 and /etc/iptables/rules.v6 for IPv4 and IPv6, respsectively.

$ sudo apt-get install -y iptables-persistent

Whenever changes are made to iptables, the files can be updated with the following commands for IPv4 and IPv6, respectively:

$ sudo iptables-save > /etc/iptables/rules.v4
$ sudo ip6tables-save > /etc/iptables/rules.v6
Email Notifications Listing Packages Pending an Upgrade

Some upgrades  have important bug or security patches that leave the system vulnerable if they remain unpatched. Email notifications may help keep the system administrator informed about important updates.

$ sudo apt-get install -y apticron
Install and Configure ufw

Uncomplicated firewall (ufw) is an easy to use front-end for netfilter.

$ sudo apt-get install ufw
$ sudo ufw allow ssh
$ sudo ufw enable
Install Optional Utilities

The tree utility lists contents of directories in a tree-like format. The file utility determines file type. The less utility is an alternative pager to the more utility that allows scrolling upwards.

$ sudo apt-get install -y tree file less

The dos2unix utility converts text files from DOS to Unix. The renameutils package provides utilities for quickly moving or copying files, editing the file name in a text editor. The bzip2 package provides utilities for bzip2 compression.

$ sudo apt-get install -y dos2unix renameutils bzip2

The sysstat package which provides iostat and mpstat. The dstat utility is an alternative to iostat and mpstat that is more pretty.

$ sudo apt-get install -y sysstat dstat

The htop utility is an alternative to top but allows you to scroll and looks pretty. The iotop utility is a top-like disk I/O monitor. The itop utility is a top-like interrupt load monitor.

$ sudo apt-get install -y htop iotop itop iftop

The dnsutils package provides dig and nslookup. The bind9-host provides the host utility.

$ sudo apt-get install -y dnsutils bind9-host

The mtr-tiny package provides the mtr utility which combines the ping and traceroute programs in a single diagnostic tool. Th telnet utility is useful for testing TCP connectivity. The nmap utility is a network exploration tool and port scanner. The tcpdump utility dumps traffic on a network.

$ sudo apt-get install -y mtr-tiny telnet nmap tcpdump

Create RAID6 Array with mdadm and xfs

danny Leave a comment

I recently used mdadm to setup RAID6 on my GNU/Linux file servers. The following is a tutorial on the commands I ran on debian 9 stretch to accomplish this task.

Prerequisites

You will need to install the parted, mdadm, and xfsprogs packages. GNU parted is used to create partition tables. mdadm is used to create and maintain multi devices. The xfsprogs package contains several xfs related utilities including mkfs.xfs and xfs_info.

$ sudo apt-get install parted mdadm xfsprogs
Create Partition Table

The first step in preparing your disk drives is to create the GPT partition table.

$ sudo parted /dev/sdb mklabel gpt
Create Partition

Next you need to create the partitions. While this step is optional since the RAID array can be built from devices, it is recommended that you use partitions so that down the road you have the option of replacing a drive with one of equal or greater capacity. For my purposes, I just created a single partition that consumes all of the available space.

The “-a optimal” option tells parted to use optimal alignment.

$ sudo parted -a optimal /dev/sdb mkpart primary 0% 100%
Enable the RAID Flag

Next you need to set the raid flag on the partition.

$ sudo parted /dev/sdb set 1 raid on
Verify the Configuration

You can verify that the Partition Table is type gpt, the primary partiton has been created with the appropriate size and that the raid flag is set on that partition.

$ sudo parted /dev/sdb print
Model: ATA HGST HDN724040AL (scsi)
Disk /dev/sdb: 4001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number Start  End    Size   File system Name    Flags
1      1049kB 4001GB 4001GB             primary raid
Create the Multi Device

After you have completed the previous steps on all disks, the next step is to create the multi device (md). In my case, I am creating a RAID6 multi device from 7 partitions.

$ sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=6 /dev/sd{b,c,d,e,f,g,h}1

The multi device is created, but it will not be preserved on reboot. To preserve the configuration on reboot we need to append to the mdadm configuration file. You may also need to update initramfs and grub to ensure the multi device is automatically assembled on reboot.

$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
$ sudo update-initramfs -u
$ sudo update-grub
Create the XFS Filesystem

Now that we have a multi device, we can create the filesystem. I recommend using xfs if the effective capacity of your multi device may eventually grow to be more than 16TB of hard drive space.

The “-d” option allows us to specify one or more disk parameters.

The sunit disk parameter stands for Stripe Unit. It is optimal when assigned to the Stripe Size in bytes divided by 512. In my case the Stripe Size is 512KB, and thus the optimal Stripe Unit is 1024.

The swidth disk parameter stands for Stripe Width. It is defined by the number of non-parity disks times Stripe Unit. In my case, I am using RAID6 with a total of 7 disks. Since RAID6 requires 2 parity disks, that leaves me with 5 non-parity disks. Thus the optimal Stripe Width is 5120 (5 non-parity disks times the 1024 Stripe Unit).

$ sudo mkfs.xfs -d sunit=1024,swidth=5120 /dev/md0

You can verify the sunit and swidth settings aftewards, but it can be tricky because the units are different. With mkfs.xfs they are defined as a multiple of 512-bytes. With xfs_info, they are defined as a multiples of the block size (bsize). In this case bsize is 4096. So the values appear as 1/8th of the value we specified in the mkfs.xfs command.

$ sudo xfs_info /dev/md0
meta-data=/dev/md0 isize=256 agcount=32, agsize=152612736 blks
 = sectsz=4096 attr=2, projid32bit=1
 = crc=0 finobt=0 spinodes=0 rmapbt=0
 = reflink=0
data = bsize=4096 blocks=4883607040, imaxpct=5
 = sunit=128 swidth=640 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=521728, version=2
 = sectsz=4096 sunit=1 blks, lazy-count
Alternative: Create the ext4 Filesystem

If you are certain you will never exceed the 16TB limit, then ext4 is a reasonable alternative choice for a filesystem. Note that you should only run this command if you are using ext4 instead of xfs.

$ sudo mkfs.ext4 -F /dev/md0
Mount the Multi Device

A mount point is simply the directory to which the device is mounted. So we just need to make sure that directory exists and mount the multi device to the newly created mount point.

$ sudo mkdir -p /mnt/md0
$ sudo mount /dev/md0 /mnt/md0

Next we update fstab to preserve the changes on reboot.

$ echo '/dev/md0 /mnt/md0 xfs defaults,nofail 0 0' | sudo tee -a /etc/fstab

Finally df can be used to verify the disk is mounted and displays the usage.

$ df -h -t xfs
Filesystem Size Used Avail Use% Mounted on
/dev/md0   19T  0T   0T    0%   /mnt/md0
Verify the Setup

You can verify the setup with lsblk. It shows the disk device and capacity, the raid partition, and multi device and its capacity, filesystem type, raid level, and mount point.

$ lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT /dev/sd{b,c,d,e,f,g,h}
NAME SIZE FSTYPE TYPE MOUNTPOINT
sdb 3.7T disk
└─sdb1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0
sdc 3.7T disk
└─sdc1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0
sdd 3.7T disk
└─sdd1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0
sde 3.7T disk
└─sde1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0
sdf 3.7T disk
└─sdf1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0
sdg 3.7T disk
└─sdg1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0
sdh 3.7T disk
└─sdh1 3.7T linux_raid_member part
 └─md0 18.2T xfs raid6 /mnt/md0

You can view the state of multi devices using the proc virtual filesystem.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdc1[4] sdg1[3] sde1[2] sdd1[7] sdf1[6] sdh1[0] sdb1[5]
 19534428160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
 bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

Lastly, you can view detailed information about a multi device using mdadm.

$ sudo mdadm --detail /dev/md0
/dev/md0:
 Version : 1.2
 Creation Time : Mon May 22 21:28:46 2017
 Raid Level : raid6
 Array Size : 19534428160 (18629.48 GiB 20003.25 GB)
 Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB)
 Raid Devices : 7
 Total Devices : 7
 Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Fri Sep 8 20:07:07 2017
 State : clean
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
 Spare Devices : 0

Layout : left-symmetric
 Chunk Size : 512K

Name : euclid:0 (local to host euclid)
 UUID : a7faccce:58914d5e:b26cce7e:795b773b
 Events : 48719

Number Major Minor RaidDevice State
 0 8 113 0 active sync /dev/sdh1
 6 8 81 1 active sync /dev/sdf1
 7 8 49 2 active sync /dev/sdd1
 3 8 97 3 active sync /dev/sdg1
 4 8 1 4 active sync /dev/sdc1
 5 8 17 5 active sync /dev/sdb1
 2 8 65 6 active sync /dev/sde1

Build Your Own Homelab

danny Leave a comment
Motivation:

When I began my research to purchase hardware for a homelab, I wanted to purchase enough parts for two complete systems that could mirror each others data. The primary goal is to have enough storage capacity to hold all of my data and reduce the odds of losing data in the event of hardware failure or user error. Although RAID6 itself provides some redundancy, it does not provide a backup.  By having the data stored on two independent file servers, a complete backup of all files is maintained. For example if I accidentally delete a file from one server, my backup server will maintain a copy of that file until the next time the drives are synchronized. A secondary goal is to use the server to reliably host a number of useful services at my home within a DMZ that can be accessed remotely.

Requirements:

Since I am running the servers as headless GNU/Linux boxes, it is very useful to have IPMI for initial setup and maintenance. I also was interested in having hot-swap hard drive bays so I could replace failed hard drives without having to power down or open the case. Noise and heat were also a major concerns for me because my homelab is located in my living room and I didn’t want servers that sounded jet engines or produced too much heat. Lastly, I am interested in the case being rack-mountable for flexibility in the future.

Server Hardware:

The SSDs are SATA rather than M.2 and a couple generations older than the rest of the system, but I used them because I already had them lying around. Similarly I already had eight 4TB HGST DeskStar NAS drives and decided to buy more to expand my capacity. With 7x 4TB drives in a software RAID6 array, each server now has 20TB of effective capacity. I initially purchased the X11SSH-LN4F motherboard intending to use it to avoid double NAT and act as a VPN gateway, but after some experimentation, I found that having my GNU/Linux server act as the router for my home network was problematic. For the second motherboard I purchased the X11SSM-F which offers additional PCIe slots at a cheaper price, but the downside to it is it is missing the M.2 slot. I chose the RAM based on the Supermicro tested memory list to avoid any incompatibilities. After a lot of research I decided to go with the 743TQ case based on its balance of price, “whisper-quiet” noise level (<28dB), compatibility with Supermicro motherboards, ability to stand on its own or be rack-mounted, and for it’s ability to expand to support 13 hard drives (with the purchase of a mobile rack that replaces the 3x 5.25″ bays). I purchased the additional PWM case fans to keep all the hard drives cool. Lastly the Xeon E3-1200 series processor is probably overkill for my use case, but it may come in handy down the road if I decide to do anything CPU intensive such as hosting many virtual servers. Although there were slightly cheaper models, I opted to pay a little more to get hyper-threading support and the higher CPU frequency. However, I did not get the most expensive models because of the diminishing returns on performance for the price.

Network Hardware:

Instead of using my GNU/Linux server as a router, I decided to purchase a Ubiquiti EdgeRouter Lite and have been very happy with the price and performance of this product. To avoid monthly rental costs, I also replaced my Xfinity router with a cable modem that I own and control. I also reused my Cisco IP Phone that I’ve used for years for my custom home telephony service.