Homelab


My precious

Homelab


My precious

The homelab rack sits in my home office, just left of me as I type this text. Because it’s so close I’ve been focused on keeping the noise as low as possible; I’ve tried buying second-hand servers, but they are just too loud. So I built the computers myself instead, focusing on keeping them quiet.

Also have a look at any posts with the homelab tag.
images/IMG_20190102_165623.jpg

Introduction

I like Linux, so all my computers run Linux; servers are Ubuntu, desktop and laptops are Arch. I run all my services virtualized with either KVM or LXC, and back up the images and configurations to my file server and the cloud every night. So if anything breaks or I have to take a server down for maintenance, it’s easy to move the guest OS or container to another machine. For that reason, I try to keep network settings the same; bridges and such on the servers.

Top to bottom

images/Homelab_rack.png
  • Two Dell Optiplex 9010 SFF computers (top of rack)
  • 120mm fan plate, to cool the Unifi switch and network equipment at the back
    • UniFi Switch 16XG (back)
    • UniFi Switch 16 POE-150W (back)
  • 1U shelf for my Atlas probe and some Raspberry Pi computers
  • Power control and distribution
    • Power distribution unit; BlueWalker PW PDU RC-16A
    • Automatic power transfer switch; BlueWalker PW ATS
    • Maintenance bypass switch; BlueWalker PW MBS Rack (back)
  • Computers
  • UPS; PowerWalker VI 1500 RT HID

Computers

OPNsense

Firewall, DHCP, DNS, SMTP, OpenVPN

  • CPU: Intel Core i5-3470 @ 3.2GHz, 4 cores
  • Memory: DDR3 1600MHz, 8GB
  • Storage: 128GB SSD
  • Network: Intel Gigabit I340-T4 Quad
  • Type: Dell Optiplex 9010 SFF

Epsilon

Test/backup server, Ubuntu 18.04

  • CPU: Intel Core i5-3470 @ 3.2GHz, 4 cores
  • Memory: DDR3 1600MHz, 16GB
  • Storage: Samsung SSD 850 EVO 250GB
  • Network: Mellanox MNPA19-XTR 10 Gbit
  • Type: Dell Optiplex 9010 SFF

Sigma

Desktop; Arch Linux, i3 tiling window manager

  • CPU: Intel Xeon X5680 @ 3.33GHz, 6 cores
  • Motherboard: ASUS P6T Deluxe V2
  • Memory: Corsair XMS3 DDR3 1600MHz, 12GB
  • Storage: Samsung 850 EVO 250GB 2.5” SSD
  • Network: Mellanox MNPA19-XTR 10 Gbit
  • Case: I&S EYE-4808BK, 4U

Alpha

Hypervisor, Ubuntu 16.04, KVM hypervisor

  • CPU: Intel Core i7-7700K @ 4.2GHz, 4 cores
  • Motherboard: ASUS Z170-P
  • Memory: Corsair Vengeance LPX DDR4 3000MHz, 48GB
  • Storage: Samsung 960 EVO 250GB M.2 PCIe SSD
  • Network: Intel X520-DA2 Dual 10 Gbit
  • Case: I&S EYE-4808BK, 4U

Zeta

File server, Ubuntu 16.04, mdadm RAID 6

  • CPU: Intel Pentium G4560 @ 3.5GHz, 2 cores
  • Motherboard: ASUS Prime B250-Plus
  • Memory: HyperX Fury DDR4 2400MHz, 8GB
  • Storage: Samsung 850 EVO 250GB 2.5” SSD, 7 x 4TB RAID 6 pool
  • Network: Intel X520-DA2 Dual 10 Gbit
  • Case: I&S EYE-4808BK, 4U

Services

  • Unifi controller
  • CCTV server
  • NTP server
  • Docker
  • Mirrors
  • Home Assistant
  • Grafana + InfluxDB
  • Prometheus
  • MQTT broker
  • Plex Media Server
  • Gitea
  • Duplicacy backup
  • Graylog2
  • Reverse proxy
  • Web server

Network

images/Homelab_network.png

I’ve got a 500500 fiber internet that comes into the home office and goes through a media converter on the wall above the rack. From there, an Ethernet cable goes to the OPNsense firewall and a 10 Gbit fiber to the Unifi 16XG switch, which is the backbone of my home network. All computers in the rack are connected to the network with 10 Gbit multi-mode fiber.

In addition to the UniFi 16XG I also have a Unifi 16 POE-150W switch, used for Gbit and PoE devices, such as Raspberry Pies and WiFi access points. Between the two switches in the rack are two Gbit aggregated fiber connections. On the wall behind the rack is a 12 port patch panel that connected the rest of the house to the network.

The network is split into multiple VLAN; LAN, DMZ, and CCTV:

LAN
This is the default network, and it can access devices in all the other networks.
DMZ
The demilitarized-zone, all services accessible from the internet is placed here — things like the NTP server, reverse proxy, SSH jump client and the Atlas probe. It also serves as the wireless guest network. No one trusts each other, and all servers are firewalled. Access to other networks is denied but can be allowed to and from known hosts on specific ports.
CCTV
No access is allowed to any of the other networks, accept from the CCTV server. This prevents the cameras from contacting any internet services and provides a bit of extra protection if someone were to connect to the Ethernet cables running outside.

Here is what /etc/network/interfaces looks like on alpha:

# Loopback
auto lo
iface lo inet loopback

# LAN
auto enp6s0f0
iface enp6s0f0 inet manual

# LAN bridge
auto br0
iface br0 inet static
        address 192.168.1.4
        network 192.168.1.0
        netmask 255.255.255.0
        gateway 192.168.1.1
        dns-nameservers 192.168.1.1
        dns-search lan.uctrl.net

        bridge_ports enp6s0f0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

# DMZ VLAN
auto enp6s0f0.10
iface enp6s0f0.10 inet manual
        vlan-raw-device enp6s0f0

# DMZ VLAN bridge
auto br_dmz
iface br_dmz inet manual
        bridge_ports enp6s0f0.10
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

# CCTV VLAN
auto enp6s0f0.20
iface enp6s0f0.20 inet manual
        vlan-raw-device enp6s0f0

# CCTV VLAN Bridge
auto br_cctv
iface br_cctv inet manual
        bridge_ports enp6s0f0.20
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

As you can see the br_dmz bridge is traffic tagged with VLAN ID 10, and br_cctv with 20. The two VLAN bridges do not have IP addresses, only the virtual networks that use those bridges get IPs.

Power

images/Homelab_power.png

The homelab rack is connected to two different branch circuits, through an automatic transfer switch. The primary power source is the dedicated circuit for the home office; the secondary is the basement circuit. The automatic transfer switch, or ATS for short, provides the ability to switch input source if one should fail. This switch typically takes 9 to 12 ms and is transparent to the load.

Any load on the rack that does not require UPS protection is connected directly to the ATS, such as my workstation computer, monitor, laptop charger and a few other things.

The UPS is connected to the ATS through a maintenance bypass switch (MBS); this allows me to bypass the UPS for maintenance or testing, without turning off any of the connected devices. When the MBS is in bypass mode, the system is without UPS protection.

Last on the power distribution chain is the power distribution unit, or PDU for short. The PDU has eight C14 sockets that can be individually controlled and measured.

The ATS, UPS, and PDU are all connected to a Raspberry Pi acting as power controller and monitor. The UPS has a network management card (NMC) that allows nut to communicate with it over Ethernet. The ATS and PDU are connected via serial port and uses a Python library, that I have written, to communicate. Collected metrics like load, running time, voltage, current, power, and temperatures are read and published as MQTT topics; which Home Assistant subscribes to. The values are also stored in InfluxDB, where they can be read by Grafana to create graphs and cool looking dashboards.

Ventilation

My home office is 10.5 m2 (113 ft2), in the basement and pretty well insulated. With the heat output from the homelab and nowhere for that heat to go, it can get pretty hot. Or; at least it could if I didn’t have a proper ventilation system, which I do.

images/IMG_20190102_222348.jpg

Exhaust

Pulls in air from the ceiling above the rack, and through a fume extraction arm on the electronics lab bench. Both inlets have valves to regulate the flow of air or shut them off completely. The fume extraction on the desk is only open when I solder or do other things that make nasty fumes.

Testing exchaust ventilation using a smoke match.

Outside inlet

An insulated duct transports fresh air from outside into the center of the room; this is done passively when the extractor fan is running. It has a valve that can be closed if it’s freezing outside.

images/20180110_170853.jpg

Inside inlet

I have two 100mm (4”) inlets going into the adjacent room, with valves that can be adjusted. I normally keep them closed and pull in outside air instead. However, in the hot summer months, they can be opened if the basement air is cooler than the outside air.

images/IMG_20190102_222305.jpg

AC

I have a portable AC unit with a dedicated and insulated exhaust duct going outside. When running it exhausts quite a lot of air, and this has to be replaced; this is done either with the outside or inside inlets.

images/IMG_20190102_222247.jpg

Also have a look at any posts with the ventilation tag.

I use a Raspberry Pi and some DS18B20 temperature sensor probes to measure the room temperature; this is published as an MQTT topic which Home Assistant subscribes to. The exhaust fan starts when the temperature reaches 25’C (77’F) and keeps running until the temperature has dropped to 23’C (73.4’C). A Home Assistant automation script is responsible for handling this.

images/2019-01-03-010045_1619x476_scrot.png