Upgrading our Internet subscription

We finally decided to upgrade the internet connection at the lab from the old 100Mbps contract, and what an upgrade.

We're switching to the Fiber7-X2 offer from Init7, with a stupidly high bandwidth of 25Gbps symmetrical.

(Oh and if you're a member and still reading, you might get some interesting mail very soon 👀)

But we signed our new contract the day the previous one ended, and need to wait up to 10 days for the new one to get activated.

In the meantime, we decided to switch to a 5G router. But hosting services like email or web applications behind a NATed address from the 5G provider is not possible.

Here is a small write-up about the solutions we put in place to make sure most of the things won't break.

Link to the yEd graph

We were running a pfSense firewall with a DMZ (De-Militarized Zone, i.e. isolated from the rest of the network) machine hosting most of the services that had been running for years, like the website, wiki and email, all on a dedicated IPv4 address; and a LAN sub-net for the WiFi and other servers running inside the lab that don't need to be directly available on internet, using a second dedicated IPv4 address.

Then we started deploying a new server with some new services like the Matrix or NextCloud instances, and also hosting virtual machines for the members that also need to be accessible from internet. To allow this traffic, we just added a few rules in the firewall to NAT from the external LAN IPv4 to this new server.

Link to the yEd graph

Since we only had less than a day to put in place a solution for keeping internet access in the lab - and the mail server running to not miss a bill - we used a Huawei 5G router that micmac had as a backup at home, and bridged it so that the device behind would get all the internet traffic directly.

Then I set up a VLAN on the main switch, connected the 5G router there, and set up a new VLAN interface on pfSense so that it would get the whole 5G traffic from the existing cabling. Switching the default gateway to this interface in pfSense, and boom, we had internet again in the lab.

The problem now are: how do we get traffic in, since there is no fixed IPv4 (and, even less so, two of them); and how do we configure the NAT to get the traffic where we need in our local sub-network.

For this, I asked Binary_Brain from FixMe to run a virtual machine with two public IPv4 address on their infrastructure, and experimented a bit with it all night.

Early in the morning, Erik Rossen came to my rescue since I kinda fucked up the DNS at some point, and helped me put in place some VPN tunnels from the VM at FixMe to the two main machines we had running in the lab (the DMZ one accetta, and the new one berserk).

That way, VPN tunnels went through the 5G router and then the VM at FixMe, and using some iptables rules, we redirected the traffic coming from the two IPv4 from FixMe to either our LAN or DMZ servers via the tunnels.

Link to the yEd graph

After installing the new Mikrotik router, we removed the pfSense firewall from the loop since most of the features are also available there, but with way more bandwidth available.

2022-03-18: We only have 1Gpbs because Init7 still need some time to enable the 25Gpbs line. Also we haven't installed the new network card yet in berserk, so this setup is working well enough. The Mikrotik config still needs some tuning but will be published on GitLab at some point (if not, ping Maxime Borges)

Link to the yEd graph

Well, not really final I hope, otherwise things will get boring pretty quickly. Actually we will still keep the other servers for some time until we have something we're happy (and confident) enough software-wise on berserk.

At the moment we're experimenting with Kubernetes and Infrastructure-as-Code processes (see the GitLab repo) but that will be for another blog post.

Anyway, in this setup berserk will be able to get all the juice from Init7, and we will move remaining services from accetta there, and maybe also move the NAS and other somewhat legacy services from coltello.