Efficient service isolation on Alpine with VRFs

Over the weekend, a reader of my blog contacted me basically asking about firewalls.  Firewalls themselves are boring in my opinion, so let’s talk about something Alpine can do that, as far as I know, no other distribution can easily do out of the box yet: service isolation using the base networking stack itself instead of netfilter.

A note on netfilter

Linux comes with a powerful network filtering framework, called netfilter. In most cases, netfilter is performant enough to be used for packet filtering, and the newer nftables project provides a mostly user friendly interface to the system, by comparison to the older iptables and ip6tables commands.

However, when trying to isolate individual services, a firewall is sometimes more complicated to use, especially when it concerns services which connect outbound. Netfilter does provide hooks that allow for matching via cgroup or PID, but these are complicated to use and carry a significant performance penalty.

seccomp and network namespaces

Two other frequently cited options are to use seccomp and network namespaces for this task. Seccomp is a natural solution to consider, but again has significant overhead, since all syscalls must be audited by the attached seccomp handler. Although seccomp handlers are eBPF programs and may be JIT compiled, the performance cost isn’t zero.

Similarly, one may find network namespaces to be a solution here. And indeed, network namespaces are a very powerful tool. But because of the flexibility afforded, network namespaces also require a lot of effort to set up. Importantly though, network namespaces allow for the use of an alternate routing table, one that can be restricted to say, the management LAN.

Introducing VRFs

Any network engineer with experience will surely be aware of VRFs. The VRF name is an acronym which stands for virtual routing and forwarding. On a router, these are interfaces that, when packets are forwarded to them, use an alternative routing table for finding the next destination. In that way, they are similar to Linux’s network namespaces, but are a lot simpler.

Thanks to the work of Cumulus Networks, Linux gained support for VRF interfaces in Linux 4.3. And since Alpine 3.13, we have supported managing VRFs and binding services to them, primarily for the purpose of low-cost service isolation. Let’s look at an example.

Setting up the VRF

On our example server, we will have a management LAN of 10.20.30.0/24. A gateway will exist at 10.20.30.1 as expected. The server itself will have an IP of 10.20.30.40. We will a single VRF, in conjunction with the system’s default route table.

Installing the needed tools

By default, Alpine comes with Busybox’s iproute2 implementation. While good for basic networking use cases, it is recommended to install the real iproute2 for production servers. To use VRFs, you will need to install the real iproute2, using apk add iproute2-minimal, which will cause the corresponding ifupdown-ng modules to be installed as well.

Configuring /etc/network/interfaces

We will assume the server’s ethernet port is the venerable eth0 interface in Alpine. First, we will want to set up the interface itself and it’s default route. If you’ve used the Alpine installer, this part should already be done, but we will include the configuration snippet for those following along.

auto eth0
iface eth0
address 10.20.30.40/24
gateway 10.20.30.1

The next step is to configure a VRF. In this case, we want to limit the network to just the management LAN, 10.20.30.0/24. At the moment, ifupdown-ng does not support configuring interface-specific routes out of the box, but it’s coming in the next version. Accordingly, we will use iproute2 directly with a post-up directive.

auto vrf-management
iface vrf-management
requires eth0
vrf-table 1
pre-up ip -4 rule add pref 32765 table local
pre-up ip -6 rule add pref 32765 table local
pre-up ip -4 rule del pref 0
pre-up ip -6 rule del pref 0
pre-up ip -4 rule add pref 2000 l3mdev unreachable
post-up ip route add 10.20.30.0/24 dev eth0 table 1

This does four things: first it creates the management VRF, vrf-management using the second kernel route table (each network namespace may have up to 4,096 routing tables).  It also asserts that the eth0 interface must be present and configured before the VRF is configured.  Next, it removes the default route lookup rules and moves them so that the VRFs will be checked first.  Finally, it then adds a route defining that the management LAN can be accessed through eth0. This allows egress packets to make their way back to clients on the management LAN.

In future versions of ifupdown-ng, the routing rule setup will be handled automatically.

Verifying the VRF works as expected

Once a VRF is configured, you can use the ip vrf exec command to run a program in the specified VRF context. In our case, the management VRF lacks a default route, so we should be able to observe a failure trying to ping hosts outside the management VRF, using ip vrf exec vrf-management ping 8.8.8.8 for example:

localhost:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=1.287 ms
^C
-– 8.8.8.8 ping statistics — 1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.287/1.287/1.287 ms
localhost:~# ip vrf exec vrf-management ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network unreachable

Success!  Now we know that using ip vrf exec, we can launch services with an alternate routing table.

Integration with OpenRC

Alpine presently uses OpenRC as its service manager, although we plan to switch to s6-rc in the future.  Our branch of OpenRC has support for using VRFs, for the declarative units.  In any of those files, you just add vrf=vrf-management, to the appropriate /etc/conf.d file, for example /etc/conf.d/sshd for the SSH daemon.

For services which have not been converted to use the declarative format, you will need to patch them to use ip vrf exec by hand.  In most cases, all you should need to do is use ${RC_VRF_EXEC} in the appropriate place.

As for performance, this is much more efficient than depending on netfilter, although the setup process is not as clean as I’d like it to be.