Control network routes for specific processes on Linux

Intro

I wanted to control the flow of packets for specific processes on my Linux (Ubuntu 22.04) such that they won't go through a tunnel device which is setup for VPN connectivity. The VPN software is provided as is and I do not administer its configuration.

Please note that this post assumes you have understanding of network routing and are equiped to troubleshoot problems. If not you should first learn how networks work first. Each network setup has its own specifics. Appliance vendors come up with defaults and automation for the most common scenarios. Custom configuration can conflict with that and might isolate you from network access. All responsibilities remain those of the reader.

What worked

Summary of the solution

Use the network classifier cgroup (src: docs.kernel.org) to tag network packets with a class identifier (classid) for a process and its subprocesses (src: unix.stackexchange.com). Subsequently use iptables to mark packets which you can use from a custom routing table using policy-based routing (serverfault.com).

Step by step for an example scenario

Imagine that for certain tasks you need to use a VPN that is provided and configured by another team. The VPN software setups up a tunnel interface on your computer and configures a default route for this tunnel device. You still want to use a public service that provides a dedicated client but for which you don't want traffic to go over the VPN (e.g. an audio streaming service like Spotify).

Step 1 : Create a novpn network class

sudo mkdir /sys/fs/cgroup/net_cls
sudo mount -t cgroup -onet_cls net_cls /sys/fs/cgroup/net_cls
sudo mkdir /sys/fs/cgroup/net_cls/novpn
echo 0x100001 | sudo tee /sys/fs/cgroup/net_cls/novpn/net_cls.classid

You can see the upstream documentation on https://docs.kernel.org/admin-guide/cgroup-v1/net_cls.html . Important is the structure of the classid. You can freely chose it as long as it is unique for your system but if you chose another id make sure to check the id doing a cat on the file as it will be needed later:

cat /sys/fs/cgroup/net_cls/novnp/net_cls.classid
1048577

Step 2 : Create custom routing table

In order to create the custom routing table the easiest way is to just 'clone' the default route from your setup when the VPN is not up. You can easily get this using ip:

ip route | grep default
default via 192.168.0.1 dev wlp0s20f3 proto dhcp metric 600

With that it is best to pick a route table number that is not in use such that there is a dedicated routing table. As per serverfault.com this can be done using

ip route show table all | grep -Po 'table \K[^\s]+' | sort -u
local

The example output above would just mean there is no tables besides the reserved routing tables. In our case we want a dedicated route table for traffic that should not go through VPN so lets register it officially with a number not taken. I chose 3 and I can register it as follows:

echo "3\tnovpn"  | sudo tee -a /etc/iproute2/rt_tables
3    novpn

For me this file looks now like:

cat /etc/iproute2/rt_tables
#
# reserved values
#
255    local
254    main
253    default
0    unspec
#
# local
#
#1    inr.ruhep
3    novpn
 

So it shows the reserved values as well as the table id we chose and its logical name 'novpn'. We should also pick a firewall mark to mark the packets. We can check all the iptables tables for marks:

for table in raw security mangle nat filter; do echo "===$table===";sudo /sbin/iptables -L -n --line-numbers --table $table -v; done | grep -o "MARK set [0-9x]*"

On my setup there were no firewall marks yet. I will therefore use mark 2 in this example. Now we have all the inputs to setup our route table

ip rule add fwmark 2 table 3
ip route add default via 192.168.0.1 dev wlp0s20f3 proto dhcp metric 1 table 3
ip route flush cache

With the first command we make sure the routing table is selected for packets with firewall mark 2. With the second command we add the default route to our routing table. And finally we clear cache to make sure new config is used.

Step 3 : Configure the marking using iptables

Since we already decided upon the mark and we have setup the cgroup and know its class id we have everything to configure iptables to mark pakkets as desired:

sudo iptables -t mangle -A OUTPUT -m cgroup --cgroup 1048577 -j MARK --set-mark 2

Step 4: Start our process such that it has the cgroup class id

It is important our processes get the cgroup class id. This can be done by registering their  process ID in /sys/fs/cgroup/net_cls/novpn/tasks . This could become quite tedious since a lot of programs spawn a lot of children. Luckily Linux keeps track of the parent child hierarchy and if we register a process all its children will also be added. So if you spawn a terminal window and register its pid and subsequently spawn the process than the process and all of its children will get associated with this novpn cgroup class.

echo $$ | sudo tee /sys/fs/cgroup/net_cls/novpn/tasks
spotify

Now this Spotify process will have all its packets adhere to routing table 3 which doesn't have a route for the VPN but rather goes as if there wasn't a VPN setup.

Final notes

• I do not represent Spotify I just use their services and it felt to be a useful example (their client application spawns multiple processes and needs to be able to communicate with other OS entities which makes other solutions like using netns more tricky).
• When using the terminal approach, one should take care not to use the terminal for actions that require VPN connectivity as per-design those wouldn't work in that terminal.
• This is a rather static solution if you move your computer and end up on a new network then likely you need a different network configuration (other default route in the custom table)
• Verify the marking of packets of a cgroup is working. An easy trick to verify the tagging is working is by changing the iptables rule to drop the packages rather than to mark them. Then you can easily verify that Spotify works when launched outside the "novpn"  terminal but that it cannot connect to any network when launched inside the "novpn"  terminal.