Setting Up Multiple VPN Routes for Load Balancing

When a single VPN tunnel becomes a bottleneck or a single point of failure, the answer is to deploy multiple VPN routes and distribute flows across them. In practice, this means combining Linux routing primitives (ECMP and Policy-Based Routing), correct MTU/MSS strategy, health-checking, and—at scale—dynamic routing (BGP/OSPF with FRRouting) so paths adapt automatically. This article is a deeply technical, copy-paste-ready guide you can drop into WordPress. It covers design trade-offs, step-by-step Linux configurations with iproute2 and nftables, options for IPsec VTI, OpenVPN, and WireGuard, as well as operational hardening, observability, and troubleshooting.

 

1) Principles of VPN Load Balancing

VPN load balancing is the practice of spreading traffic across multiple encrypted tunnels to improve throughput and resilience. In IP routing terms you will use:

• ECMP (Equal-Cost Multi-Path): install multiple next-hops of equal cost; the kernel hashes flows across them.
• PBR (Policy-Based Routing): steer traffic based on rules—packet marks, source networks, ports—into dedicated routing tables.
• Dynamic routing: run BGP or OSPF across the tunnels so the control plane withdraws bad paths and installs alternatives automatically; multipath occurs when costs are equal.

Design around per-flow distribution (to avoid TCP reordering), explicit health checks (to avoid blackholes), and consistent MTU/MSS settings (to avoid PMTU blackholes).

---

2) Architecture Patterns & Design Choices

• Static ECMP: simplest; install multiple default routes with equal weight via tunnel interfaces. Add an external health mechanism to remove dead next-hops.
• PBR with Marks: granular control; e.g., send 70% of flows via tunnel A and 30% via tunnel B, or segregate applications by port or source.
• Dynamic (BGP/OSPF): highly scalable; sessions ride the tunnels (IPsec VTI, GRE-over-IPsec, or WireGuard) and multipath arises naturally with equal costs. Integrate BFD for sub-second failure detection.

Addressing: assign unique /30 or /31 tunnel subnets or use loopbacks as BGP endpoints. Keep addressing consistent for simpler policy and observability.

---

3) Prerequisites & Preparation

• Linux (5.x+ recommended) with iproute2, nft (or iptables), and root access.
• Two or more working VPN tunnels with unique tunnel IPs and reachable next-hops (examples below show tun0 → 10.0.0.1, tun1 → 10.0.1.1).
• Clear MTU/MSS plan (see Section 6) and log/metric collection in place.
• (Recommended) FRRouting (FRR) if you plan to scale with BGP/OSPF + BFD.

---

4) ECMP on Linux with iproute2

Goal: split flows evenly across two tunnels using the kernel’s multipath hash.

# Set conservative tunnel MTU (adjust as needed) ip link set dev tun0 mtu 1420 ip link set dev tun1 mtu 1420
Install an ECMP default route with two equal-cost next-hops

ip route replace default scope global
nexthop via 10.0.0.1 dev tun0 weight 1
nexthop via 10.0.1.1 dev tun1 weight 1

Make sure layer-4 ports are considered in the flow hash (better distribution)

sysctl -w net.ipv4.fib_multipath_hash_policy=1

The kernel chooses a next-hop based on a hash (src/dst, proto, L4 ports when enabled). ECMP is simple and fast, but it will not automatically remove a next-hop that is “up” administratively yet failing trafficwise; add health checks (Section 7).

---

5) Policy-Based Routing (PBR) with Packet Marking

Goal: direct subsets of traffic to specific tunnels by marking packets (with nftables) and mapping marks to routing tables.

5.1 Create per-tunnel routing tables

echo "200 vpn1" >> /etc/iproute2/rt_tables echo "201 vpn2" >> /etc/iproute2/rt_tables

ip route add default via 10.0.0.1 dev tun0 table vpn1
ip route add default via 10.0.1.1 dev tun1 table vpn2

5.2 Mark traffic with nftables (per-flow balanced example)

We’ll evenly mark TCP/UDP flows to alternate tunnels using a stable hash on the 5-tuple. Adjust to your needs (e.g., weight or application-based split).

# Create mangle table/chains nft add table inet mangle nft 'add chain inet mangle prerouting { type filter hook prerouting priority mangle; }' nft 'add chain inet mangle output { type route hook output priority mangle; }'
TCP MSS clamp to avoid PMTU issues (tune value to your MTU)

nft 'add rule inet mangle output tcp flags syn tcp option maxseg size set 1360'

Evenly split by hashing L4 ports (per-flow). This is a simple demo.

nft 'add rule inet mangle output meta l4proto { tcp, udp } th dport . th sport mod 2 == 0 meta mark set 1'
nft 'add rule inet mangle output meta l4proto { tcp, udp } th dport . th sport mod 2 == 1 meta mark set 2'

5.3 Map marks to routing tables

ip rule add fwmark 1 lookup vpn1 ip rule add fwmark 2 lookup vpn2 

Weighted splits (e.g., 70/30): either use ECMP weights (weight 7 vs weight 3) or send more hash outcomes to mark 1 than mark 2 by altering your nftables logic (e.g., mod 10 < 7 → mark 1).

---

6) Per-flow vs Per-packet, MTU & MSS

• Per-flow: keep a TCP/UDP flow on the same tunnel. Minimizes reordering and maximizes throughput. Recommended.
• Per-packet: typically harmful for TCP (reordering). Avoid unless you fully control both ends and can tolerate out-of-order delivery.
• MTU/MSS: VPN encapsulation adds overhead (40–80+ bytes). Set tunnel MTU conservatively (e.g., 1420 as a starting point) and clamp TCP MSS (e.g., 1360) to avoid blackholes.

# Adjust MTU per tunnel (examples) ip link set dev tun0 mtu 1420 ip link set dev tun1 mtu 1420
MSS clamp already shown in nftables rule above; iptables equivalent:
iptables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360

---

7) Health Checks & Automatic Failover

Do not rely on interface “UP” alone—detect data-plane failure and remove bad next-hops quickly.

• BFD (Bidirectional Forwarding Detection) via FRR: sub-second liveness for tunnel next-hops; ideal with BGP/OSPF.
• Keepalived (VRRP + track_script): periodic checks (ping/HTTP) that can adjust routes or fail a VIP between nodes.
• Custom bash scripts: low friction; ping from each tunnel and rewrite the default route or ECMP set.

7.1 Minimal bash health check (ECMP toggle)

#!/usr/bin/env bash # /usr/local/bin/check_tunnels.sh T0_NH=10.0.0.1 T1_NH=10.0.1.1

ping -I tun0 -c 2 -W 1 "$T0_NH" >/dev/null; T0=$?
ping -I tun1 -c 2 -W 1 "$T1_NH" >/dev/null; T1=$?

if [[ $T0 -eq 0 && $T1 -eq 0 ]]; then
ip route replace default
nexthop via $T0_NH dev tun0 weight 1
nexthop via $T1_NH dev tun1 weight 1
elif [[ $T0 -eq 0 ]]; then
ip route replace default via $T0_NH dev tun0
elif [[ $T1 -eq 0 ]]; then
ip route replace default via $T1_NH dev tun1
else
logger -t vpn "All tunnels down"
fi

7.2 Run every 5s with systemd

# /etc/systemd/system/vpn-health.service [Unit] Description=VPN health check [Service] Type=oneshot ExecStart=/usr/local/bin/check_tunnels.sh
/etc/systemd/system/vpn-health.timer

[Unit]
Description=Run VPN health check periodically
[Timer]
OnBootSec=10
OnUnitActiveSec=5
Unit=vpn-health.service
[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now vpn-health.timer

7.3 Keepalived example (track script)

# /etc/keepalived/keepalived.conf vrrp_script chk_tunnels { script "/usr/local/bin/check_tunnels.sh" interval 5 timeout 3 }

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 120
advert_int 1
track_script { chk_tunnels }
}

---

8) Dynamic Routing (BGP/OSPF) & Multipath with FRRouting (FRR)

In larger environments, run a routing protocol across tunnels. When both tunnels offer equal cost, ECMP is installed automatically; when one fails, routes are withdrawn. Add BFD for sub-second convergence.

8.1 FRR BGP with multipath

# /etc/frr/daemons bgpd=yes bfdd=yes
/etc/frr/bfdd.conf

bfd
peer 10.0.0.1
transmit-interval 50
receive-interval 50
multiplier 3
!
peer 10.0.1.1
transmit-interval 50
receive-interval 50
multiplier 3
!

/etc/frr/bgpd.conf

router bgp 65001
bgp router-id 192.0.2.10
timers bgp 10 30
neighbor TUNNEL peer-group
neighbor TUNNEL remote-as 65002
neighbor 10.0.0.2 peer-group TUNNEL
neighbor 10.0.1.2 peer-group TUNNEL
!
address-family ipv4 unicast
network 10.10.0.0/24
maximum-paths 4
neighbor TUNNEL activate
neighbor TUNNEL bfd
neighbor TUNNEL send-community both
exit-address-family

8.2 OSPF alternative

Enable OSPF over the tunnel interfaces, set equal cost, and optionally tie interfaces to BFD. OSPF will ECMP across equal-cost neighbors and reconverge rapidly on failure.

---

9) Stack-Specific Notes: IPsec VTI, OpenVPN, WireGuard

9.1 IPsec (VTI / xfrm)

• VTI gives you a routed interface (vti0) per tunnel. This is ideal for BGP/OSPF and simplifies ECMP/PBR.
• Align IKE/Child SA lifetimes (enable make-before-break) so rekeys don’t flap tunnels and trigger route churn.
• With plain xfrm (no VTI), policy-based selectors can complicate multipath; prefer VTI where possible for clean routing semantics.

9.2 OpenVPN

• Use dev tun, assign unique tunnel IPs (tun0, tun1), and prefer UDP mode for performance.
• Avoid TCP-over-TCP where possible (head-of-line blocking). If unavoidable, ECMP carefully and keep per-flow hashing.
• Use --mssfix judiciously; prefer system-wide MSS clamp via nftables for consistency across apps.

9.3 WireGuard

• Lightweight and fast; easily spin multiple peers/interfaces and do ECMP/PBR on top.
• For NAT traversal, set PersistentKeepalive (e.g., 15s) on the WG peer.
• WG’s low CPU cost makes it a good candidate for more parallel tunnels when bandwidth is the limiting factor.

---

10) Security Hardening & Operational Risk

• Control-plane isolation: do not expose BGP TCP/179 to the Internet; peer BGP/OSPF only inside the tunnels.
• Prefix hygiene: for BGP, enforce prefix-lists and maximum-prefix; never accept/announce unexpected networks.
• Key/cert hygiene: rotate keys, protect private keys (0600), and consider HSM/PKCS#11 for IPsec.
• Observability: export per-tunnel latency, packet loss, BFD status, route counts, and rekey events to Prometheus/Grafana.
• Symmetry: ensure both directions see the same “best path”; asymmetry can trigger stateful firewall drops.

---

11) Verification & Troubleshooting

11.1 Quick checks

# Routes and ECMP ip route show default ip route get 8.8.8.8
Policy and tables

ip rule show
ip route show table vpn1
ip route show table vpn2

Interface health & counters

ip -s link show tun0
ip -s link show tun1

nftables rules/counters

nft list ruleset

11.2 Path tests

Launch multiple TCP flows to different destinations/ports and observe distribution. Use mtr/traceroute with -s/-p variations, confirm flows are sticky to a single tunnel. Verify both tunnel next-hops answer ping -I tunX.

11.3 Common pitfalls

• Asymmetric routing: ensure per-flow hashing and matching policies both ways (dynamic routing greatly helps).
• PMTU blackhole: fix with correct tunnel MTU and TCP MSS clamp; test with ping -M do -s <size>.
• Blackhole next-hops: add health checks to remove unusable ECMP members rapidly.

---

12) End-to-End Reference Implementation (ECMP + PBR + Health)

Goal: two tunnels (tun0, tun1) with per-flow load sharing, MSS clamp, and automatic failover.

12.1 MTU and ECMP default

ip link set tun0 mtu 1420 ip link set tun1 mtu 1420

ip route replace default
nexthop via 10.0.0.1 dev tun0 weight 1
nexthop via 10.0.1.1 dev tun1 weight 1

sysctl -w net.ipv4.fib_multipath_hash_policy=1

12.2 nftables per-flow split + MSS clamp

nft add table inet mangle nft 'add chain inet mangle output { type route hook output priority mangle; }' nft 'add rule inet mangle output tcp flags syn tcp option maxseg size set 1360' nft 'add rule inet mangle output meta l4proto { tcp, udp } th dport . th sport mod 2 == 0 meta mark set 1' nft 'add rule inet mangle output meta l4proto { tcp, udp } th dport . th sport mod 2 == 1 meta mark set 2' 

12.3 PBR rules

echo "200 vpn1" >> /etc/iproute2/rt_tables echo "201 vpn2" >> /etc/iproute2/rt_tables

ip route add default via 10.0.0.1 dev tun0 table vpn1
ip route add default via 10.0.1.1 dev tun1 table vpn2

ip rule add fwmark 1 lookup vpn1
ip rule add fwmark 2 lookup vpn2

12.4 Health check + systemd timer

# /usr/local/bin/check_tunnels.sh (script from Section 7.1) chmod +x /usr/local/bin/check_tunnels.sh
/etc/systemd/system/vpn-health.service

[Unit]
Description=VPN health check
[Service]
Type=oneshot
ExecStart=/usr/local/bin/check_tunnels.sh

/etc/systemd/system/vpn-health.timer

[Unit]
Description=Run VPN health check periodically
[Timer]
OnBootSec=10
OnUnitActiveSec=5
Unit=vpn-health.service
[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now vpn-health.timer

Result: per-flow load sharing across both tunnels; if one next-hop fails, the service removes it from the default route and restores ECMP when healthy again.

---

FAQ — Setting Up Multiple VPN Routes for Load Balancing

Is ECMP enough or do I also need PBR?

For simple 50/50 distribution, ECMP is sufficient. Use PBR for granular control (per application, per VLAN, weighted splits), or when you need deterministic steering beyond equal-cost behavior.

Can I mix IPsec, OpenVPN, and WireGuard?

Yes. As long as each tunnel presents a routable interface/next-hop, you can ECMP/PBR across them. Be mindful of differing overheads (MTU) and implement health checks per tunnel.

How do I prevent asymmetric routing?

Prefer per-flow hashing (not per-packet), keep costs/policies symmetric in both directions, and consider dynamic routing so each side chooses the same best path.

How fast can failover be?

With BFD you can achieve sub-second detection (e.g., 150–300 ms). With simple ping scripts, expect 1–5 seconds depending on intervals/timeouts.

What values should I use for MTU/MSS?

It depends on protocol and underlay. A common starting point is tunnel MTU ~1420 and TCP MSS ~1360; then test and adjust with PMTU-aware pings.

---

Conclusion

Setting Up Multiple VPN Routes for Load Balancing is about combining the right routing primitives (ECMP and PBR), consistent MTU/MSS tuning, and robust health-checking—then graduating to dynamic routing (BGP/OSPF with FRR) when scale demands it. Start with two tunnels and ECMP. Add PBR for precision. Introduce BFD and a routing protocol to withdraw bad paths automatically. With proper observability and security hardening, you’ll run a resilient, high-throughput VPN fabric that adapts to change without manual intervention.