Optimizing OpenVPN Performance by Adjusting MTU and Fragmentation

OpenVPN is a robust, open-source VPN used for secure site-to-site and remote-access connectivity. One of the biggest performance levers—often overlooked—is how you handle packet sizing: MTU (Maximum Transmission Unit), TCP MSS (Maximum Segment Size), and fragmentation behavior. Mis-sized packets create stalls, loss, and strange “works for ping but not for web” issues. This guide gives you a technical, step-by-step approach to tune MTU and MSS safely, with copy-paste configs, test commands, and troubleshooting tips for Linux, Windows, and mixed IPv4/IPv6 paths.

Key Concepts: MTU, MSS, Encapsulation Overhead

Rule of thumb: Avoid IP-level fragmentation of the outer VPN packets. Teach the stack to send appropriately sized inner packets instead.

How Big Is the Overhead?

Overhead depends on your mode (TUN vs TAP), transport (UDP vs TCP), and cryptography. The table below shows typical overhead ranges (conservative estimates) for planning. Real values vary; that’s why we test (next sections).

Implication: Even on an Ethernet 1500-byte link, the outer OpenVPN packet may exceed MTU unless you lower inner TCP MSS or (rarely) the tunnel MTU.

Quick Reference: The OpenVPN Knobs

Note: Syntax defaults and behaviors can vary by OpenVPN version/build. The guidance below is conservative and works across current releases.

A Safe, Repeatable Tuning Procedure

Step 0 — Prefer UDP and TUN

• Use proto udp unless you must tunnel over TCP (e.g., corporate proxy, DPI).
• Use TUN (L3) unless you specifically need L2 features (broadcasts, NetBIOS, non-IP).

Step 1 — Enable PMTUD and baseline logging

# On both server and client:
proto udp
tun-mtu 1500
mtu-disc yes
# helpful logging during tuning
verb 4
status /var/log/openvpn-status.log

Let the OS discover path limits. If middleboxes drop ICMP “Fragmentation Needed,” you may need the later steps.

Step 2 — Start with MSS clamping (no guesswork)

Enable mssfix so inner TCP flows don’t force oversize packets. If you omit a size, OpenVPN applies a safe default; but it’s best to compute and set it explicitly after tests.

# Add on both ends:
mssfix 1450  

Why MSS first? Most real traffic is TCP. Clamping MSS prevents inner payloads from creating outer packets that exceed path MTU. It’s minimally invasive and usually enough.

Step 3 — Empirically discover the largest unfragmented packet

Use ICMP “do not fragment” probes to find your ceiling. Remember that ping sizes refer to payload; add header overhead (28 bytes for IPv4, 48 bytes for IPv6) to convert to MTU.

# Start at 1472 (1472 + 28 = 1500) and go down until it stops fragging
ping -M do -s 1472 <peer-or-gateway>
# If frag needed, try smaller:
ping -M do -s 1452 <host>
# Keep binary searching to find the max that works.

# -f = "don't fragment", -l = payload size
ping -f -l 1472 <host>
# Decrease until success, then derive MTU = payload + 28

# Linux
ping -6 -M do -s 1452 <host>  # 1452 + 48 = 1500
# Adjust and find the largest success; MTU = payload + 48

Let’s say your largest unfragmented outer IPv4 payload is 1430 bytes (=> path MTU 1458). You now size your MSS so that inner TCP + inner IP + VPN overhead + outer UDP/IP stays ≤ 1458.

Step 4 — Compute MSS explicitly (simple model)

For IPv4 over UDP + AEAD, a conservative estimate is:

MSS ≈ PathMTU
      - OuterIP(20) - UDP(8)
      - OpenVPN+AEAD (~40–60)
      - InnerIP(20) - TCP(20)

# Example, PathMTU=1458:
MSS ≈ 1458 - 20 - 8 - 50 - 20 - 20 = 1340  (round down to a clean value, e.g., 1330–1340)

Set the result in mssfix on both ends:

mssfix 1340

Tip: If you use TAP or IPv6, increase the overhead budget (IPv6 adds +20 vs IPv4; TAP adds Ethernet). Always leave a few bytes of headroom.

Step 5 — Retest and observe

• Reconnect OpenVPN and test real traffic (web, file copy, scp, RDP/SSH).
• Watch status and logs for retransmits or TLS errors; use iftop / ss -s to check for drops.
• Only if you still see blackholing on some paths, consider mtu-test or the fragment fallback (next).

When (and How) to Use fragment and mtu-test

fragment is a last resort for UDP when intermediate devices break PMTUD (dropping ICMP) and you cannot change them. It makes OpenVPN split its own datagrams before IP sees them, reducing IP-level fragmentation risk at the cost of extra overhead and CPU.

# Use on both ends if absolutely necessary:
fragment 1300
mssfix 1200
# keep mtu-disc enabled; test with/without

mtu-test runs a discovery routine at connect time to negotiate a safe size dynamically; it slows initial handshake but can save guesswork on variable paths:

mtu-test

Prefer good MSS sizing and PMTUD first; reach for fragment only if you must interoperate with broken middleboxes.

Complete Example Configurations

UDP/TUN with MSS clamping (recommended baseline)

# server.conf
port 1194
proto udp
dev tun

# crypto (example; match your policy)
cipher AES-256-GCM
data-ciphers AES-256-GCM:AES-128-GCM
ncp-ciphers AES-256-GCM
tls-version-min 1.2

# MTU / MSS policy
tun-mtu 1500
mtu-disc yes
mssfix 1340         # set from your calculation
;fragment 1300      # only if absolutely necessary
;mtu-test           # optional

# standard server bits...
server 10.8.0.0 255.255.255.0
push "redirect-gateway def1"
push "dhcp-option DNS 10.8.0.1"

keepalive 10 60
persist-key
persist-tun
verb 3
status /var/log/openvpn-status.log

# client.ovpn
client
dev tun
proto udp
remote vpn.example.com 1194

cipher AES-256-GCM
ncp-ciphers AES-256-GCM:AES-128-GCM
tls-version-min 1.2

tun-mtu 1500
mtu-disc yes
mssfix 1340
;fragment 1300
;mtu-test

persist-key
persist-tun
verb 3

Low-MTU WAN (e.g., PPPoE, LTE) scenario

PPPoE often yields an effective MTU ≈ 1492, and mobile networks may be smaller. If your PMTUD shows a path MTU of ~1400–1450, reduce mssfix accordingly (e.g., 1280–1360). If you still see intermittent stalls, consider a modest fragment value (e.g., 1200–1300) while keeping MSS low and logs at verb 4 until stable.

Testing Toolkit

Ping/DF tests (recap)

• Linux IPv4: ping -M do -s <payload> <host> (MTU = payload + 28)
• Windows IPv4: ping -f -l <payload> <host> (MTU = payload + 28)
• IPv6: ping -6 -M do -s <payload> <host> (MTU = payload + 48)

Inside the tunnel

Ping the peer’s tun IPs to ensure inner path is healthy. If inner pings succeed but web is slow, you likely have a TCP MSS/fragmentation mismatch.

Traffic checks

• ss -tuap (Linux) to see retransmits/drops
• iftop / nload to observe throughput
• Browser large downloads and sustained scp transfers for real-world behavior

Troubleshooting Matrix

Do You Ever Lower tun-mtu?

Yes—but sparingly. Lowering the tunnel MTU reduces the maximum size of inner IP packets, which can help on consistently small paths. If all users traverse a known small MTU (e.g., 1400), you can set:

tun-mtu 1400
mssfix 1340   

This pushes all inner hosts to naturally send smaller packets, reducing reliance on MSS clamping. Be sure to test for apps that assume Ethernet-sized frames (some legacy protocols break with small MTUs).

Windows & Mixed-OS Notes

• Windows DF ping: ping -f -l is the easiest way to find payload ceilings.
• NIC offloads: Hardware offload features can mask fragmentation in packet captures. When analyzing, temporarily disable large-send offload to see true sizes.
• Firewall/ICMP: Some enterprise firewalls drop ICMP Fragmentation Needed. If you control the path, allow the PMTUD ICMP types; otherwise expect to rely more on mssfix or fragment.

Security & Reliability Sidebars

• UDP preferred: Avoid TCP-over-TCP (“meltdown”) where retransmission and congestion control at two layers cause collapses under loss.
• AEAD ciphers (e.g., AES-GCM): Lower overhead and better performance than CBC+HMAC on modern CPUs.
• Consistent configuration: Keep MTU/MSS directives symmetric on server and clients to avoid mismatched expectations.

Checklist: From “It’s Slow” to Stable

1. Use UDP/TUN. Keep cipher AEAD where possible.
2. Enable mtu-disc yes and set verb 4 temporarily.
3. Enable mssfix (with no size initially), test, then compute and set an explicit size.
4. Run DF pings (IPv4/IPv6) to find the real path MTU.
5. Compute MSS with headroom (see formula) and set mssfix <size> on both ends.
6. If paths vary or blackhole ICMP, try mtu-test.
7. Only if necessary, add fragment (UDP only) with conservative values (1200–1300) and re-test.
8. Once stable, reduce verb to 3 and keep status logging for visibility.

FAQ

Copy-Paste Summary (Minimal Stable Baseline)

# server + client (UDP/TUN baseline)
proto udp
dev tun
tun-mtu 1500
mtu-disc yes
mssfix 1340         # adjust from DF ping and formula
;fragment 1300      # only as last resort
;mtu-test           # optional if paths vary
cipher AES-256-GCM
tls-version-min 1.2
verb 3              # use 4 while tuning
status /var/log/openvpn-status.log

With a structured approach—PMTUD, explicit MSS sizing, and careful testing—you’ll eliminate IP-fragmentation surprises and unlock the throughput OpenVPN is capable of on your network.