pfSense v2.1.5 PMTUD issue with IPv4 and NAT

ICMP too big packets are being filtered

Solution: Add the sysctl 'net.inet.icmp.reply_from_interface' with the value '1' to the pfSense "System: Advanced: System Tunables" page, or from the command line:

sysctl net.inet.icmp.reply_from_interface=1

Description

IPv4 path MTU discovery is broken when traversing the pfSense router. The PPPoE connection to the ADSL modem has an MTU of 1492 which is less than the ethernet MTU of 1500. IPv6 support through a tunnel did not exhibit the same issue.

The ICMP responses can be observed on the LAN interface (re0) with the following command from the pfSense appliance:

tcpdump -nn -i re0 -e 'icmp[icmptype] == 3 && icmp[icmpcode] == 4'

Before the 'fix' the responses came from 127.0.0.1. After the 'fix' the ICMP responses came from the LAN interface address.

Environment:

pfSense 2.1.5
PCEngines APU
Draytek Vigor
two physical interfaces

LAN 1500MTU (re0)
WAN 1500MTU (re2)

two other interfaces

IPv4 only PPPoE over the WAN interface

MTU 1492
NAT

IPv6 GIF tunnel

MTU 1280

Appendices

Failing ICMP

10:48:52.959769 00:0d:b9:33:06:f8 > 00:30:48:bf:eb:b9, ethertype IPv4 
      (0x0800), length 70: (tos 0x0, ttl 63, id 20860, offset 0, flags [DF], 
      proto ICMP (1), length 56, bad cksum 0 (->5c31)!)
    127.0.0.1 > 10.20.5.3: ICMP 202.37.100.24 unreachable - need to frag (mtu 1492), length 36
        (tos 0x0, ttl 6, id 0, offset 0, flags [DF], proto UDP (17), length 1500, bad cksum 30bd (->31bd)!)
    10.20.5.3.43079 > 202.37.100.24.33354: UDP, length 1472

Working ICMP

10:55:10.241704 00:0d:b9:33:06:f8 > 00:30:48:bf:eb:b9, ethertype IPv4
       (0x0800), length 70: (tos 0x0, ttl 63, id 14407, offset 0, flags [DF], 
       proto ICMP (1), length 56, bad cksum 0 (->ea52)!)
    10.20.0.1 > 10.20.5.3: ICMP 74.125.237.119 unreachable - need to frag (mtu 1492), length 36
        (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 1500, bad cksum 2c06 (->2d06)!)
    10.20.5.3.38119 > 74.125.237.119.33337: UDP, length 1472

Failing tracepath

# tracepath www.snap.net.nz/33333
 1?: [LOCALHOST]     pmtu 9000
 1:  gbe-5.orange.lucidsolutions.co.nz (10.20.5.1)          0.846ms
 1:  gbe-5.orange.lucidsolutions.co.nz (10.20.5.1)          0.643ms
 2:  gbe-5.orange.lucidsolutions.co.nz (10.20.5.1)          0.545ms pmtu 1500
 2:  e0.marathon.lucidsolutions.co.nz (10.20.0.1)           1.026ms
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply

Working tracepath

# tracepath www.snap.net.nz/33333
 1?: [LOCALHOST]     pmtu 9000
 1:  gbe-5.orange.lucidsolutions.co.nz (10.20.5.1)          1.049ms
 1:  gbe-5.orange.lucidsolutions.co.nz (10.20.5.1)          0.855ms
 2:  gbe-5.orange.lucidsolutions.co.nz (10.20.5.1)          0.784ms pmtu 1500
 2:  e0.marathon.lucidsolutions.co.nz (10.20.0.1)           1.187ms
 3:  e0.marathon.lucidsolutions.co.nz (10.20.0.1)           1.160ms pmtu 1492
 3:  111-69-1-254.core.snap.net.nz (111.69.1.254)          19.094ms
 4:  32.54.69.111.static.snap.net.nz (111.69.54.32)        18.725ms
 5:  30.100.37.202.static.snap.net.nz (202.37.100.30)      17.761ms

Error

The following error is generated by mpd if the mtu is set to 1500:

mpd_wan.conf:29: Error in 'set link mtu 1500': max MTU on type "pppoe" links is 1492

Sections

Personal tools