SOLVED: Can’t Connect to Linux Server Behind NAT From Linux/FreeBSD But Can From Windows

Strange networking behavior while trying to access a Linux server via NAT? Does it work from one computer but not another? We may have the solution!

We recently ran across a situation where a Linux server was behind NAT (for VPN purposes), and we could access it just fine from a Windows 10 computer, however it didn’t work from a Linux, FreeBSD, or MacOS computer on the same LAN, gateway and VPN link as the Windows 10 computer.

After ensuring that there was no firewall issues at work, I ran two tcpdumps on the Linux server, the first is while trying to connect from the Windows 10 computer:

As you can see the connection works as there is back and forth. I then tried from a FreeBSD computer (again, same physical LAN as the Windows 10 computer):

You can see the first packet is ignored, and there is a subsequent TCP retry in red.

The only difference I could tell was that FreeBSD was using the TCP timestamp header, where as Windows 10 was not. This initially had me simply turn of TCP timestamps on the Linux server which did fix the problem and allow the FreeBSD (and other computers) to successfully connect. Turning off TCP timestamps however can lead to even harder to chase performance issues and is not recommended. Keep it turned on.

Ultimately I ran across this Server Fault post: Why would a server not send a SYN/ACK packet in response to a SYN packet.

After checking, indeed the Linux server in question had net.ipv4.tcp_tw_recycle set to 1, despite it being behind NAT, per the docs:

tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4)
Enable fast recycling of TIME_WAIT sockets. Enabling this
option is not recommended for devices communicating with the
general Internet or using NAT (Network Address Translation).
Since some NAT gateways pass through IP timestamp values, one
IP can appear to have non-increasing timestamps. See RFC 1323
(PAWS), RFC 6191.

Setting it to 0 immediate fixed the problem and all other operating systems including FreeBSD and Linux could now connect to the server:

sysctl -w net.ipv4.tcp_tw_recycle=0

NOTE: Be sure to check /etc/sysctl.conf as this is likely being set on boot there.

One interesting thing is while you can set this to 0 and have it take immediate effect, setting it back to 1 does not appear to apply the change. A reboot is required to re-enable it (though why would you?).