Further experimenting with the load balanced system we found that while a few simultaneous connections get multi-megabyte throughput, running 10 or more connections at the same time resulted in some of the connections being very slow.
Running tcpdump revealed that the slow connections received SYN cookies; the SYN/ACK packet did not contain window scaling options, and receive window was limited at 64KB, again limiting the throughput at 640KB/s.
The TTL on the SYN/ACK packet was different from the TTL on all other packets on the connection; this allowed us to determine that SYN/ACK did not come from the server, but was sent by a firewall along the way.
The firewall was a Checkpoint device configured with very eager SYN defense settings. After adjusting these settings, the problem was eliminated and we were finally able to enjoy fast transfers on all connections.
No comments:
Post a Comment