Further experimenting with the load balanced system we found that while a few simultaneous connections get multi-megabyte throughput, running 10 or more connections at the same time resulted in some of the connections being very slow.
Running tcpdump revealed that the slow connections received SYN cookies; the SYN/ACK packet did not contain window scaling options, and receive window was limited at 64KB, again limiting the throughput at 640KB/s.
The TTL on the SYN/ACK packet was different from the TTL on all other packets on the connection; this allowed us to determine that SYN/ACK did not come from the server, but was sent by a firewall along the way.
The firewall was a Checkpoint device configured with very eager SYN defense settings. After adjusting these settings, the problem was eliminated and we were finally able to enjoy fast transfers on all connections.
Thursday, September 20, 2018
TCP war story 1: F5 BIGIP load balancer
Recently I run into trouble using F5 load balancer; it was configured with standard TCP profile, and provided great performance within a data center. However, transfers crossing WAN boundaries had their throughput severely limited.
I don't normally deal with network devices, so it was a surprise for me when I found that the load balancer with TCP profile is in fact a proxy. The TCP profile limits send buffer size to 64 KB, and therefore limits throughput to 64KB / RTT, in our case 640KB/s.
After raising the send buffer size to 1MB we were able to get 10MB/s transfers, which were acceptable for our uses. Fortunately the F5 device had sufficient memory to support that buffer size.
Only later I found that F5 also supports fastL4 mode, in which case it does not act as a proxy, but rather as a regular router. In that mode the send buffer is controlled by the server directly. This reduces the memory requirements, allowing F5 to serve more connections, and shifts the responsibility for throughput to the application.
I don't normally deal with network devices, so it was a surprise for me when I found that the load balancer with TCP profile is in fact a proxy. The TCP profile limits send buffer size to 64 KB, and therefore limits throughput to 64KB / RTT, in our case 640KB/s.
After raising the send buffer size to 1MB we were able to get 10MB/s transfers, which were acceptable for our uses. Fortunately the F5 device had sufficient memory to support that buffer size.
Only later I found that F5 also supports fastL4 mode, in which case it does not act as a proxy, but rather as a regular router. In that mode the send buffer is controlled by the server directly. This reduces the memory requirements, allowing F5 to serve more connections, and shifts the responsibility for throughput to the application.
Thursday, September 13, 2018
[MSVC] Linking to DLLs when no .lib is available
Reference:
https://stackoverflow.com/a/16127548/7707617
Step 1) Generate exports file
>dumpbin /exports libcurl.dll > libcurl.exports
Step 2) Edit exports file to leave just the word EXPORTS in the first line and function names in the following lines. The result is a .def file.
Step 3) Create lib from def file:
>lib /def:libcurl.def /out:libcurl.lib
Step 4) Pass the resulting lib file to linker as usual.
https://stackoverflow.com/a/16127548/7707617
Step 1) Generate exports file
>dumpbin /exports libcurl.dll > libcurl.exports
Step 2) Edit exports file to leave just the word EXPORTS in the first line and function names in the following lines. The result is a .def file.
Step 3) Create lib from def file:
>lib /def:libcurl.def /out:libcurl.lib
Step 4) Pass the resulting lib file to linker as usual.
Subscribe to:
Posts (Atom)