Java HTTP3 / QUIC implementation: theory

History of HTTP

Since HTTP/1.0, the HTTP protocol was a means for request - response communication. The client sends a request to the server, and the server sends a response. The request contains at least a request method (GET, POST etc.) and requested resource path, and can contain other headers and request body. The response contains at least the status code, and can contain other headers and response body.

HTTP/1.0 used a single TCP connection per request. The client opened a connection, sent the request, read the response and closed the connection. This scheme was inefficient for a few reasons:

• TCP connections start out in a so-called "slow start" state. When they start, the data transfer rates are artificially limited, and the transfer rates increase as more and more data is transferred. Since HTTP/1.0 uses a different TCP connection for each request, all requests observe the slow transfers.
• Using encryption (TLS) makes things even worse. TLS requires large amounts of CPU during connection establishment, and more connections mean more CPU usage.

HTTP/1.1 addressed these points by reusing connections. In HTTP/1.1, the client could send multiple requests over the same connection, and the server would respond with multiple responses over the same connection. This required some changes to the HTTP protocol, specifically, both the client and the server are now required to declare where the message body ends, either by sending a content-length header, a transfer-encoding header, or by using methods that are known to send no content. Other than this, the protocol looks exactly like before.

HTTP/1.1 improved the transfer speeds a lot, but still left some room for improvement:

• the responses had to be sent in the order in which requests were received. So, if the client sent multiple requests and generating a response for the first request took time, the connection could not be used for transferring other data and was idle.
• while multiple requests could be sent at the same time in theory, in practice many servers were unable to handle such pipelined requests due to implementation bugs. In practice the clients often wait for the server response before sending a follow-up request, leaving the connection idle again.

HTTP/2 addressed these limitations by introducing multiplexing. Multiplexing means that it is now possible to send multiple streams over a single TCP connection. Each stream enables data transfer in both directions, and the connection can alternate between different streams at any time. Each request / response exchange is done on its own stream.

HTTP/2 can fully utilize a TCP link. Both the client and the server can send data over any stream at any time, so the connection is only idle when there is nothing to send on any of the active streams.

Full speed of TCP was not sufficient for the authors of HTTP/3. TCP also has some room for improvement:

• before any data can be exchanged, the connection goes through a 3-way handshake, which takes one round-trip time to complete,
• if any TCP packet is lost, data in subsequent packets is not deliverable until the lost packet is retransmitted and received,
• there are many other points where a new protocol could improve upon TCP, I will discuss them later.

Most importantly though, TCP is next to impossible to evolve (it's "ossified"). There were attempts to improve the TCP protocol (see TCP Fast Open for example), but they met with resistance. It turned out that in order to support TCP Fast Open, it is not sufficient to have two endpoints that understand TCP Fast Open. Many devices in the network infrastructure have their own understanding of which TCP packets are correct and which ones are not, and need to be updated to understand TCP Fast Open, otherwise they simply drop these packets, negating any possible performance gains.

Compared to HTTP/2, HTTP/3 offers only cosmetic changes. The big change comes from replacing TCP with QUIC as the underlying protocol.

QUIC protocol

QUIC replaces TCP as the underlying transport for HTTP/3. Similar to TCP, it offers reliable in-order delivery. Unlike TCP, QUIC is always encrypted. Also QUIC supports multiple data streams. HTTP/2 had to implement its own multiplexing, HTTP/3 delegates that to the QUIC layer. One advantage of multiplexing on the QUIC layer is that data loss on one stream does not block data delivery on other streams.

Importantly though, everything in QUIC is encrypted end to end, including packet numbers, acknowledgements, and reset packets. This limits the options for the network devices to interfere with QUIC traffic, or to ossify on a specific QUIC version.

Compared to TCP, I find the following differences interesting:

Path MTU detection (PMTUD)

Both QUIC and TCP prevent packet fragmentation and detect the maximum packet size (maximum transfer unit, MTU) that is supported by the path. Using larger packet sizes improves efficiency, that is, the same payload can be delivered in a smaller number of packets.

With TCP, MTU detection can be performed using one of the following methods:

- The SYN packet includes a Maximum Segment Size extension. This extension can be modified by routers along the way, and the recipient calculates the maximum transfer unit based on the received MSS.

- If a router on path receives a packet larger than it can handle without fragmentation, it drops the packet, and sends an ICMP message back to the sender with information about maximum supported size.

None of these methods is authenticated. When an endpoint receives a MSS or an ICMP packet, it has no way to determine if it is authentic or forged.

There were cases where the ICMP packets were used to trick TCP endpoints to use very small packet sizes, and as a result many implementations ignore or block the ICMP packets. This can sometimes lead to a situation where the TCP stack selects a MTU larger than supported by the network, and then the connection breaks once the connected parties try to send data.

With QUIC, MTU detection can be performed using one of the following methods:

- The handshake is performed using 1200-byte datagrams, and will fail if the network does not support this datagram size

- support for larger datagram sizes is probed. If a given packet size is acknowledged by the peer, that packet size is supported. ICMP packets may be used to drive the selection of the packet sizes to probe, but they cannot be used to reduce the packet size below 1200 bytes.

- an endpoint can advertise the maximum datagram size it is willing to accept. If this number is modified in transit, the connection will fail.

Connection resilience

When running TCP+TLS, a corruption of a single bit is usually enough to terminate the connection. QUIC on the other hand is able to detect and discard a corrupted packet, and continue processing non-corrupted packets. Once the handshake is completed, it is practically impossible for a third party to create a QUIC packet that would cause connection termination.

Connection closure

TCP options to close a connection are limited to:

• closing the sending side of the connection
• resetting the connection

This works well enough in many cases, but doesn't work when one peer needs to send a message and abruptly close a connection where the other peer is actively sending. In that case, the connection will usually be reset, and the final message will be lost.

QUIC separates closing the stream from closing the connection. For closing the stream, it offers the following options:

• closing the sending side of the stream
• resetting the sending side of the stream
• notifying the peer of closing the receiving side of the stream

And for closing the connection:

• closing the connection with an error message
• resetting the connection, used only when the peer is sending over a connection that no longer exists
• timing out after a negotiated period of inactivity

Congestion control

TCP only offers limited information to the congestion controller:

• last acknowledged sequence number is always available
• optionally, the endpoints can negotiate support for selective acknowledgements (SACKs) to acknowledge data received out of order (supported by most implementations). SACKs can be reneged, i.e. an endpoint can request retransmission of data it previously acknowledged.
• optionally, the endpoints can support timestamps (only supported by some implementations) to indicate the order in which packets were transmitted.
• optionally, the endpoints can negotiate support of ECN. This has to be supported by the devices on path, and there used to be bugs that prevented its adoption.

Compared to that, QUIC offers more information:

• QUIC acknowledges packets, not sequence numbers. This way when a packet is retransmitted and later acknowledged, it is clear if the acknowledgement applies to the original packet, to the retransmitted one, or both.
• Packets are always acknowledged, even in the presence of packet loss
• Packets cannot be reneged - once acknowledged, the data may not be discarded
• Acknowledgements contain timing information - it is always clear if an acknowledgement was delayed by the sender and by how much
• ECN support is detected, and ECN information is only used when no bugs are detected.

The QUIC congestion control algorithm defined by RFC 9001 does not match the performance of the CUBIC TCP controller, but some QUIC implementations already offer CUBIC.

Handshake improvements

DoS prevention: when a TCP server deals with a flood of TCP SYN packets, it starts sending SYN cookies. They permit the server to defer allocating state for a connection until the client address is confirmed. However, the SYN cookies lose information about TCP extensions present in the SYN packet, like MSS or TCP window scale.

When a QUIC server deals with a flood of initial packets, it starts sending retry packets. They also permit the server to defer allocating state for a connection until the client address is confirmed. They do not lose any information, but they cost one round trip time.

Timing improvements: TCP + TLS handshake costs at least 1 RTT (1 RTT for TCP, 0 RTT for TLS 1.3); QUIC can send data in the first packet making it true 0-RTT.

MTU validation: QUIC sends 1200-byte datagrams during handshake, validating that the path supports this datagram size.

Path migration

A TCP connection is initiated between 2 given addresses. Changing any of the addresses requires establishing a new connection and, in case of TLS, performing a new handshake.

A QUIC connection is established between 2 given addresses. Changing the client address can be performed any time without affecting connection state, but requires path validation to remove the anti-amplification limit. Changing the server address has the same requirements as with TLS.

Java Http3/QUIC implementation security, part 5: HTTP/3

...continued from part 4

RFC 9204 QPACK: Field Compression for HTTP/3

7.1 Probing Dynamic Table Size

HttpClient only uses the dynamic table for known-safe fields: ":authority" and "user-agent".

Fields "cookie", "authorization" and "proxy-authorization" are flagged with never-indexed bit.

7.2. Static Huffman Encoding

No additional requirements.

7.3 Memory Consumption

HttpClient limits the maximum size of the dynamic table to 4096. Blocked streams are disallowed by default.

The encoder table size is limited to 4KB even if the decoder advertises a larger table size.

The decoder limits the allowed field section size to 384KB. When that size is reached, the processing is aborted.

We currently do not monitor the amount of unsent data on the encoder and the decoder stream.

7.4 Implementation Limits

Integer values that can't be encoded on a Java long are rejected. String literals longer than 2GB are rejected, but only after parsing. This will be improved before the final release.

EDIT 24.04.2025:

Long string literals are rejected without parsing. Maximum acceptable length of a header field is configurable.

Java Http3/QUIC implementation security, part 4: HTTP/3

 ...continued from part 3

RFC 9114 HTTP/3

10.1 Server Authority

HTTP/3 uses QUIC and TLS to verify the server authority. We always set endpoint identification algorithm to HTTPS to ensure that the server certificate identity is authoritative for the URL host name.

10.2 Cross-Protocol Attacks

The underlying TLS implementation ensures that both parties agree on the ALPN.

10.3 Intermediary-Encapsulation Attacks

HttpClient validates incoming field names and values. Responses containing invalid fields are treated as malformed, and are not delivered to the application.

10.4 Cacheability of Pushed Responses

HttpClient does not cache any responses. 

The default PushPromiseHandler rejects push promises where the :authority header does not match the hostname that was used to establish the connection. Custom push promise handlers might choose to implement different checks.

10.5 Denial-of-Service Considerations

The number of PUSH_PROMISE frames is limited to a maximum of 100 concurrently used push IDs at any given time.

The maximum allowable SETTINGS frame size is limited to 1280 bytes, which is more than enough to hold all defined settings.

HttpClient does not monitor the use of unknown frame types and unknown stream types. H3_EXCESSIVE_LOAD error is not generated.

HttpClient limits the maximum size of a field section and the maximum size of a field.

10.6 Use of Compression

HttpClient does not support compression. The Accept-Encoding and Content-Encoding headers are not set by the client. They may be set by the application.

10.7 Padding and Traffic Analysis

No additional requirements.

10.8 Frame Parsing

HttpClient checks the frame lengths.

10.9 Early data

HttpClient does not implement 0-RTT

10.10 Migration

No additional requirements.

10.11 Privacy Considerations

No additional requirements.

continued in part 5...

Java Http3/QUIC implementation security, part 3: QUIC

 ...continued from part 2

RFC 8999, 9368, 9369

The security considerations sections of these documents focus on downgrade prevention. No additional requirements beyond what is already discussed elsewhere in the documents.

RFC 9001 Using TLS to Secure QUIC

9.1 Session Linkability

JSSE TLS implementation does not reuse session tickets. It is also possible to prevent session resumption by using a different SSLContext for every connection.

9.2 Replay Attacks with 0-RTT

0-RTT requires support in HttpClient, QUIC and TLS. None of these is implemented.

9.3 Packet Reflection Attack Mitigation

This section discusses server anti-amplification limit. The requirements do not apply to the client side.

9.4 Header Protection Analysis

No additional requirements

9.5 Header Protection Timing Side Channels

We do not discard packets with duplicate packet number without decrypting them first.

We do not generate packet decryption keys while decrypting.

The packet decryption time might differ between current, previous and next key space. It might need further improvement.

9.6 Key Diversity

No additional requirements

9.7 Randomness

Connection IDs are generated with a secure random number generator.

RFC 9002 QUIC Loss Detection and Congestion Control

8.1 Loss and Congestion Signals

No additional requirements

8.2 Traffic Analysis

No additional requirements

8.3 Misreporting ECN Markings

Our QUIC implementation does not currently support sending or receiving ECN.

This concludes the overview of QUIC RFCs.

continued in part 4...

Java Http3/QUIC implementation security, part 2: QUIC

...continued from part 1

21.5 Request Forgery Attacks

This paragraph focuses on the risk posed by reflected datagrams. The concerns are somewhat similar to these in the anti-amplification section, except that here the focus is on sending datagrams to otherwise inaccessible services, and forging datagrams that would make the inaccessible services react in a specific way.

Most of the concerns listed here apply to the server side; other than using the server-supplied preferred address, the client does not migrate to other addresses. We do not support preferred address at the moment, so that doesn't apply either.

21.6 Slowloris attack

Slowloris aims at making the endpoint keep as many open connections as possible.

HttpClient may keep multiple connections to the same server. The number of open connections to a single server is at most the number of outstanding requests plus one. It's the user's responsibility to limit the number of concurrently executing requests.

21.7 Stream Fragmentation and Reassembly Attacks

QUIC implementation needs to buffer stream data on on the sending side until the data is acknowledged by peer, and on the receiving side until the data is received by the higher layer. If there are gaps in the received stream, the data needs to be buffered until the gaps are filled. This can lead to excessive memory consumption.

On the receiver side, HttpClient limits the MAX_DATA QUIC parameter to a maximum of 15 MB per connection at all times. If certain portions of stream data are received multiple times, only one copy is preserved until the data is received by the application. Buffer memory utilization is therefore bounded.

Memory structures used to store discontinuous ranges of stream data might consume excessive amounts of memory. The maximum memory usage has not been measured.

Crypto stream receive buffer is limited to 64KB per connection.

On the sender side, we buffer as much data as the congestion controller allows. This might lead to memory overcommit if the receiver successfully inflates the congestion window.

EDIT 24.04.2025:

We now detect when the peer sends excessive number of small frames, and close the connection if that happens. The detector is configured to verify that the average fragment size is above a certain threshold when the number of undeliverable fragments gets large.

21.8 Stream Commitment Attack

We limit the number of streams the peer can open at any time to 100 per stream type per connection.

21.9 Peer Denial of Service

This section recommends to "track cost of processing relative to progress and treat [excess] as indicative of an attack".

We do not track the cost of processing.

EDIT 24.04.2025:

see the edit on point 21.7 above

21.10 Explicit Congestion Notification Attacks

No additional requirements.

HttpClient's QUIC implementation does not support sending or receiving ECN yet.

21.11 Stateless Reset Oracle

Every QUIC endpoint uses a different randomly generated key for generating stateless reset tokens. The keys are never shared, so a stateless reset is only generated if a connection ID is not in use.

EDIT 24.04.2025:

Recipe for a stateless reset oracle:

The client can open multiple endpoints. All endpoints use the same key to generate stateless reset token, but each endpoint keeps its own list of active connection IDs.

Our client keeps a different key on every associated endpoint.

21.12 Version Downgrade

The current implementation only supports QUIC v1 and v2. These versions offer identical security properties, so version downgrade is not a concern.

21.13 Targeted Attacks by Routing

This section describes deployment concerns, as opposed to implementation concerns. No additional implementation requirements.

21.14 Traffic Analysis

Currently our QUIC API offers no way to obscure the length of the packet content.

This concludes the review of RFC 9000 security considerations.

continued in part 3...

Java Http3/QUIC implementation security, part 1: QUIC

Support HTTP/3 in the HttpClient

As part of the JEP, we implement:

• RFC 9114: HTTP/3
• RFC 9204: QPACK: Field Compression for HTTP/3
• RFC 8999: Version-Independent Properties of QUIC
• RFC 9000: A UDP-Based Multiplexed and Secure Transport
• RFC 9001: Using TLS to Secure QUIC
• RFC 9002: QUIC Loss Detection and Congestion Control
• RFC 9368: Compatible Version Negotiation for QUIC
• RFC 9369: QUIC Version 2

QUIC is implemented on top of TLS 1.3, defined in RFC 8446. TLS 1.3 support in JSSE was implemented in a prior JEP, and this JEP builds on top of that work.

The goal of the JEP is to implement a working implementation of HTTP/3 in the HttpClient. The QUIC implementation is supposed to be in a reasonably usable state; in particular, optional features and features that are only required by the server might not exist.

QUIC security considerations

This section structure mirrors the formal requirements specified in RFC 9000

21.1.1.1 Anti-Amplification

The QUIC implementation is supposed to limit the number of bytes it sends to an unvalidated address.

- The client only sends stateless reset messages to unvalidated addresses. We make sure that the stateless reset messages are only sent when they are strictly smaller than the incoming datagram.

- The server also sends handshake messages to unvalidated clients. Our server-side implementation does not have anti-amplification limit.

21.1.1.2 Server-Side DoS

In order to filter out forged handshake packets, the server can implement secure token generation, either in a retry packet, or in a new_token frame.

- Our client implementation of new_token and retry is complete

- The tokens generated by our server are not secure and easily forged, offering no protection against DoS.

21.1.1.3 On-Path Handshake Termination

We offer no extra protection against forged initial/retry packets.

21.1.1.4 Parameter Negotiation

No additional requirements

21.1.2 Protected packets

No additional requirements

21.1.3 Connection Migration

The server can offer a preferred address, and the client can choose to migrate to the preferred address or stay on the original one.

The client can switch addresses as a result of a (local) network change or as a result of a (remote) NAT rebinding.

In order to tell apart a real and a spoofed address migration, the QUIC endpoints are supposed to implement path validation. Until the path validation succeeds, the new address is subject to anti-amplification limit.

Our implementation:

- does not perform path validation. The handling of connection migration needs to be reevaluated.

- always sends packets to the same remote address. This is good enough on the client side, but not good enough on the server side.

- selects source address individually for each packet. The client source address might change in the middle of a connection if the routing tables change. This would be reasonable if we implemented path validation.

- optionally filters the source address on the incoming packets. This is good on the client side, but might be counterproductive on the server side.

EDIT 24.04.2025:

If we accidentally migrate to a different address, the server will send a PATH_CHALLENGE frame. We respond to that with a PATH_RESPONSE. This should enable the server to migrate to the new path.

We do not switch connection IDs when that happens, but we send connection IDs for the server to use.

21.2 Handshake Denial of Service

No additional requirements

21.3 Amplification Attack

Server guidance only. Our server implementation does not offer any guarantees for token validity.

21.4 Optimistic ACK Attack

(optional) We are vulnerable to optimistic ACK attack. We do not detect acknowledgements of non-existent packet numbers other than packet numbers that were not assigned yet.

continued in part 2...

Running cross-translation-unit static analysis on OpenJDK

Scan-build discusses in an earlier post is pretty effective at detecting issues within a single C file. Additional insights can be gained by applying cross-translation-unit analysis. This post will discuss how to install and use CodeChecker.

Prerequisites

Running build of OpenJDK. Clang tools v10 installed as discussed in the previous post.

Set up clang

By default clang v10 is only available as "clang-10" and not as "clang". This can be corrected using update-alternatives script:

sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-10   81 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-10 

sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-10   81

Install pip

Required to install CodeChecker

sudo apt install python3-pip

Install CodeChecker

pip3 install codechecker

Generate compilation database

CodeChecker log --build "make" --output ./compile_commands_ori.json

sed "s/-fno-lifetime-dse //" compile_commands_ori.json >compile_commands.json 

Run analysis

CodeChecker analyze --ctu compile_commands.json -o reports

Limiting the number of worker processes (with -j2) may be necessary; some processes consumed 6G of memory.

Results

None so far; the process takes ages to complete. Will try again on a more powerful machine.

Sources

https://askubuntu.com/a/1187858

https://github.com/Ericsson/codechecker

https://codechecker.readthedocs.io/en/latest/usage/ 

Running clang's scan-build on OpenJDK (Ubuntu 18.04)

Prerequisites

We will start with a working OpenJDK build, i.e. a situation where running

bash configure

followed by

make images

produces a working build.

Install clang tools

Latest version available at the time of writing was 10:

sudo apt install clang-tools-10

Reconfigure

Run configure via scan-build

scan-build-10 bash configure

Optionally add --enable-debug, it can reduce the number of false positives.

Scan

scan-build-10 -o scan make

Found problems are reported on console in text format and stored in a subdirectory of "scan" directory in HTML format.

Sample results

You can check example output of the scan here

Source

https://clang-analyzer.llvm.org/scan-build.html

Building latest OpenJDK on Windows (Dec 2021)

Prerequisites:

Windows 64 bit. I'm using Windows 10, but anything Vista+ should work.

Install Cygwin

Download setup script https://cygwin.com/setup-x86_64.exe

Run the script with additional packages selected:

setup-x86_64.exe -P git -P diffutils -P binutils -P make -P m4 -P cpio -P gawk -P file -P zip -P unzip -P procps-ng -P autoconf -P automake -P ssh -P wget

Install Visual Studio 2019

Community version is free for OpenSource development, and is sufficient to build OpenJDK.

Download installer here:

https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes

Install C++ development tools. Default installation location worked fine for me.

Clone the repository

Use Cygwin's git. Repository can be found here:

https://github.com/openjdk/jdk

 

The remaining steps are the same between Windows and Linux:

Download boot JDK

Binaries can be found here:

http://jdk.java.net/

Download and unpack the latest JDK. Save the full (cygwin) path, you will need it later.

Download JTREG (optional)

Required for running tests, not needed for building.

Binaries can be found here:

https://ci.adoptopenjdk.net/view/Dependencies/job/dependency_pipeline/lastSuccessfulBuild/artifact/jtreg/

Download and unpack latest version (jtregtip.tar.gz). Save the full (cygwin) path.

Download GoogleTest (optional)

Required for running a subset of tests, not needed for building.

git clone -b release-1.8.1 https://github.com/google/googletest

Save the full (cygwin) path.

Download JMH (optional)

Required for running microbenchmarks, not needed for building or regression testing.

Starting from the directory where you cloned the JDK, run in Cygwin shell:

sh make/devkit/createJMHBundle.sh

After this JMH will be available under build/jmh/jars

Build

From the cloned JDK directory, run in cygwin shell:

bash configure --with-boot-jdk=/path/to/boot/jdk --with-jtreg=/path/to/jtreg --with-gtest=/path/to/googletest --with-jmh=build/jmh/jars

If configure succeeds, run make to build the JDK.

 

Later after changing branches or updating the code, make may ask you to run configure again. In this case usually the following is sufficient:

make reconfigure clean images

Sources / additional information:

https://stuefe.de/posts/build-openjdk-on-windows/

https://github.com/openjdk/jdk/blob/master/doc/building.md

https://github.com/openjdk/jdk/blob/master/doc/testing.md

Creating a test certificate chain with openSSL

To create a certificate chain (starting from root CA with one intermediate CA) for use in a Java SSL server, create a batch file with the following:

rem create root ca

openssl req -x509 -newkey rsa:2048 -keyout root.key -out root.crt -days 36500 -subj "/C=US/ST=CA/O=Root CA, Inc." -nodes

rem create intermediate cert

openssl genrsa -out ca.key 2048

openssl req -new -sha256 -nodes -key ca.key -subj "/C=US/ST=CA/O=Intermediate CA, Inc./CN=my.ca.com" -out ca.csr

openssl x509 -req -in ca.csr -CA root.crt -CAkey root.key -CAcreateserial -out ca.crt -days 500 -sha256 -extfile caext.txt

rem create server cert

openssl genrsa -out server.key 2048

openssl req -new -sha256 -nodes -key server.key -subj "/C=US/ST=CA/O=Daniel's, Inc./CN=my.server.com" -out server.csr -addext "basicConstraints=CA:false"

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256

rem create keystore

openssl pkcs12 -export -in server.crt -inkey server.key -out keystore.p12 -certfile ca.crt -name server -password pass:changeit

rem convert to jks if required

keytool -importkeystore -srcstorepass changeit -deststorepass changeit -destkeystore KeyStore.jks -srckeystore keystore.p12 -srcstoretype PKCS12 -alias server

Create a text file caext.txt with the following content:

basicConstraints=critical, CA:true, pathlen:0

If you want to connect a Java client to that server, you also need a trust store. The following line will create one:

keytool -import -noprompt -alias 1 -keystore rootca.jks -file root.crt -storepass changeit

The extensions file is necessary; without it the intermediate CA certificate will be v1, and Java works only with v3 CA certificates.

TCP war story 3: excessive packet reordering

So there is this user who says that suddenly our server started responding very slow. Earlier a page would load in a second, now it takes 4-5 minutes to load, he says. We run some checks, and the server is snappy as always. So we ask him for a Fiddler capture. Indeed, the loading times are high. We replay the same requests, and they are super fast here. We tell the user to go to network support. Network support says ping is good, packet loss is zero, no other users are complaining, must be a problem with the application.

How can there be a problem with the network if ping is good and there's no packet loss?

Well apparently there can.

We asked the user for a packet capture. This wasn't his first encounter with the network support, so he knew how to use Wireshark. Good for us. So we open the pcap file and immediately notice a lot of black color. Every second or so a packet arrives 5 milliseconds late, and other packets arrive before it. TCP stack reacts correctly by sending duplicate ACKs, and once the late packet arrives, there's one cumulative ack for everything. But that's too late, 3+ duplicate acks were sent. Once the server receives these, it stops slow start mode and implements congestion control, which is CTCP, so send rate is halved.
To make things worse, MSS is only 587 bytes, and RTT is on the order of 220 milliseconds. Resulting transfer rate is about 20 KB/s, far below 1Gbit/s that is normally available.

How do you convince the network support team that it's a problem that they need to fix? Well, I'm still trying to figure it out.

TCP war story 2: overzealous SYN defense

Further experimenting with the load balanced system we found that while a few simultaneous connections get multi-megabyte throughput, running 10 or more connections at the same time resulted in some of the connections being very slow.

Running tcpdump revealed that the slow connections received SYN cookies; the SYN/ACK packet did not contain window scaling options, and receive window was limited at 64KB, again limiting the throughput at 640KB/s.

The TTL on the SYN/ACK packet was different from the TTL on all other packets on the connection; this allowed us to determine that SYN/ACK did not come from the server, but was sent by a firewall along the way.

The firewall was a Checkpoint device configured with very eager SYN defense settings. After adjusting these settings, the problem was eliminated and we were finally able to enjoy fast transfers on all connections.

TCP war story 1: F5 BIGIP load balancer

Recently I run into trouble using F5 load balancer; it was configured with standard TCP profile, and provided great performance within a data center. However, transfers crossing WAN boundaries had their throughput severely limited.

I don't normally deal with network devices, so it was a surprise for me when I found that the load balancer with TCP profile is in fact a proxy. The TCP profile limits send buffer size to 64 KB, and therefore limits throughput to 64KB / RTT, in our case 640KB/s.

After raising the send buffer size to 1MB we were able to get 10MB/s transfers, which were acceptable for our uses. Fortunately the F5 device had sufficient memory to support that buffer size.

Only later I found that F5 also supports fastL4 mode, in which case it does not act as a proxy, but rather as a regular router. In that mode the send buffer is controlled by the server directly. This reduces the memory requirements, allowing F5 to serve more connections, and shifts the responsibility for throughput to the application.

[MSVC] Linking to DLLs when no .lib is available

Reference:
https://stackoverflow.com/a/16127548/7707617

Step 1) Generate exports file
>dumpbin /exports libcurl.dll > libcurl.exports

Step 2) Edit exports file to leave just the word EXPORTS in the first line and function names in the following lines. The result is a .def file.

Step 3) Create lib from def file:
>lib /def:libcurl.def /out:libcurl.lib

Step 4) Pass the resulting lib file to linker as usual.

Libcurl and slow uploads on Windows

Recently I started using libCurl 7.60 to upload files from Windows machines to Amazon cloud using HTTP POST. The uploads don't perform well. They are limited by the send buffer used by Windows.
On Windows 2008R2 (and probably on more recent versions as well) the system only puts on the wire the bytes it has buffered, and this is limited by SO_SNDBUF (default 8KB) or by the send buffer used by application code (CURL_MAX_WRITE_SIZE, 16KB), whichever is higher. In case of CURL uploads, the system buffers 16KB, then waits for acknowledgement from the other end before buffering and sending the next batch. This is readily visible in Wireshark traces.

Starting with Windows 7 / 2008R2, Windows implements send buffer autotuning. This is well described here.

Theoretically on these systems the send buffer should be automatically adjusted to optimize throughput. I run a couple experiments to confirm that, and found that the send buffer is only adjusted if the socket is blocking and application buffer size is reasonably large (in my experiment 16KB was too small, but 20KB was sufficient), and buffer stays at 8192 if the socket is nonblocking regardless of the application buffer size.

Curl is using nonblocking sockets, and switching to blocking would break existing functionality. In order to use the optimal buffer size, it would need to periodically update SO_SNDBUF to a value provided by SIO_IDEAL_SEND_BACKLOG_QUERY or SIO_IDEAL_SEND_BACKLOG_CHANGE.

I sent a proof-of-concept patch to curl-library mailing list. With some luck the patch will find its way to the next curl release.

RDP session hijacking (without password / as administrator)

1. Get session ID of the session you want to connect to
2. Get PsExec
3. Run CMD under System account from admin CMD: psexec -i -s -d cmd
4. Run tscon.exe <sessionID>

 Sources:

1.  PsExec
2. Getting a CMD prompt as SYSTEM in Windows Vista and Windows Server 2008 
3. RDP hijacking — how to hijack RDS... 

Bulk loading data to SQL server

Source: tab-separated file
Destination: table with the same structure as the file
Command:
bcp <table> in <file> -c -S <server> -U <user>

Other options:
-t : column separator (tab is the default)
-r : row separator (default: newline)
-f : format file - can be used to import files that do not match the list of columns in DB table

TCP gotchas

When I first learned about TCP, I found a few things surprising. Here's a short list:

Server socket has an "accept" but not a "reject" method

Connection set up is handled by the operating system (OS). The application has no way of examining the client before establishing the connection. If you want to ban connections from a certain IP, your can only use a firewall or close the connection immediately after accept.

Blocking "send" does not block until the data is delivered to the other end

Send operation just copies the data to OS buffer for transmission. If the OS has sufficient free buffer space, send operation returns immediately. Send only blocks when OS buffer is full.

"ACK" packet does not mean that the application on the other end successfully received the message

ACKs are sent by the receiving operating system when it stores the message in its internal buffer. The receiving application may read the data at a later point, or not at all.

TCP does not guarantee that a broken connection will raise an error

If you send some data and then close a connection, both operations may succeed even if no data is actually delivered to the other end, for example because the machine on the other end lost its network connection. And conversely, if the sending machine crashes, the receiving machine may never notice.

As far as reliability is concerned, TCP guarantees only the following:

• No data will be delivered out of order
• Everything you send will be delivered at most once
• If the OS can tell that an operation cannot succeed, it will return an error (like when you send to a connection that is already known to be broken)

Any further guarantees are the responsibility of the application.

Range of Random.nextGaussian, continued

In the first part I described a theoretical range of values returned by nextGaussian; in this part I will describe the way used to find out the actual minimum and maximum values.

The simplest way would require using brute force to check all 2⁴⁸ possible seed values. This would take a few years on my computer, so I had to do better than that.

As explained in the first part, in order to maximize the values of nextGaussian, the values of v1 and v2 need to be as close to zero as possible. For that, the value returned by nextDouble needs to be close to 0.5. So, how do we make the random number generator return the values we want?

NextDouble calls next two times, first to get top 26 significant bits of the result, then again to get next 27 bits. If we can get the top bits to be 1 << 25, the resulting double will be very close to 0.5.

A call to next first updates the seed value used by Random, then returns top N bits of the new seed. So, we know the desired seed value after the call to next. In order to find out the value for seed before the call, we can use the simple reversing algorithm found here:

private static final long multiplier = 0x5DEECE66DL;
private static final long addend = 0xBL;

private static long reverseSeed (long seed) {
    // reverse the addend from the seed
    seed -= addend;
    long result = 0;
    // iterate through the seeds bits
    for (int i = 0; i < 48; i++) {
        long mask = 1L << i;
        // find the next bit
        long bit = seed & mask;
        // add it to the result
        result |= bit;
        if (bit == mask) {
            // if the bit was 1, subtract its effects from the seed
            seed -= multiplier << i;
        }
    }
    return result;
} 

Then, we also need to counter the initial scrambling that happens in setSeed operation:

private static long unscramble(long seed) {
    return seed ^ multiplier;
}

 And we're ready to start hacking:

long seedRange = 0x2000000L;
Random rand = new Random();
double minSoFar = 0, maxSoFar = 0;
for(long seed = -seedRange;seed<seedRange;seed++) {
    rand.setSeed(unscramble(reverseSeed(0x800000000000L + seed)));
    double d = rand.nextGaussian();
    if(minSoFar > d) {
        minSoFar = d;
        System.out.println("Seed: "+seed+ " min: "+d);
    }
    if(maxSoFar < d) {
        maxSoFar = d;
        System.out.println("Seed: "+seed+ " max: "+d);
    }

    d = rand.nextGaussian();
    if(minSoFar > d) {
        minSoFar = d;
        System.out.println("Seed: "+seed+ " second min: "+d);
    }
    if(maxSoFar < d) {
        maxSoFar = d;
        System.out.println("Seed: "+seed+ " second max: "+d);
    }
}

The lowest value found was: -7.844680087923773 (on second call to nextGaussian with seed = 994892)
The highest value found was: 7.995084298635286 (on first call to nextGaussian with seed = 14005843)
These are the real maximum & minimum values returned by Oracle Java 8 implementation of nextGaussian.

Range of Random.nextGaussian()

Recently I had a look at the source code of a language detection library. The library internally uses Random.nextGaussian to determine its behavior, but uses it in such a way that any value lower than -10 would result in incorrect calculations. I was curious if getting such a value is even possible.


The implementation of nextGaussian in Java 8 is as follows:

 private double nextNextGaussian;
 private boolean haveNextNextGaussian = false;

 public double nextGaussian() {
   if (haveNextNextGaussian) {
     haveNextNextGaussian = false;
     return nextNextGaussian;
   } else {
     double v1, v2, s;
     do {
       v1 = 2 * nextDouble() - 1;   // between -1.0 and 1.0
       v2 = 2 * nextDouble() - 1;   // between -1.0 and 1.0
       s = v1 * v1 + v2 * v2;
     } while (s >= 1 || s == 0);
     double multiplier = StrictMath.sqrt(-2 * StrictMath.log(s)/s);
     nextNextGaussian = v2 * multiplier;
     haveNextNextGaussian = true;
     return v1 * multiplier;
   }
 }

The function used to generate next values has extremes when both v1 and v2 are as close to zero as possible, without being both equal to zero.

Quick check of nextDouble indicates that the function generates only multiples of 1 / (double)(1L << 53). This means that nextGaussian will return extreme values for v1 = +/- 2 / (double)(1L << 53)and v2 = 0.

For these values nextGaussian would return +/- 12.00727336061225.

That does not mean that it is possible to get these values. There's a limited number of values that can be returned by nextDouble. But it does mean that nextGaussian (in its Oracle Java 8 implementation) will never return a value outside of the range between -12.00727336061225 and 12.00727336061225.