Running cross-translation-unit static analysis on OpenJDK

Scan-build discusses in an earlier post is pretty effective at detecting issues within a single C file. Additional insights can be gained by applying cross-translation-unit analysis. This post will discuss how to install and use CodeChecker.

Prerequisites

Running build of OpenJDK. Clang tools v10 installed as discussed in the previous post.

Set up clang

By default clang v10 is only available as "clang-10" and not as "clang". This can be corrected using update-alternatives script:

sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-10   81 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-10 

sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-10   81

Install pip

Required to install CodeChecker

sudo apt install python3-pip

Install CodeChecker

pip3 install codechecker

Generate compilation database

CodeChecker log --build "make" --output ./compile_commands_ori.json

sed "s/-fno-lifetime-dse //" compile_commands_ori.json >compile_commands.json 

Run analysis

CodeChecker analyze --ctu compile_commands.json -o reports

Limiting the number of worker processes (with -j2) may be necessary; some processes consumed 6G of memory.

Results

None so far; the process takes ages to complete. Will try again on a more powerful machine.

Sources

https://askubuntu.com/a/1187858

https://github.com/Ericsson/codechecker

https://codechecker.readthedocs.io/en/latest/usage/ 

Running clang's scan-build on OpenJDK (Ubuntu 18.04)

Prerequisites

We will start with a working OpenJDK build, i.e. a situation where running

bash configure

followed by

make images

produces a working build.

Install clang tools

Latest version available at the time of writing was 10:

sudo apt install clang-tools-10

Reconfigure

Run configure via scan-build

scan-build-10 bash configure

Optionally add --enable-debug, it can reduce the number of false positives.

Scan

scan-build-10 -o scan make

Found problems are reported on console in text format and stored in a subdirectory of "scan" directory in HTML format.

Sample results

You can check example output of the scan here

Source

https://clang-analyzer.llvm.org/scan-build.html

Building latest OpenJDK on Windows (Dec 2021)

Prerequisites:

Windows 64 bit. I'm using Windows 10, but anything Vista+ should work.

Install Cygwin

Download setup script https://cygwin.com/setup-x86_64.exe

Run the script with additional packages selected:

setup-x86_64.exe -P git -P diffutils -P binutils -P make -P m4 -P cpio -P gawk -P file -P zip -P unzip -P procps-ng -P autoconf -P automake -P ssh -P wget

Install Visual Studio 2019

Community version is free for OpenSource development, and is sufficient to build OpenJDK.

Download installer here:

https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes

Install C++ development tools. Default installation location worked fine for me.

Clone the repository

Use Cygwin's git. Repository can be found here:

https://github.com/openjdk/jdk

 

The remaining steps are the same between Windows and Linux:

Download boot JDK

Binaries can be found here:

http://jdk.java.net/

Download and unpack the latest JDK. Save the full (cygwin) path, you will need it later.

Download JTREG (optional)

Required for running tests, not needed for building.

Binaries can be found here:

https://ci.adoptopenjdk.net/view/Dependencies/job/dependency_pipeline/lastSuccessfulBuild/artifact/jtreg/

Download and unpack latest version (jtregtip.tar.gz). Save the full (cygwin) path.

Download GoogleTest (optional)

Required for running a subset of tests, not needed for building.

git clone -b release-1.8.1 https://github.com/google/googletest

Save the full (cygwin) path.

Download JMH (optional)

Required for running microbenchmarks, not needed for building or regression testing.

Starting from the directory where you cloned the JDK, run in Cygwin shell:

sh make/devkit/createJMHBundle.sh

After this JMH will be available under build/jmh/jars

Build

From the cloned JDK directory, run in cygwin shell:

bash configure --with-boot-jdk=/path/to/boot/jdk --with-jtreg=/path/to/jtreg --with-gtest=/path/to/googletest --with-jmh=build/jmh/jars

If configure succeeds, run make to build the JDK.

 

Later after changing branches or updating the code, make may ask you to run configure again. In this case usually the following is sufficient:

make reconfigure clean images

Sources / additional information:

https://stuefe.de/posts/build-openjdk-on-windows/

https://github.com/openjdk/jdk/blob/master/doc/building.md

https://github.com/openjdk/jdk/blob/master/doc/testing.md

Creating a test certificate chain with openSSL

To create a certificate chain (starting from root CA with one intermediate CA) for use in a Java SSL server, create a batch file with the following:

rem create root ca

openssl req -x509 -newkey rsa:2048 -keyout root.key -out root.crt -days 36500 -subj "/C=US/ST=CA/O=Root CA, Inc." -nodes

rem create intermediate cert

openssl genrsa -out ca.key 2048

openssl req -new -sha256 -nodes -key ca.key -subj "/C=US/ST=CA/O=Intermediate CA, Inc./CN=my.ca.com" -out ca.csr

openssl x509 -req -in ca.csr -CA root.crt -CAkey root.key -CAcreateserial -out ca.crt -days 500 -sha256 -extfile caext.txt

rem create server cert

openssl genrsa -out server.key 2048

openssl req -new -sha256 -nodes -key server.key -subj "/C=US/ST=CA/O=Daniel's, Inc./CN=my.server.com" -out server.csr -addext "basicConstraints=CA:false"

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256

rem create keystore

openssl pkcs12 -export -in server.crt -inkey server.key -out keystore.p12 -certfile ca.crt -name server -password pass:changeit

rem convert to jks if required

keytool -importkeystore -srcstorepass changeit -deststorepass changeit -destkeystore KeyStore.jks -srckeystore keystore.p12 -srcstoretype PKCS12 -alias server

Create a text file caext.txt with the following content:

basicConstraints=critical, CA:true, pathlen:0

If you want to connect a Java client to that server, you also need a trust store. The following line will create one:

keytool -import -noprompt -alias 1 -keystore rootca.jks -file root.crt -storepass changeit

The extensions file is necessary; without it the intermediate CA certificate will be v1, and Java works only with v3 CA certificates.

TCP war story 3: excessive packet reordering

So there is this user who says that suddenly our server started responding very slow. Earlier a page would load in a second, now it takes 4-5 minutes to load, he says. We run some checks, and the server is snappy as always. So we ask him for a Fiddler capture. Indeed, the loading times are high. We replay the same requests, and they are super fast here. We tell the user to go to network support. Network support says ping is good, packet loss is zero, no other users are complaining, must be a problem with the application.

How can there be a problem with the network if ping is good and there's no packet loss?

Well apparently there can.

We asked the user for a packet capture. This wasn't his first encounter with the network support, so he knew how to use Wireshark. Good for us. So we open the pcap file and immediately notice a lot of black color. Every second or so a packet arrives 5 milliseconds late, and other packets arrive before it. TCP stack reacts correctly by sending duplicate ACKs, and once the late packet arrives, there's one cumulative ack for everything. But that's too late, 3+ duplicate acks were sent. Once the server receives these, it stops slow start mode and implements congestion control, which is CTCP, so send rate is halved.
To make things worse, MSS is only 587 bytes, and RTT is on the order of 220 milliseconds. Resulting transfer rate is about 20 KB/s, far below 1Gbit/s that is normally available.

How do you convince the network support team that it's a problem that they need to fix? Well, I'm still trying to figure it out.

TCP war story 2: overzealous SYN defense

Further experimenting with the load balanced system we found that while a few simultaneous connections get multi-megabyte throughput, running 10 or more connections at the same time resulted in some of the connections being very slow.

Running tcpdump revealed that the slow connections received SYN cookies; the SYN/ACK packet did not contain window scaling options, and receive window was limited at 64KB, again limiting the throughput at 640KB/s.

The TTL on the SYN/ACK packet was different from the TTL on all other packets on the connection; this allowed us to determine that SYN/ACK did not come from the server, but was sent by a firewall along the way.

The firewall was a Checkpoint device configured with very eager SYN defense settings. After adjusting these settings, the problem was eliminated and we were finally able to enjoy fast transfers on all connections.

TCP war story 1: F5 BIGIP load balancer

Recently I run into trouble using F5 load balancer; it was configured with standard TCP profile, and provided great performance within a data center. However, transfers crossing WAN boundaries had their throughput severely limited.

I don't normally deal with network devices, so it was a surprise for me when I found that the load balancer with TCP profile is in fact a proxy. The TCP profile limits send buffer size to 64 KB, and therefore limits throughput to 64KB / RTT, in our case 640KB/s.

After raising the send buffer size to 1MB we were able to get 10MB/s transfers, which were acceptable for our uses. Fortunately the F5 device had sufficient memory to support that buffer size.

Only later I found that F5 also supports fastL4 mode, in which case it does not act as a proxy, but rather as a regular router. In that mode the send buffer is controlled by the server directly. This reduces the memory requirements, allowing F5 to serve more connections, and shifts the responsibility for throughput to the application.