[MSVC] Linking to DLLs when no .lib is available

Reference:
https://stackoverflow.com/a/16127548/7707617

Step 1) Generate exports file
>dumpbin /exports libcurl.dll > libcurl.exports

Step 2) Edit exports file to leave just the word EXPORTS in the first line and function names in the following lines. The result is a .def file.

Step 3) Create lib from def file:
>lib /def:libcurl.def /out:libcurl.lib

Step 4) Pass the resulting lib file to linker as usual.

Libcurl and slow uploads on Windows

Recently I started using libCurl 7.60 to upload files from Windows machines to Amazon cloud using HTTP POST. The uploads don't perform well. They are limited by the send buffer used by Windows.
On Windows 2008R2 (and probably on more recent versions as well) the system only puts on the wire the bytes it has buffered, and this is limited by SO_SNDBUF (default 8KB) or by the send buffer used by application code (CURL_MAX_WRITE_SIZE, 16KB), whichever is higher. In case of CURL uploads, the system buffers 16KB, then waits for acknowledgement from the other end before buffering and sending the next batch. This is readily visible in Wireshark traces.

Starting with Windows 7 / 2008R2, Windows implements send buffer autotuning. This is well described here.

Theoretically on these systems the send buffer should be automatically adjusted to optimize throughput. I run a couple experiments to confirm that, and found that the send buffer is only adjusted if the socket is blocking and application buffer size is reasonably large (in my experiment 16KB was too small, but 20KB was sufficient), and buffer stays at 8192 if the socket is nonblocking regardless of the application buffer size.

Curl is using nonblocking sockets, and switching to blocking would break existing functionality. In order to use the optimal buffer size, it would need to periodically update SO_SNDBUF to a value provided by SIO_IDEAL_SEND_BACKLOG_QUERY or SIO_IDEAL_SEND_BACKLOG_CHANGE.

I sent a proof-of-concept patch to curl-library mailing list. With some luck the patch will find its way to the next curl release.

RDP session hijacking (without password / as administrator)

1. Get session ID of the session you want to connect to
2. Get PsExec
3. Run CMD under System account from admin CMD: psexec -i -s -d cmd
4. Run tscon.exe <sessionID>

 Sources:

1.  PsExec
2. Getting a CMD prompt as SYSTEM in Windows Vista and Windows Server 2008 
3. RDP hijacking — how to hijack RDS... 

Bulk loading data to SQL server

Source: tab-separated file
Destination: table with the same structure as the file
Command:
bcp <table> in <file> -c -S <server> -U <user>

Other options:
-t : column separator (tab is the default)
-r : row separator (default: newline)
-f : format file - can be used to import files that do not match the list of columns in DB table

TCP gotchas

When I first learned about TCP, I found a few things surprising. Here's a short list:

Server socket has an "accept" but not a "reject" method

Connection set up is handled by the operating system (OS). The application has no way of examining the client before establishing the connection. If you want to ban connections from a certain IP, your can only use a firewall or close the connection immediately after accept.

Blocking "send" does not block until the data is delivered to the other end

Send operation just copies the data to OS buffer for transmission. If the OS has sufficient free buffer space, send operation returns immediately. Send only blocks when OS buffer is full.

"ACK" packet does not mean that the application on the other end successfully received the message

ACKs are sent by the receiving operating system when it stores the message in its internal buffer. The receiving application may read the data at a later point, or not at all.

TCP does not guarantee that a broken connection will raise an error

If you send some data and then close a connection, both operations may succeed even if no data is actually delivered to the other end, for example because the machine on the other end lost its network connection. And conversely, if the sending machine crashes, the receiving machine may never notice.

As far as reliability is concerned, TCP guarantees only the following:

• No data will be delivered out of order
• Everything you send will be delivered at most once
• If the OS can tell that an operation cannot succeed, it will return an error (like when you send to a connection that is already known to be broken)

Any further guarantees are the responsibility of the application.

Range of Random.nextGaussian, continued

In the first part I described a theoretical range of values returned by nextGaussian; in this part I will describe the way used to find out the actual minimum and maximum values.

The simplest way would require using brute force to check all 2⁴⁸ possible seed values. This would take a few years on my computer, so I had to do better than that.

As explained in the first part, in order to maximize the values of nextGaussian, the values of v1 and v2 need to be as close to zero as possible. For that, the value returned by nextDouble needs to be close to 0.5. So, how do we make the random number generator return the values we want?

NextDouble calls next two times, first to get top 26 significant bits of the result, then again to get next 27 bits. If we can get the top bits to be 1 << 25, the resulting double will be very close to 0.5.

A call to next first updates the seed value used by Random, then returns top N bits of the new seed. So, we know the desired seed value after the call to next. In order to find out the value for seed before the call, we can use the simple reversing algorithm found here:

private static final long multiplier = 0x5DEECE66DL;
private static final long addend = 0xBL;

private static long reverseSeed (long seed) {
    // reverse the addend from the seed
    seed -= addend;
    long result = 0;
    // iterate through the seeds bits
    for (int i = 0; i < 48; i++) {
        long mask = 1L << i;
        // find the next bit
        long bit = seed & mask;
        // add it to the result
        result |= bit;
        if (bit == mask) {
            // if the bit was 1, subtract its effects from the seed
            seed -= multiplier << i;
        }
    }
    return result;
} 

Then, we also need to counter the initial scrambling that happens in setSeed operation:

private static long unscramble(long seed) {
    return seed ^ multiplier;
}

 And we're ready to start hacking:

long seedRange = 0x2000000L;
Random rand = new Random();
double minSoFar = 0, maxSoFar = 0;
for(long seed = -seedRange;seed<seedRange;seed++) {
    rand.setSeed(unscramble(reverseSeed(0x800000000000L + seed)));
    double d = rand.nextGaussian();
    if(minSoFar > d) {
        minSoFar = d;
        System.out.println("Seed: "+seed+ " min: "+d);
    }
    if(maxSoFar < d) {
        maxSoFar = d;
        System.out.println("Seed: "+seed+ " max: "+d);
    }

    d = rand.nextGaussian();
    if(minSoFar > d) {
        minSoFar = d;
        System.out.println("Seed: "+seed+ " second min: "+d);
    }
    if(maxSoFar < d) {
        maxSoFar = d;
        System.out.println("Seed: "+seed+ " second max: "+d);
    }
}

The lowest value found was: -7.844680087923773 (on second call to nextGaussian with seed = 994892)
The highest value found was: 7.995084298635286 (on first call to nextGaussian with seed = 14005843)
These are the real maximum & minimum values returned by Oracle Java 8 implementation of nextGaussian.

Range of Random.nextGaussian()

Recently I had a look at the source code of a language detection library. The library internally uses Random.nextGaussian to determine its behavior, but uses it in such a way that any value lower than -10 would result in incorrect calculations. I was curious if getting such a value is even possible.


The implementation of nextGaussian in Java 8 is as follows:

 private double nextNextGaussian;
 private boolean haveNextNextGaussian = false;

 public double nextGaussian() {
   if (haveNextNextGaussian) {
     haveNextNextGaussian = false;
     return nextNextGaussian;
   } else {
     double v1, v2, s;
     do {
       v1 = 2 * nextDouble() - 1;   // between -1.0 and 1.0
       v2 = 2 * nextDouble() - 1;   // between -1.0 and 1.0
       s = v1 * v1 + v2 * v2;
     } while (s >= 1 || s == 0);
     double multiplier = StrictMath.sqrt(-2 * StrictMath.log(s)/s);
     nextNextGaussian = v2 * multiplier;
     haveNextNextGaussian = true;
     return v1 * multiplier;
   }
 }

The function used to generate next values has extremes when both v1 and v2 are as close to zero as possible, without being both equal to zero.

Quick check of nextDouble indicates that the function generates only multiples of 1 / (double)(1L << 53). This means that nextGaussian will return extreme values for v1 = +/- 2 / (double)(1L << 53)and v2 = 0.

For these values nextGaussian would return +/- 12.00727336061225.

That does not mean that it is possible to get these values. There's a limited number of values that can be returned by nextDouble. But it does mean that nextGaussian (in its Oracle Java 8 implementation) will never return a value outside of the range between -12.00727336061225 and 12.00727336061225.