TCP Fast Open in Nginx

← Back

In the Securing public facing services post, the sysctl section specified a parameter to enable TCP Fast Open for both client and server, but Nginx needs to be configured to use it.

sysctl

# /etc/sysctl.conf

# Enable TCP Fast Open for both clients and the server
# https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html
# 0 - disabled
# 1 - enables sending data in the opening SYN on the client.
# 2 - enables the server support, i.e., allowing data in a SYN packet to be accepted and passed to the application before 3-way handshake finishes.
# 3 - enables the client and server support, so inbound and outbound
# 1024 - enable all listeners to support Fast Open by default without explicit TCP_FASTOPEN socket option.
net.ipv4.tcp_fastopen = 3

# To prevent flood attack
# To protect the server, it is important to limit the maximum number of total pending TFO connection requests, i.e., PendingFastOpenRequests
# When the limit is exceeded, the server temporarily disables TFO entirely
# Then, subsequent TFO requests will be downgraded to regular connection requests, i.e., with the data dropped and only SYNs acknowledged.
# This allows regular SYN flood defense techniques like SYN cookies to kick in and prevent further service disruption.

PendingFastOpenRequests = 256
sudo sysctl -p /etc/sysctl.conf

Nginx configuration

The Nginx configuration below uses HTTP/3 over UDP, so TCP Fast Open is mainly useful when clients fall back to HTTP/2. The value after fastopen= is the queue size for requests. A larger value allows more connections, but also increases flood risk. 

location / {
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://anubis;
}

location ~ /.ht {
    deny all;
}

# certbot

# TCP FAST OPEN
listen 443 ssl http2 fastopen=256; # enable fast open
listen [::]:443 ssl http2 fastopen=256;  # enable fast open

# Listen on UDP for QUIC+HTTP/3
listen 443 quic reuseport; # reuseport to upgrade from http2 to http3
listen [::]:443 quic reuseport; # reuseport to upgrade from http2 to http3
};

# http redirection
server {
    if ($host = www.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80 fastopen=256;  # enable fast open
    server_name example.com www.example.com;
    return 404; # managed by Certbot
}

Testing

Test client side:

curl --tcp-fastopen -I http://your-server-ip/

Test server side:

nstat -az | grep -i FastOpen

TcpExtTCPFastOpenPassive or TcpExtTCPFastOpenCookieReqd should increase. If only TcpExtTCPFastOpenActive changes, you are seeing client-side TFO from that host rather than server-side TFO accepted by Nginx.

Handshake behavior

Default handshake

1. Client sends SYN.
2. Server replies with SYN-ACK.
3. Client sends ACK.
4. Client sends the HTTP request.
5. Server sends the HTTP response.

Fast open on new connections

1. The client sends a SYN with a Fast Open option with an empty cookie field to request a cookie
2. The server generates a cookie and sends it through the Fast Open option of a SYN-ACK packet
3. The client caches the cookie for future TCP Fast Open connections

Fast open on established connections

1. The client sends a SYN with data and the cookie in the Fast Open option.
2. The server validates the cookie:
   a. If the cookie is valid, the server sends a SYN-ACK acknowledging both the SYN and the data.
      The server then delivers the data to the application.
   b. Otherwise, the server drops the data and sends a SYN-ACK acknowledging only the SYN sequence number.

3. If the server accepts the data in the SYN packet, it may send the response data before the handshake finishes.
   The maximum amount is governed by TCP's congestion control [RFC5681].

4. The client sends an ACK acknowledging the SYN and the server data.
   If the client's data is not acknowledged, the client retransmits the data in the ACK packet.

5. The rest of the connection proceeds like a normal TCP connection.
   The client can repeat many Fast Open operations once it acquires a cookie (until the cookie is expired by the server).
   Thus, TFO is useful for applications that have temporal locality on client and server connections.

References