L4 forwarding: client IP to the origin
The three ways L4 forwarding passes the client IP to the origin, the TOA option format and limits, installing a TOA receiver, and how to verify it.
The three options
"Pass client IP to origin" on the L4 forwarding page has three options; pick one. All of them are for TCP only.
| Option | What the origin needs | Notes |
|---|---|---|
| Off | Nothing | The origin sees the node's address |
| PROXY protocol v1 | The origin program supports PROXY protocol (nginx listen ... proxy_protocol, HAProxy accept-proxy) | An application setting, no kernel change; carries IPv4 and IPv6 clients |
| TOA | A TOA receiver on the origin (kernel module or eBPF) | No application change; getpeername() returns the client address; IPv4 only |
How TOA works
On the TCP connection to the origin, from the handshake ACK onwards, the node adds a TCP option to the first few segments: kind 254, length 8, carrying the client port (2 bytes) and IPv4 address (4 bytes) in network byte order. Once the origin has acknowledged data, the node stops adding it. The origin's TOA receiver reads the option and makes getpeername() on that connection return the client address.
Limits:
- IPv4 clients only. Connections from IPv6 clients are forwarded as usual without TOA, so the origin sees the node; use PROXY protocol when you need IPv6 client addresses.
- Without a receiver the origin kernel ignores the option; the connection works, only the client address is missing.
- Nodes need Linux 5.10 or later.
- Devices between node and origin that strip unknown TCP options (some firewalls and load balancers) remove TOA.
Installing a receiver on the origin
We do not ship an origin module. Common open source implementations are below; check each repository's instructions and licence before use:
| Implementation | Repository | Notes |
|---|---|---|
| Huawei Cloud ELB TOA | https://github.com/huaweicloud/elb-toa | Kernel module |
| Huawei TCP_option_address | https://github.com/Huawei/TCP_option_address | Kernel module; old, may not build on new kernels |
| Alibaba Cloud CDN TOA | https://github.com/aliyun/alibabacloud-cdn-tool-toa | Kernel module and eBPF version (eBPF needs kernel 5.10+ and cgroup v2) |
Kernel modules usually need headers matching the running kernel, such as linux-headers-$(uname -r) on Debian/Ubuntu or kernel-devel-$(uname -r) on RHEL/Rocky; then build and insmod as the repository describes.
Verifying
- Choose TOA in the L4 forwarding and publish.
- Connect to the forwarded port from a client and check the peer address in the origin application's log: it should be the client's public address.
- Do not judge by
ss -tnornetstat: kernel module receivers only change whatgetpeername()returns, not the kernel connection table, so these commands still show the node. - If it does not work, capture on the origin:
tcpdump -nn -v -i <interface> 'tcp port <origin port>'. The handshake ACK options should include kind 254 (tcpdump shows it asexp-). Present but the application still sees the node: the receiver is not installed correctly. Absent: check the node kernel and devices in between.
Common questions
TOA is installed but the log still shows the node
Capture first to see whether the option reaches the origin. If it does, check that the receiver is loaded and that the application uses getpeername() rather than parsing some header.
The client uses IPv6
TOA does not carry IPv6 clients; use PROXY protocol v1.