I’ve been doing a lot of traffic engineering studies now that I have my own ASN; apparently traffic engineering is a massive field with many interesting problems you never get to experience unless you’re doing network operations.

I have several upstreams and IXPs terminated on my router. What are they? Upstreams are providers that offer me transit, aka the broad “access to the rest of the internet.” They basically agree to transit packets from my network to networks they can reach (and they always can reach more) for free, or for a small fee (or for a very large fee if you have lots of traffic).

IXPs are internet exchange points—these are physical locations that boil down to a very large switch. Different companies have ports in that switch (or many switches, IXPs can even span countries) just to be able to send traffic to other companies laterally instead of upwards. As upstreams cost (significant) money it’s usually cheaper to send traffic sideways, and to prevent cables running all over the towns it’s feasible to find a single physical location where all of the cables go. IXPs can also be more or less “virtual”, meaning that you don’t get a physical port but rather a L2 tunnel.

Back to the problem at hand. If you open https://as219458.farcaller.net the packets will reach the webserver via one of six ways:

  • the primary uplinks being route64 and lagrange.cloud (I pay little money for both)
  • the secondary uplink of bgp.tunnel
  • or one of 3 IXPs I’m connected to virtually: bgp.exchange in London and Amsterdam (my actual physical ISP has fiber that runs very close to both locations so that’s efficient for me) and 4ixp in Zurich.

The question I want to answer is: how did you reach the webserver?

If you think in common terms, the answer is obvious: DNS. Instead of a single 2a0b:4e07:290f::fa2c:a11e:2 just allocate an IP address per upstream/ixp and then server-side you know how the traffic comes in. Of course that doesn’t quite scale well because the minimal subnet size you can announce publicly in practice is /24 for IPv4 and /48 for IPv6 (everything else will probably get filtered) and thus it’d require six whole IPv6 subnets just for one website which is extremely wasteful. Also, it wouldn’t quite work in practice as IXPs are not globally connected to the internet, so if your PC decided to pick the random DNS value of 2a0b:4e07:290f::fa2c:a11e:201 which I’d announce for the bgp.exchange in Amsterdam then you’d most probably fail to load the website.

A single IP then, which means a single VIP for the https server. It’s a trivial setup, nothing exciting going on in here—I just set the IP as the ServiceType=LoadBalancer and let cilium announce it to my router over internal BGP. Now the traffic flows in because my router announces the overarching /48 to the world and cilium tells the router that the /128 is reachable through the cluster.

Traffic flows in from any upstream, gets to the server, you get a neat http page. But how does it know where it came from?

There are several solutions to it which boil down to two primary options: you either mangle the packets or you encapsulate them.

Encapsulation in a nutshell is when the router decides to send traffic from each upstream via unique tunnel, or even separate VLANs. The webserver could figure then where the traffic came from based on the source interface that is local to it. A bit messy to set up, also boring as it’s objectively straightforward.

Mangling has a bunch of options, the easiest being NAT. Unlike the common src-nat that you probably use right now (packets from your internal IPv4 address get the source rewritten to the address of the router), dst-nat rewrites the destination. Instead of reaching my webserver on [fde0::1]:443, it rewrites the destination port based on the source interface. You came from lagrange? You’re rewritten to [fde0::1]:1443. From route64? [fde0::1]:2443, etc. This way the webserver has to listen on several ports but then it’s trivial to know where the packet came from. Again, it’s a bit of a boring setup. We can overengineer it still :-)

If you look at the IPv4 packet diagram long enough you’ll start noticing things.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  0 |Version|  IHL  |    DSCP   |ECN|          Total Length         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  4 |         Identification        |Flags|      Fragment Offset    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  8 |  Time to Live |    Protocol   |         Header Checksum       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 12 |                       Source Address                          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 16 |                    Destination Address                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 20 |              Options (0-40 bytes) + Padding                   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

There’s a bunch of things you expect, like the source and destination addresses, the inner protocol, length, checksum, etc. There are also flags which all are useful, and there’s a whole byte for DSCP and ECN. What are those?

ECN is used for congestion notifications, a way to control the flow between parties that understand it. What it does in practice (at least for TCP) is that when ECN support is negotiated and a router marks a packet as “congestion experienced”, the receiver reports that signal to the sender, which reduces its congestion window. The marked packet was still delivered. It’s a neat trick, it might be useful in practice, and either way, these 2 bits are well defined.

DSCP is mostly used for QoS, providing priority lanes for traffic that’s more important (e.g. discord). It’s also well defined, but there are a whole of 6 bits, and the registryspecifies 3 pools where the space of xxxx11 is reserved for “experimental or local use”.

Looks like we just found ourselves a whole 4 bits of data to use! Let’s see how we can make it travel down this path:

Ingress interface → DSCP label → saved SYN in HAProxy → PROXY v2 TLV → Envoy HTTP header.

With DSCP spare bits, and without abusing anything else in the packets, we can encode 16 different values in them (IPv6 has the same byte for DSCP+ECN) with a trivial firewall rule that checks the ingress interface and sets the DSCP bit without amending anything else. The modified packets then travel to our http server that… What does it do?

There’s a mechanism in linux called TCP_SAVE_SYN that allows the listener to retrieve the original SYN packet that opened the TCP stream. You just need to setsockopt requesting it, and then the packet is available via getsockopt once you accept() the connection. This gives the process the raw packet (with or without the ethernet frame), which also includes the bits we care about.

The great thing is that you don’t even need to vibecode anything at 2am with your LLM because haproxy 3.4+ actually supports this mechanism! All it takes is something like

bind ... tcp-ss 1

tcp-request connection set-var(sess.syn) fc_saved_syn
tcp-request connection set-var(sess.dscp) var(sess.syn),ip.tos,and(252),div(4)

Now you have the upstream of the router as a variable in your http server and it’s kube-friendly and everything. I could have wrapped it here but I don’t want to terminate TLS in haproxy so instead I use the proxy protocol v2 to pass it down to the real ingress (in my case, istio’s envoy):

  frontend labasn_public_asn_blog_tls
    bind 44.30.165.73:443 tcp-ss 1
    bind [2a0b:4e07:290f::fa2c:a11e:2]:443 v6only tcp-ss 1

  tcp-request connection set-var(sess.syn) fc_saved_syn
  tcp-request connection set-var(sess.dscp) var(sess.syn),ip.tos,and(252),div(4)

  tcp-request connection set-var(sess.uplink) str(unknown)
  tcp-request connection set-var(sess.uplink) str(upstream-route64-ams1) if { var(sess.dscp) -m int eq 3 }
  tcp-request connection set-var(sess.uplink) str(upstream-lagrange-lon) if { var(sess.dscp) -m int eq 7 }
  tcp-request connection set-var(sess.uplink) str(ixp-bgpex-lon) if { var(sess.dscp) -m int eq 11 }
  tcp-request connection set-var(sess.uplink) str(ixp-bgpex-ams) if { var(sess.dscp) -m int eq 15 }
  tcp-request connection set-var(sess.uplink) str(ixp-4ixp-zur01) if { var(sess.dscp) -m int eq 19 }
  tcp-request connection set-var(sess.uplink) str(upstream-bgptunnel-ams) if { var(sess.dscp) -m int eq 23 }


    default_backend asn_blog_ingress_tls

  backend asn_blog_ingress_tls
    server asn-blog-ingress [fde0::1]:443 source [fde0::2] send-proxy-v2 set-proxy-v2-tlv-fmt(0xE0) %[var(sess.uplink)] check check-ssl check-sni as219458.farcaller.net ca-file /nix/store/zdl7dn1gmi1cxdw5a8hw6xsf4cz0rjmg-nss-cacert-3.123/etc/ssl/certs/ca-bundle.crt verify required verifyhost as219458.farcaller.net

and then on istio side there’s a little bit of envoy configuration to read proxy protocol and convert it to a normal http header that the web server can reason with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: blog-proxy-protocol-tlv
spec:
  workloadSelector:
    labels:
      istio: ingressgateway-labasn
  configPatches:
    - applyTo: LISTENER_FILTER
      match:
        context: GATEWAY
        listener:
          portNumber: 8443
          listenerFilter: envoy.filters.listener.tls_inspector
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.listener.proxy_protocol
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
            stat_prefix: blog_proxy_protocol
            allow_requests_without_proxy_protocol: false
            tlv_location: FILTER_STATE
            rules:
              # same matching key as above. The text comes from haproxy
              - tlv_type: 224
                on_tlv_present:
                  key: uplink
                  value_string_encoding: SANITIZED_UTF8
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: blog-uplink-header
spec:
  workloadSelector:
    labels:
      istio: ingressgateway-labasn
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          portNumber: 8443
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: envoy.filters.http.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            default_source_code:
              inline_string: |
                function envoy_on_request(handle)
                  local headers = handle:headers()

                  -- make sure it didn't come from the client
                  headers:remove("x-uplink-name")

                  local uplink = nil
                  local fs = handle:streamInfo():filterState()

                  if fs ~= nil then
                    -- read it from the previous filter
                    uplink = fs:get("envoy.network.proxy_protocol.tlv", "uplink")
                  end

                  if uplink == nil or uplink == "" then
                    uplink = "unknown"
                  end

                  headers:add("x-uplink-name", uplink)
                end

And just like that, the ASN landing page tells you which ingress you came from!