I finally passed my ham exam and now I can do more exciting radio transmissions than just an 802.15.4 mesh. With the license, I got access to the awesome 44net, too. 44net is a community for the licensed radio operators to experiment and learn using real reachable IPv4 addresses. They used to own the whole 44/8, but now it’s a /9+/10 (one /10 went off to Amazon in exchange for grants, as far as I know). There are different ways to get onto 44net. In practice, this comes down to either using a WireGuard tunnel which will get you access to networks of any size up to /25, or getting a /24 for your ASN that you can then publicly announce. While the latter is what I want to experiment with along with multihoming, the former is what’s easy to access. I got a /28 allocated within days with no real blockers. 44net supports either static routes or BGP Peering over private ASNs. While static routing is easier, this blog is about not doing things because they are easy but because they are explicitly hard. Apparently, BGP isn’t even that hard, as long as you have a fully upgraded RouterOS! There’s an example config for RouterOS even, but as it focuses on static routing, I had to make some adjustments. Before routing publicly reachable IP addresses I decided to set up a VRF—Virtual Routing and Forwarding instance. While VRFs mainly solve the problem of overlapping IP regions, they also provide further isolation of routing. Think of it as a tiny virtual router inside the main router that has its own routing tables that don’t cross into the main traffic. VRFs make firewalling a little awkward in both directions: firewall rules can surprise VRF traffic, and VRF membership can surprise interface-based firewall rules, but it’s not too bad if you’re careful. VRF needs to have at least one interface, so first we create a wireguard tunnel, and then hook it up to a VRF:

/interface wireguard add listen-port=32184 mtu=1380 name=wg-44net
/ip vrf add interfaces=wg-44net name=44net

It’s important to order VRFs properly: they are cascading from top to bottom and any interface not intercepted in an earlier VRF will fall into the main. We set up a BGP instance next:

/routing bgp instance add as=12345 name=bgp-44net router-id=44.2.3.4 routing-table=44net vrf=44net

BGP cares about both a VRF and a routing table. A VRF is the “routing instance” BGP will use to make its own TCP sessions (we want them to happen over the wireguard tunnel that’s inside 44net vrf). Routing table is where it will sync the routes, which is 44net (each VRF creates a same-named routing table automatically). We can set up the connection now, first over wireguard:

/interface wireguard peers add allowed-address=0.0.0.0/0 client-allowed-address=::/0 endpoint-address=44.27.227.1 endpoint-port=44000 interface=wg-44net name=wg-44net-fra-02 persistent-keepalive=25s public-key=xxxyyy=
/ip address add address=44.2.3.4 interface=wg-44net network=44.6.7.8

The IP addresses all come from the 44net dashboard. You have 3:

  • the IP address of the remote end of the wireguard tunnel (44.27.227.1 above)
  • the IP address of the remote BGP server (44.6.7.8)
  • the IP address of your local BGP server (44.2.3.4, note that it’s also used as the BGP ID) At this step, the wireguard tunnel should be operational. If it’s not, then the problem is probably the firewall or wrong IP addresses. Next step is setting up the BGP session. As you know, S in BGP stands for Secure, thus we need to add a layer of security on top of it. The most important rule with BGP is that you don’t import routes you don’t need and you don’t export routes you mustn’t. BGP does that with filtering rules:
/routing filter rule add chain=44net-bgp-out rule="if (dst in 44.10.11.12/28 && dst-len >= 28 && dst-len <= 32) { accept }"
/routing filter rule add chain=44net-bgp-out rule=reject

/routing filter rule add chain=44net-bgp-in rule="if (dst == 0.0.0.0/0) { set distance 20; accept }"
/routing filter rule add chain=44net-bgp-in rule="if (dst in 44.0.0.0/9 && dst-len >= 9 && dst-len <= 32) { set distance 20; accept }"
/routing filter rule add chain=44net-bgp-in rule="if (dst in 44.128.0.0/10 && dst-len >= 10 && dst-len <= 32) { set distance 20; accept }"
/routing filter rule add chain=44net-bgp-in disabled=no rule=reject

The rules are pretty straightforward: we only export routes that fit into our subnet of 44.10.11.12/28, we only import either a default gateway or an IP that belongs to 44net. Note that importing 0.0.0.0/0 is entirely safe as it will be inside of the 44net routing table and thus won’t affect any usual routing flows. Filtering non 44net subnets is defensive in this case; those are the only subnets the upstream session will export to you either way. On RouterOS, what you export is defined by Address Lists (yeah, the firewall ones). Probably Mikrotik people decided to just reuse the concept? Either way we need to declare what we’re actually exporting:

/ip firewall address-list add address=44.10.11.12/28 list=EXPORTED_SUBNET

and with all that in place, we can create a connection:

/routing bgp connection add
  afi=ip
  as=12345
  connect=yes
  input.filter=44net-bgp-in
  instance=bgp-44net
  listen=no
  local.address=44.2.3.4
  .role=ebgp
  name=to-44net-connect
  output.filter-chain=44net-bgp-out
  .network=EXPORTED_SUBNET
  .network-blackhole=yes
  remote.address=44.6.7.8/32
  routing-table=44net
  use-bfd=no
  vrf=44net

and, after some waiting, check on it in /routing/bgp/session/print (note that WinBox UI is cursed when it comes to showing the active connections and it might not reflect what’s happening for real. Use the CLI). You should see received routes in IP routes and advertised routes in /routing/bgp/advertisements/print. If you change the addresses in EXPORTED_SUBNET (while making sure they are still within the filter) you’ll see the advertisements follow. Note that 44.2.3.4 (the BGP client address) is on 44/8 and thus your router is publicly reachable over it. None of the router services are accessible on VRF 44net, but it’s important to have the input firewall policies tight. Another note: if you don’t import 0.0.0.0 from 44net, you can technically return the traffic asymmetrically, by e.g. leaking /ip route add dst-address=0.0.0.0/0 routing-table=44net gateway=<isp-gateway>@main distance=1 which returns the packets via your primary ethernet (or whatever). However it’s not guaranteed to work (my ISP drops all the packets that aren’t originated from me, which is actually a reasonable thing to do). At this point the announced network is fully functional, you can redistribute the IP addresses by DHCP, add static routes, DNAT them, whatever. It works. But it wouldn’t be fun to get all the way here and use DHCP so let’s talk Kubernetes and announcing the LoadBalancer services on 44net. I’m using cilium which has reasonably decent support for BGP using gobgp. To make services work, the plan is to route them over a network separate from the main VLAN the kube nodes are on. I tried a second VLAN but got stuck with unrelated IPv6 breakage, so in this setup I just have a second ethernet on the compute node plugged right into my mikrotik router. There are several ways to make BGP work; in the end it’s just a TCP connection so it can go over IPv4, IPv6 or “magical IPv6” (BGP unnumbered is when two BGP servers find each other over a direct connection using IPv6 link local addresses). Unfortunately, cilium doesn’t support BGP unnumbered, and I don’t want to carve out even more IPv4 addresses for routing so IPv6 it is. Also, we can’t use the link-local fe80:: addresses as they freak mikrotik out and it drops the session (BGP was only around for what, 37 years? Look, things are complicated), so we’ll use ULA (aka the “private” IPv6). It’s only a few resources, really:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: lab-router
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  bgpInstances:
    - name: kube-44
      localASN: 12345
      peers:
        - name: lab-router-44net
          peerASN: 12345
          peerAddress: "fde0:1234:5678:44::1"
          peerConfigRef:
            name: lab-router-44net
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: lab-router-44net
spec:
  transport:
    peerPort: 179
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          routerops.io/export: lab-router-44net
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: lab-router-44net-routes
  labels:
    routerops.io/export: lab-router-44net
spec:
  advertisements:
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchLabels:
          lb: 44net
---
apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
  name: fortyfournet
spec:
  blocks:
  - cidr: "44.10.11.12/28"
  serviceSelector:
    matchLabels:
      lb: 44net
  disabled: false

First, we tell cilium that we peer ASN 12345 to ASN 12345 (i.e. iBGP—the internal one) and that the other endpoint is at fde0:1234:5678:44::1, then we tell it that this peer only advertises whatever’s labeled with routerops.io/export=lab-router-44net, and then the advertisement says it’s any Service type LoadBalancer with lb=44net. Finally, we create a load balancer pool (you also want to spin up some pods and services). On RouterOS, the changes are even smaller. The connection doesn’t use the output.network and instead does output.redistribute=bgp:

/routing bgp connection add
  afi=ip
  as=12345
  connect=yes
  input.filter=44net-bgp-in
  instance=bgp-44net
  listen=no
  local.address=44.2.3.4
  .role=ebgp
  name=to-44net-connect
  output.filter-chain=44net-bgp-out
  .redistribute=bgp
  remote.address=44.6.7.8/32
  routing-table=44net
  use-bfd=no
  vrf=44net

The filter chain tightens:

/routing filter rule add chain=44net-bgp-out \
  rule="if (protocol bgp && bgp-input-remote-as == 12345 && dst in 44.10.11.12/28 && dst-len >= 28 && dst-len <= 32) { accept }"

we only advertise routes that came from BGP with another ASN 12345.

We need to set up the IP address cilium expects and add the interface from cilium to the VRF:

/ipv6 address add address=fde0:1234:5678:44::1 advertise=no interface=eth14-cilium
/ip vrf add interfaces=eth14-cilium,wg-44net name=44net

And then set up the filters and a new connection (on the same instance):

/routing filter rule add chain=44net-cilium-in rule="if (dst in 44.10.11.12/28 && dst-len >= 28 && dst-len <= 32) { accept }"
/routing filter rule add chain=44net-cilium-in rule=reject

/routing filter rule add chain=44net-cilium-out rule="if (dst in 44.0.0.0/8 && dst-len >= 8 && dst-len <= 32) { reject }"
/routing filter rule add chain=44net-cilium-out rule=accept

/routing bgp connection add
  afi=ip
  as=12345
  connect=no
  input.filter=44net-cilium-in
  instance=bgp-44net
  listen=yes
  local.address=fde0:1234:5678:44::1
  .role=ibgp
  name=cilium-44net
  output.filter-chain=44net-cilium-out 
  remote.address=fde0:1234:5678:44::2/128
  .as=12345
  routing-table=44net
  use-bfd=no
  vrf=44net

Here we only accept our 44.10.11.12/28 from cilium. We also don’t advertise any 44/8 routes into it (otherwise cilium will see 400+ routes towards the other 44net subnets that it shouldn’t bother about). In practice, we don’t even need a filter and we shouldn’t advertise anything at all towards cilium but this shows how you can be flexible (my cilium has other LB pools including IPv6 and this helps in propagating routes where they should go). Now, the only thing left to do is to convince the packets to flow as intended. In my case cilium falls back to normal linux kernel routing, so I just add the source routing policy with systemd-networkd:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# /etc/systemd/network/50-motherboard-eth.network
[Match]
Name=eth0

[Network]
Address=fde0:1234:5678:44::2/64
IPv6AcceptRA=no

[Route]
Destination=0.0.0.0/0
Gateway=fde0:1234:5678:44::1
GatewayOnLink=yes
Table=44

[RoutingPolicyRule]
From=44.10.11.12/28
Priority=1044
Table=44

I am not accepting RAs on the interface because otherwise this can become a default gateway for the whole box by accident. I also hardcode the local and remote IP addresses and add a routing policy that nudges the return traffic from 44.10.11.12/28 towards this interface (so it lands on ether14-cilium routeros side and will be inside the VRF that knows how to route it further into the internet via 44net tunnel). So here’s how the traffic flows:

  • curl http://44.farcaller.net leaves towards the nearest 44net presence via my ISP
  • the packet enters 44net and is routed towards my mikrotik over wireguard
  • it redirects it to kube box following the route learned from cilium
  • cilium picks it off eth0 and tosses it into the pod via ebpf
  • the pod replies
  • cilium returns the reply to linux kernel stack for routing
  • the kernel uses the source routing to send this off eth0, not the primary interface
  • it returns to mikrotik and follows out through tunnel as that’s the default gateway in 44net VRF
  • the packet makes a round trip and returns to my ISP, through mikrotik LAN and into my macbook again Notice how my macbook cannot talk to the pod directly because VRF isolates the LAN and 44net.