It looks like every time I get onto a new ingress, the second thing I try to figure out is the whole OAuth thing. Now that I’m toying with Traefik in my homelab, the time has came to set up OAuth out there too, so that I could verify it’s Homekit submitting the temperature measurements into VictoriaMetrics.

The requirements I have are very simple. User interactions should have transparent authentication, i.e. if I go to grafana.homelab.example.com or traefik.homelab.example.com/dashboard/ that needs to take a round-trip to Auth0 first. Some backends have native support for that, e.g. in case of grafana or argocd the ingress doesn’t have to do anything. Other backends like traefik’s own dashboard or victorametrics’s insert daemon expect the whole authentication and authorization to be done outside.

A standard solution for that is oauth2-proxy, a reverse proxy that sits in front of your app and does the authentication (and some authorization, potentially). It’s a bit annoying to deploy it as a sidecar next to every service, though, and instead it’s much better used as a external authentication agent. Previously, I wrote about doing that with Istio, so here’s how to do it with Traefik.

Deploy oauth2-proxy

I used the oauth2-proxy’s official helm chart with the following modifications:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

config:
  existingSecret: "auth0-secrets" # I provision the secrets outside of helm
ingress:
  enable: false # we don't need the ingress on oauth2-proxy itself
extraEnv:
    # do all the logging
  - name: OAUTH2_PROXY_AUTH_LOGGING
    value: "true"
  - name: OAUTH2_PROXY_REQUEST_LOGGING
    value: "true"
  - name: OAUTH2_PROXY_STANDARD_LOGGING
    value: "true"
  - name: OAUTH2_PROXY_SILENCE_PING_LOGGING
    value: "true"
    
    # authorize against a generic OIDC provider
  - name: OAUTH2_PROXY_PROVIDER
    value: "oidc"
    # in this case, it's auth0
  - name: OAUTH2_PROXY_OIDC_ISSUER_URL
    value: "https://example.eu.auth0.com/"
    # only authenticate users with emails *@farcaller.net
  - name: OAUTH2_PROXY_EMAIL_DOMAINS
    value: "farcaller.net"
    
    # store the auth info in a cookie (the other option is server-side redis)
  - name: OAUTH2_PROXY_COOKIE_HTTPONLY
    value: "true"
  - name: OAUTH2_PROXY_COOKIE_REFRESH
    value: "1h"
  - name: OAUTH2_PROXY_COOKIE_SECURE
    value: "true"
    
    # set the cookie on my homelab's domain so it's available to any subdomain
  - name: OAUTH2_PROXY_COOKIE_DOMAINS
    value: ".homelab.example.com"
    
    # return the JWT in the Authorization header
  - name: OAUTH2_PROXY_SET_AUTHORIZATION_HEADER
    value: "true"    
    
    # don't show the frontend to pick the auth provider, we only have one
  - name: OAUTH2_PROXY_SKIP_PROVIDER_BUTTON
    value: "true"
    
    # do not strip auth headers
  - name: OAUTH2_PROXY_SKIP_AUTH_STRIP_HEADERS
    value: "false"
    
    # skip oauth2-proxy if the request has a JWT already
  - name: OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS
    value: "true"
    
    # do not reverse proxy, just return http 202 if auth is correct
  - name: OAUTH2_PROXY_UPSTREAMS
    value: "static://202"
  - name: OAUTH2_PROXY_REVERSE_PROXY
    value: "true"
    
    # redirect with a URL relative to the requesting domain
  - name: OAUTH2_PROXY_REDIRECT_URL
    value: "/oauth2/callback"
    
    # use the secure code challenge
  - name: OAUTH2_PROXY_CODE_CHALLENGE_METHOD
    value: "S256"

Let’s walk through that mess of the environment variables.

oauth2-proxy supports either cookies or redis as the session storage. Using cookies allows it to be completely stateless, but you must set the cookie-secret (in the setup above it’s being passed through the existingSecret) to make sure oauth2-proxy has a way to cryprographically verify the cookie. You want the cookie-domains to be set rather wide so that if you log into one backend you can then use the same JWT for another one.

We ask it to return the JWT in the Authorization header back to Traefik for further authorization decisions. oauth2-proxy only serves to verify who the requester is by sending them to Auth0 to enter login and password, or tap the hardware key, or whatever. The result of that is a signed JWT. We don’t do anything if the request already bears a JWT that oauth2-proxy deems acceptable.

Finally, we enable reverse-proxy, that actually tells oauth2-proxy to trust the X-Forwarded-* headers, and we set the upstream to static://202 so that it actually doesn’t proxy any backend but just returns 202.

Configure Traefik

Let’s start with a simple case: secure the Traefik dashboard. In this case, we don’t care about authorization, just authentication, i.e. anyone who can pass oauth2-proxy (by logging in to Auth0 with a matching email-domain, or presenting a good token) can access it. For that, we first define the following middleware:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: oauth-auth
  namespace: oauth2-proxy
spec:
  forwardAuth:
    address: http://oauth2-proxy.oauth2-proxy.svc.cluster.local/
    authResponseHeaders:
    - Authorization
    trustForwardHeader: true

You might say, but it’s very much the same as the official example of using oauth2-proxy with Traefik! Almost. Note how they use https://oauth.example.com/ as an address. If you enable oauth2-proxy’s ingress (at e.g. auth.homelab.example.com) and use it as the address, then Traefik will go through itself to reach oauth2-proxy, and when it does that, it strips the crucial X-Forwarded-* headers, because Traefik itself isn’t reverse-proxied. Thus, in k8s setup we must use the service address. In practice, you can use whatever address that maintains X-Forwarded-* headers, of course.

With the middleware in place, we update the Traefik helm chart to use it:

1
2
3
4
5

ingressRoute:
  dashboard:
    middlewares:
      - namespace: "oauth2-proxy"
        name: "oauth-auth"

Contrary to what the example in values.yaml says, in kubernetes you’re expected to pass in an object with name & value in here. I find the bit where Traefik configuration differs based on what’s the source of that configuration slightly annoying, but alas.

With these two bits in place, here’s what’s happening:

Traefik validates the request with oauth2-proxy, which doesn’t recognize the user and returns 302 redirect (because we have the skip-provider-button). Traefik sees it’s not http 2xx and forwards the response verbatim to the user. The user’s browser then redirects to Auth0, which goes through all the oauth bits and redirects to traefik.homelab.example.com/oauth/callback. Now, that route doesn’t exist and we get 404. Whoops. Let’s fix that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-oauth-redirects
  namespace: oauth2-proxy
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: HostRegexp(`^.+\.homelab\.example\.com$`) && PathPrefix(`/oauth2/`)
    middlewares:
    - name: auth-headers
      namespace: oauth2-proxy
    priority: 1
    services:
    - kind: Service
      name: oauth2-proxy
      port: http
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: auth-headers
  namespace: oauth2-proxy
spec:
  headers:
    browserXssFilter: true
    contentTypeNosniff: true
    forceSTSHeader: true
    frameDeny: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 315360000

This route matches *.homelab.example.com if the path starts with /oauth2, and sends the traffic into oauth2-proxy. It has the priority of 1 so that it’s more important than any of the actual backends on those subdomains, and it has an auth-headers middleware that throws in a ton of security headers that disallow using oauth endpoints insecurely. We also only serve it off the websecure entrypoint, meaning it never has an option of being http traffic (because redirects are bad).

With this route in place, Auth0 can call the oauth2-proxy’s callback endpoint, which sets the cookie, sets the header, and then redirects back to theoriginal page of /dashboard/, where the same flow from above now succeeds, because there’s a vaild cookie. Yay, cookies!

Machine to machine interaction

What if you actually want to further authorize the endpoint or do a machine-to-machine flow? Apparently, Traefik proxy doesn’t support that unless you cash out for the enterprise version (because no normal people do JWTs?..). Luckily, the open-source one supports plugins and there’s a plugin for that. Let’s enable it in the traefik chart:

1
2
3
4
5

experimental:
  plugins:
    jwt:
      moduleName: "github.com/Brainnwave/jwt-middleware"
      version: "v1.2.1"

And then create a middleware:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: vmcluster-jwt
spec:
  plugin:
    jwt:
      issuers:
        - https://jwts.farcaller.net
      skipPrefetch: true
      secrets:
        vicmet: |
          -----BEGIN EC PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE5OEKUhuyWPQV7Cf+Y1WohGNVr86Y
          OBpuIeL4TqJhGp35CYnrZahUsh9bEzXRstrXMUjTWKkaEcP2bv7cqKoe2Q==
          -----END EC PUBLIC KEY-----          
      require:
        aud: vminsert.homelab.example.com

Adding this middleware (I used the Ingress annotation of traefik.ingress.kubernetes.io/router.middlewares: "victoria-metrics-vmcluster-jwt@kubernetescrd") does the following checks. First, the request must carry a walid JWT in the Authorization header. Second, that JWT must be signed by the public key provided (and we skip prefetching the jwks because the actual issuer isn’t real). Finally, the audience claim must be set to vminsert.homelab.example.com. We can easily generate a token that matches the specs using https://jwt.io. The header would be

1
2
3
4
5

{
  "alg": "ES256",
  "typ": "JWT",
  "kid": "vicmet"
}

Notice the additional kid being the key id. This way you can specify several different keys in the middleware, if needed. The payload is:

1
2
3
4
5
6
7

{
  "iss": "https://jwts.farcaller.net",
  "sub": "homekit",
  "aud": "vminsert.homelab.example.com",
  "iat": 1719591241,
  "exp": 2540042041
}

In here, the iss and aud must match with what we verify. The subject is purely informational and the iat and exp provide the time window for which this token is valid.

Now, copy and paste your ec private key into jwt.io securely sign the JWT and plug it into the requester side. Tada, your requests succeed again.

Closing words

Generally, you should always do both authentication and authorization and there’s a very narrow use case (like an OIDC provider that only authenticates you) where authorization isn’t useful. Traefik is definitely fit to do both if paired with oauth2-proxy, and, with plugins like this, it can also do the full-blown OPA validation of the requests, meaning you can write policies for individual endpoints.

While the oauth2-proxy setup documented on their page can be useful, it might be an overkill. Also, the option where the errors middleware redirects to the correct page on 401 is currently broken if Traefik has metrics collection enabled.

JWTs provide a simple way to manage credentials. While the best way to use them is to have something like OpenBao or Vault or Auth0’s M2M to supply your consumers with short-lived tokens there’s nothing stopping you from signing a token manually and pasting it in the abysmal UI of Apple’s Scripts. It’s slightly more secure than just having a password, because you can limit the given aud claim to a specific IP address in your OPA policy, and thus, even if the token is stolen, the attacker will still have a hard time using it.