Isn’t it nice when you wake up and your auth is broken, despite you not changing anything? Isn’t it fun to debug a third-party integration?
Anyways, I stumbled on an annoying gRPC error:
Given the backend didn’t log the request, I blamed the middle man, aka envoy. Indeed, bumping the envoy noisiness with -l trace reveals the following:
[2020-08-19 10:06:08.255][12][debug][pool] [source/common/http/conn_pool_base.cc:71] queueing request due to no available connections
[2020-08-19 10:06:08.255][12][debug][pool] [source/common/conn_pool/conn_pool_base.cc:53] creating a new connection
[2020-08-19 10:06:08.255][12][debug][client] [source/common/http/codec_client.cc:35] [C1] connecting
[2020-08-19 10:06:08.255][12][debug][connection] [source/common/network/connection_impl.cc:753] [C1] connecting to [2606:4700::6810:605e]:443
[2020-08-19 10:06:08.255][12][debug][connection] [source/common/network/connection_impl.cc:773] [C1] immediate connection error: 101
It seems that auth0 provides IPv6 endpoints now. Which is, generally, awesome, but this cluster has no IPv6 connectivity and thus envoy fails to fetch the JWKS, fails to validate the signature, and rejects the request. Makes me wonder why envoy never tries the IPv4 address.
The fix is trivial, add the following to your cluster config:
dns_lookup_family: V4_ONLY
Which will stop envoy from trying to use IPv6 lookups in that particular cluster.
Have a good day!