Secure, Redundant DNS Proxy - Part 2

Mohammed Al-Sahaf

A few days ago I saw a post by Cloudflare about an incident impacting their DNS service 1.1.1.1 on June 27th1. HN and the quote tweets were people saying they noticed because their internet wasn’t stable due to DNS failures. Interestingly, I did not feel a blip even though I use Cloudflare in my DNS setup (see part 1). How come? The keyword is the redundancy in my setup. My DNS recurser uses 3 upstreams with random selection policy: Cloudflare, Google, Quad9. When Cloudflare DNS failed, CoreDNS would fallback to Google or Quad9. This prompted me to work on part 2 of my DNS setup which I’ve upgraded since the first post. Yes, the redundancy and failover benefit is already covered in part 1.

The setup described in part 1 works well for setting up DNS server within my own network or on the computer. Using your private DNS server on Android requires either pushing the address via DHCP, which limits where and how it can be used, using one of the DNS apps, which create an always-active VPN profile to capture DNS requests, or using the Private DNS feature in the network settings. I already have Caddy in my infrastructure, so I decided to venture into setting up DNS-over-HTTPS (DoH). CoreDNS enables DoH, leaving TLS termination for downstream, by prefixing the site address with with https://. In other words, the config posted in the part 1 turns into:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

.:53 {
	hosts /path/to/hosts {
		fallthrough
	}
	cache {
		prefetch 2 30m
	}
	forward . 127.0.0.1:5301 127.0.0.1:5302 [::1]:5301 [::1]:5302 127.0.0.1:5303 [::1]:5303 {
		policy random
	}
}
https://:5380 {
	hosts /path/to/hosts {
		fallthrough
	}
	cache {
		prefetch 2 30m
	}
	forward . 127.0.0.1:5301 127.0.0.1:5302 [::1]:5301 [::1]:5302 127.0.0.1:5303 [::1]:5303 {
		policy random
	}
}
.:5301 {
	forward . tls://8.8.8.8 tls://8.8.4.4 {
		tls_servername dns.google
	}
}
.:5302 {
	forward . tls://1.1.1.1 tls://1.0.0.1 {
		tls_servername cloudflare-dns.com
	}
}
.:5303 {
	cache
	forward . tls://9.9.9.9 {
		tls_servername dns.quad9.net
	}
}

Perfect, so I can add one more site to my existing Caddy server to terminate TLS, then plain HTTP reverse_proxy to the CoreDNS port which serves DoH. Happiness didn’t last. I quickly learned Android support for private DNS is limited to DNS-over-TLS (DoT). XDA-Developers published news on 2022-04-152 of Android 13 supporting DoH, but a blogpost by Google in July 193 of the same year proves it comes with hard-coded DoH support for Google and Cloudflare only. Bummer. Time to move forward.

DoT is simply the plaintext DNS message passed through TLS-encrypted channel4, meaning it has lower overhead than DoH. The difference between DoH and DoT is that DoH blends the DNS resolution through HTTPS on (typically) the port 443, while DoT operates on the standard port 853 and making it clear the connection is DoT request/response. I need Caddy to listen on port 853 for TLS connection, which Caddy terminates TLS and proxies the underlying data to upstream. This operation not being HTTP means I need the caddy-l4 app. You can find the instructions to build the custom Caddy executable with l4 support on the repository’s README and the Caddy documentation. Next is amending the configuration of CoreDNS and Caddy.

For CoreDNS, simply removing the block with the https:// address in the Corefile is enough. The bulk of the work to be done is on the Caddy front. We have to switch to using the JSON configuration format, so we start by adapt-ing the existing Caddyfile then adding the layer4 app configuration. Starting with this Caddyfile

1
2
3
4
5
6
7

{
	email user@example.com
}
example.com {
	root /var/www/public
	file_server
}

The adapted version is this beautiful JSON:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "example.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "vars",
                          "root": "/var/www/public"
                        },
                        {
                          "handler": "file_server",
                          "hide": [
                            "./Caddyfile"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [
              "example.com"
            ],
            "issuers": [
              {
                "email": "user@example.com",
                "module": "acme"
              },
              {
                "ca": "https://acme.zerossl.com/v2/DV90",
                "email": "user@example.com",
                "module": "acme"
              }
            ]
          }
        ]
      }
    }
  }
}

To configure the layer4 app to proxy DoT traffic, 2 things are to be added: the domain inside the app.tls.automation.policies[0].subjects array, and the layer4 configuration inside apps. The layer4 app listens on port 853, because that is the standard port for DoT, matches connections based on the SNI of the TLS request, and matching connections have their TLS terminated and the plaintext stream is proxied to CoreDNS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

*** from.json
--- to.json
***************
*** 47,53 ****
          "policies": [
            {
              "subjects": [
!               "example.com"
              ],
              "issuers": [
                {
--- 47,54 ----
          "policies": [
            {
              "subjects": [
!               "example.com",
!               "dns.example.com"
              ],
              "issuers": [
                {
***************
*** 58,63 ****
--- 59,101 ----
                  "ca": "https://acme.zerossl.com/v2/DV90",
                  "email": "user@example.com",
                  "module": "acme"
+               }
+             ]
+           }
+         ]
+       }
+     }
+   },
+   "layer4": {
+     "servers": {
+       "dot": {
+         "listen": [
+           ":853"
+         ],
+         "routes": [
+           {
+             "match": [
+               {
+                 "tls": {
+                   "sni": [
+                     "dns.example.com"
+                   ]
+                 }
+               }
+             ],
+             "handle": [
+               {
+                 "handler": "tls"
+               },
+               {
+                 "handler": "proxy",
+                 "upstreams": [
+                   {
+                     "dial": [
+                       "coredns:53"
+                     ]
+                   }
+                 ]
                }
              ]
            }

To make everyone’s life easier, here’s the final form of the full configuration:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "example.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "vars",
                          "root": "/var/www/public"
                        },
                        {
                          "handler": "file_server",
                          "hide": [
                            "./Caddyfile"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [
              "example.com",
              "dns.example.com"
            ],
            "issuers": [
              {
                "email": "user@example.com",
                "module": "acme"
              },
              {
                "ca": "https://acme.zerossl.com/v2/DV90",
                "email": "user@example.com",
                "module": "acme"
              }
            ]
          }
        ]
      }
    }
  },
  "layer4": {
    "servers": {
      "dot": {
        "listen": [
          ":853"
        ],
        "routes": [
          {
            "match": [
              {
                "tls": {
                  "sni": [
                    "dns.example.com"
                  ]
                }
              }
            ],
            "handle": [
              {
                "handler": "tls"
              },
              {
                "handler": "proxy",
                "upstreams": [
                  {
                    "dial": [
                      "coredns:53"
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    }
  }
}

This configuration serves a website on HTTPS (with automatic TLS) from a root directory and proxies all DoT traffic to CoreDNS at the same time.

Bon appetite.

  1. https://blog.cloudflare.com/cloudflare-1111-incident-on-june-27-2024 ↩︎

  2. https://www.xda-developers.com/android-13-dns-https-support/ ↩︎

  3. https://security.googleblog.com/2022/07/dns-over-http3-in-android.html ↩︎

  4. https://www.cloudflare.com/learning/dns/dns-over-tls/ ↩︎