On my tailnet I have a couple internal services running off various devices (glance on nuc:8020 or upsnap on pi:8090 for example), using a mostly vanilla tailnet setup. This means I have to memorize a mapping of port to service, which is fine for three or four services but becomes annoying with 10 or more, and I can't get https, since tailscale serve only runs on :443 which means I can only run one service.

Custom DNS Server

To solve the issue of memorizing port numbers, I ended up running a custom DNS server and reverse proxies on each service. I was using tailscale magic DNS, however that doesn't support wildcard subdomains, and so I had to spin up my own.

CoreDNS

I decided to use coredns to replace tailscale dns, since it seemed to be a simple, minimal DNS server. I roughly followed this guide, and have the simplified steps here, mainly for my own future reference.

# compose.yml
services:
  coredns:
    container_name: coredns
    image: coredns/coredns:1.14.1
    command: -conf /etc/coredns/Corefile -dns.port 53
    volumes:
      - ./coredns:/etc/coredns
    ports:
      - "53:53"
      - "53:53/udp"
    restart: unless-stopped
    healthcheck: { test: curl -f http://localhost:8080 }

I opted to use .tailnet as my custom top level domain, and setup the following Corefile (in ./coredns/Corefile)

# coredns/Corefile
.:53 {
	# cloudflare and google
	forward . 1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4
	log
	errors
	health :8080
}

tailnet:53 {
	file /etc/coredns/db.tailnet
	log
	errors
}

.:53 matches all DNS queries on port 53, forwards the query to cloudflare (or google if it fails), logs the request, logs errors, and sets up health monitoring (only required once).

tailnet:53 matches all .tailnet DNS queries on port 53 (higher specificity) and responds based on the db.tailnet zonefile, as well has logging and erroring.

$ORIGIN             tailnet.
$TTL                3600
@                   IN SOA dns.tailnet admin.vielle.dev. ( 
                        0000000001  ; serial
                        86400       ; refresh rate (24h)
                        7200        ; retry rate (2h)
                        3600000     ; expiry (1000h)
                        3600        ; default ttl
                    )

dns.tailnet.        IN A 100.67.138.93
dns.tailnet.        IN AAAA fd7a:115c:a1e0::8c37:8a5d

nuc.tailnet.        IN A 100.67.138.93
nuc.tailnet.        IN AAAA fd7a:115c:a1e0::8c37:8a5d
*.nuc.tailnet.      IN A 100.67.138.93
*.nuc.tailnet.      IN AAAA fd7a:115c:a1e0::8c37:8a5d

pi.tailnet.         IN A 100.84.64.24
pi.tailnet.         IN AAAA fd7a:115c:a1e0::2c01:4026
*.pi.tailnet.       IN A 100.84.64.24
*.pi.tailnet.       IN AAAA fd7a:115c:a1e0::2c01:4026

compuper.tailnet.   IN A 100.96.62.82
compuper.tailnet.   IN AAAA fd7a:115c:a1e0::2501:3e52
*.compuper.tailnet. IN A 100.96.62.82
*.compuper.tailnet. IN AAAA fd7a:115c:a1e0::2501:3e52

Tailscale DNS config

Opening the tailscale dashboard, I opened the DNS tab and

  • Selected "Override DNS Servers"

  • Removed the default list (cloudflare and google)

  • Added the IP of my dns.tailnet. server as the only global nameserver

  • Added tailnet, nuc.tailnet, pi.tailnet, and compuper.tailnet as search domains, so I can have short domains (like http://glance/ -> http://glance.nuc.tailnet/)

  • Disabled MagicDNS

Caddy config

I also created a caddy container in the compose file

services:
  caddy:
    container_name: caddy
    image: caddy:2.11-alpine
    volumes:
      - ./caddy:/etc/caddy
    ports:
      - "100.67.138.93:80:80"
      - "[fd7a:115c:a1e0::8c37:8a5d]:80:80"
      - "100.67.138.93:443:443"
      - "[fd7a:115c:a1e0::8c37:8a5d]:443:443"
    restart: unless-stopped
{
	auto_https disable_certs
	log default {
		output file log.json
	}
}

(tls) {
	tls /etc/caddy/nuc.tailnet.crt /etc/caddy/nuc.tailnet.key
}

# lander
nuc.tailnet, nuc {
	log
	import tls
	root * /etc/caddy/srv
	file_server
}

# https
glance.nuc.tailnet {
	log
	import tls
	reverse_proxy glance:8080
}

immich.nuc.tailnet {
	log
	import tls
	reverse_proxy immich:2283
}

copyparty.nuc.tailnet {
	log
	import tls
	reverse_proxy host.docker.internal:8000
}

# http only -- not all apps support system certs
glance.nuc.tailnet:80, glance.nuc:80, glance:80 {
	log
	reverse_proxy glance:8080
}

immich.nuc.tailnet:80, immich.nuc:80, immich:80 {
	log
	reverse_proxy immich:2283
}

copyparty.nuc.tailnet:80, copyparty.nuc:80, copyparty:80 {
	log
	reverse_proxy host.docker.internal:8000
}

Custom TLS Certificates

While I can just generate some self-signed certs and add exeptions in my browsers, I wanted to get trusted certificates for the whimsy + SOME apps that dont support exceptions could still have support (but not all bc security or w/e 😒😒)

First, we create the configuration for the certificate authority and generate the certificate:

[req]
prompt = no
default_bits = 2048
distinguished_name = subject
x509_extensions = ext
req_extensions = ext

[subject]
C = GB
ST = Scotland
CN = tailnet

[ext]
keyUsage = keyCertSign
basicConstraints = CA:TRUE
openssl req -x509 \
    -newkey RSA:2048 \
    -keyout ca.key \
    -noenc \
    -out ca.crt \
    -days 3650 \
    -config ca.cfg

This should generate two files: ca.crt, which is used to trust the certificate, and ca.key which is used to sign other certificates. You can use openssl x509 -in ca.crt -noout -text to verify that it contains the proper fields

Next, we configure and generate the certificates for each device.

[req]
prompt = no
default_bits = 2048
distinguished_name = subject
x509_extensions = ext
req_extensios = ext

[subject]
C = GB
ST = Scotland
CN = machine.tailnet

[ext]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = "machine.tailnet"
DNS.2 = "machine"
DNS.3 = "*.machine.tailnet"
DNS.4 = "*.machine"
IP.1 = "100.67.138.93"

openssl req -x509 \                          
    -CA ca.crt \
    -CAkey ca.key \
    -newkey RSA:2048 \
    -keyout machine.tailnet.key \
    -noenc \
    -out machine.tailnet.crt \
    -days 3650 \
    -config machine.tailnet.cfg

This should generate the certificate and private key which will be used to encrypt HTTPS traffic. The certificate can be verified with openssl x509 -in machine.tailnet.crt -noout -text as before

Signing HTTPS requests

Assuming you use the same setup as I did above, you can simply copy machine.tailnet.crt and machine.tailnet.key into the same folder as the caddyfile and restart the caddy container.

Upon opening one of the services over https, you'll recieve a warning like this, as the root certificate is not yet trusted.

A firefox warning screen warning about "a potentially serious security issue" with the site's certificate. The error is SEC_ERROR_UNKNOWN_ISSUER.

Trusting the certificate

The process for trusting the certificate depends on the platform. On arch linux, you can run sudo trust anchor ./ca.crt to trust ca.crt. This will allow you to make HTTPS requests using most tools.

To trust the certificate in firefox desktop, go to about:preferences#privacy, click "Certificates" > "View Certificates" > "Authorities" > "Import" and select ca.crt. Trust the certificate for websites and click "Ok". Then, open one of the services in firefox and reload the page if needed. The SEC_ERROR_UNKNOWN_ISSUER should be gone.

To trust the certificate on android, send the .crt file to your phone and find the certificate settings (usually under Settings and Privacy). Install the .crt file and restart the device.

Drawbacks

Overall, makes deploying new services on existing devices much easier, but registering new devices becomes a little more finicky as they need to either generate and sign a new certificate and setup caddy, or install a new certificate authority.