Setting Up Your Own Mesh VPN Server with Headscale

Introduction: Why Your Own Mesh VPN?

Two servers, a laptop, and a media server at home... You want to access all of these securely from outside. The classic approach: port forwarding on each one separately, setting up SSH tunnels, maybe spinning up an OpenVPN/WireGuard server.

But what if all these devices could see each other directly? What if there was a secure, encrypted network between them? That's exactly the mesh VPN concept.

Tailscale is the most popular solution in this space — with a single command, all your devices connect to a virtual network. But Tailscale is closed-source and the coordination server runs in the cloud. All your traffic (control plane) is managed through Tailscale's third-party servers.

Headscale comes in at this point. As an open-source implementation of Tailscale, it uses the same protocol but runs on your own server. You have full control over your traffic. Like the name suggests: head (self-managed) + scale (scalable).

Mesh VPN Architecture: How Does It Work?

To understand mesh VPN, let's look at how it differs from classic VPNs:

Hub-and-Spoke (Traditional VPN):

• Each client connects to a central VPN server
• All traffic passes through this server
• Bottleneck: single server, all bandwidth
• Single point of failure

Mesh VPN (Headscale/Tailscale):

• Each node can establish direct WireGuard connections with every other node
• The coordination server is only used for key exchange and configuration
• Traffic goes directly peer-to-peer
• For devices behind NAT, DERP relay kicks in

Headscale Components

┌─────────────────────────────────────────────────────┐
│                 Headscale Server                     │
│  ┌─────────────┐  ┌──────────────┐  ┌───────────┐  │
│  │ Control API  │  │   DERP/STUN  │  │  SQLite   │  │
│  │ (HTTPS)      │  │   (UDP:3478) │  │ (or PG)   │  │
│  └─────────────┘  └──────────────┘  └───────────┘  │
│         │                                              │
│         ├── HTTPS (Noise protocol)                    │
│         │   ├── Node registration, authentication      │
│         │   ├── ACL policies                          │
│         │   └── DNS configuration                     │
│         │                                              │
│         └── DERP/STUN (UDP)                            │
│             ├── NAT traversal                          │
│             └── Relay if direct connection not possible │
└─────────────────────────────────────────────────────┘

┌──────────┐           WireGuard (UDP:51820)         ┌──────────┐
│ Node A   │◄──────────────────────────────────────►│ Node B   │
│ (Home)   │                                         │ (Cloud)  │
└──────────┘                                         └──────────┘

Communication Flow:

1. Each node connects to the Headscale server over HTTPS (authentication via Noise protocol)
2. Headscale distributes other nodes' public keys and IP addresses
3. Nodes establish direct WireGuard connections with each other
4. If they can't connect directly because they're behind NAT, they go through DERP relay

Installation: Setting Up Your Own Headscale Server

Option 1: Binary Installation (Recommended)

The simplest and most performant method. A single Go binary, no dependencies.

# Debian/Ubuntu
cd /usr/local/src
HEADSCALE_VERSION="0.24.1"
wget https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64.tar.gz
tar xzf headscale_${HEADSCALE_VERSION}_linux_amd64.tar.gz
cp headscale /usr/local/bin/
mkdir -p /etc/headscale /var/lib/headscale

# Verification
headscale version

# Arch Linux (AUR)
yay -S headscale
# or manually
cd /tmp
wget https://github.com/juanfont/headscale/releases/download/v0.24.1/headscale_0.24.1_linux_amd64.tar.gz
tar xzf headscale_0.24.1_linux_amd64.tar.gz
sudo install -m755 headscale /usr/local/bin/

Option 2: Docker Compose

If you want container isolation or already have Docker infrastructure:

# docker-compose.yml
version: '3.9'

services:
  headscale:
    image: headscale/headscale:0.24.1
    container_name: headscale
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"    # API (behind reverse proxy)
      - "3478:3478/udp"           # STUN/DERP
    volumes:
      - ./config:/etc/headscale
      - ./data:/var/lib/headscale
    command: headscale serve

Option 3: Package Manager

# Ubuntu/Debian (headscale's own repo)
echo "deb [signed-by=/usr/share/keyrings/headscale.gpg] https://deb.ory.sh/headscale stable main" | sudo tee /etc/apt/sources.list.d/headscale.list
wget -O- https://deb.ory.sh/headscale/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/headscale.gpg
sudo apt update && sudo apt install headscale

Configuration: config.yaml

The heart of Headscale is /etc/headscale/config.yaml. Here's a detailed explanation of each section:

# server_url: Address clients will use to connect to headscale
# This address must be accessible from clients
server_url: https://vpn.youraddress.com:443

# listen_addr: Address Headscale will listen on
# If running behind reverse proxy, use 127.0.0.1:8080
listen_addr: 127.0.0.1:8080

# metrics_listen_addr: Prometheus metrics (optional)
metrics_listen_addr: 127.0.0.1:9090

# DERP server: For NAT traversal
derp:
  server:
    # Enable internal DERP server
    enabled: true
    # Each region must have a unique ID (999 is fine for testing)
    region_id: 999
    region_code: "headscale"
    region_name: "Headscale DERP"
    # UDP port for STUN queries
    stun_listen_addr: "0.0.0.0:3478"
    # WebSocket relay path
    # Important: Your reverse proxy must also proxy this path
    relay_path: /derp

  # Public DERP maps
  # You can also use tailscale's public DERP servers
  urls:
    - https://controlplane.tailscale.com/derpmap/default

  auto_update_enabled: false

  # Map file for custom DERP servers
  paths: []

# DNS configuration
dns:
  # MagicDNS: Automatically assigns hostname to each node
  # e.g. node-name.vpn.youraddress.com
  magic_dns: true
  base_domain: "vpn.youraddress.com"
  nameservers:
    global:
      - 1.1.1.1
      - 8.8.8.8

  # Split DNS: Route specific domains to internal DNS
  split_dns:
    internal.company.com:
      - 192.168.1.53
    jenkins.internal:
      - 10.10.0.54

# Database
database:
  # sqlite: sufficient for small scale (10-20 nodes)
  # postgres: for large-scale deployments
  type: sqlite
  sqlite:
    path: /var/lib/headscale/db.sqlite

# ACL policy file
acl:
  policy_path: /etc/headscale/acl.hujson

# Log level
log:
  level: info

# Advanced settings
randomize_client_port: true

Client Connection

All clients need Tailscale CLI installed. After installation, you point them to your own headscale server:

# Debian/Ubuntu client installation
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list
sudo apt update && sudo apt install tailscale

# Arch Linux client
yay -S tailscale
sudo systemctl enable --now tailscaled

Connecting

# Method 1: Browser authentication
sudo tailscale up --login-server=https://vpn.youraddress.com

# Method 2: Pre-auth key (ideal for scripts)
sudo tailscale up --login-server=https://vpn.youraddress.com \
  --authkey=tskey-auth-nnnn-xxxxx

# Method 3: As a subnet router
sudo tailscale up --login-server=https://vpn.youraddress.com \
  --advertise-routes=192.168.1.0/24 \
  --accept-routes=true

Important: If using pre-auth key, you must first create a key on the headscale server:

headscale preauthkeys create --user admin --reusable --expiration 24h

Advanced Features

1. Traffic Control with ACL

ACLs are written in huJSON format (JSON with comment support). Here's a real scenario:

// /etc/headscale/acl.hujson
{
  // Groups: group users
  "groups": {
    "group:admin": ["[email protected]"],
    "group:dev":   ["[email protected]", "[email protected]"]
  },

  // Tags: tag servers
  "tagOwners": {
    "tag:server": ["group:admin"],
    "tag:database": ["group:admin"]
  },

  // Access rules
  "acls": [
    // Admins can access everything
    {"action": "accept", "src": ["group:admin"], "dst": ["*:*"]},

    // Dev team can only do SSH
    {"action": "accept", "src": ["group:dev"], "dst": ["tag:server:22"]},

    // Only admins can access the database
    {"action": "deny",  "src": ["*"], "dst": ["tag:database:5432", "tag:database:3306"]},

    // Everyone can ping each other
    {"action": "accept", "src": ["*"], "dst": ["*:icmp"]}
  ]
}

2. Subnet Routing

Not only devices with tailscale installed can be part of the mesh VPN, but also the entire network behind them:

# On the gateway machine: announce 192.168.1.0/24 network to the mesh
sudo tailscale up --login-server=https://vpn.youraddress.com \
  --advertise-routes=192.168.1.0/24 \
  --accept-routes=true

# On Headscale: approve the route
headscale routes list
headscale routes enable -r 1

# On other nodes: accept routes
sudo tailscale up --login-server=https://vpn.youraddress.com \
  --accept-routes=true

Now every device in the mesh can access all devices behind the gateway. Your printer at home, NAS at the office, database in the cloud... All of them appear to be on the same network.

3. Hostname Resolution with MagicDNS

When MagicDNS is active, each node is automatically assigned a DNS record in the format <hostname>.vpn.youraddress.com. You can access them with tailscale ping or ping node-name.

4. NAT Traversal with DERP Relay

If both devices are behind NAT (for example one at home and the other on mobile data), WireGuard may not be able to establish a direct connection. This is where DERP comes in:

Node A (Home NAT) -> STUN -> Headscale server
Node B (Mobile) -> STUN -> Headscale server
Headscale -> Derpmap -> Node A and Node B
Node A -> DERP Relay (Headscale) -> Node B (encrypted tunnel)

Traffic is still end-to-end encrypted. Headscale acts as a relay but cannot see the contents.

Reverse Proxy and SSL

It's recommended to run Headscale behind a reverse proxy instead of exposing it directly to the internet.

Caddy (Simple, Automatic SSL)

# /etc/caddy/Caddyfile
vpn.youraddress.com {
    reverse_proxy localhost:8080
}

Nginx (More Control)

# /etc/nginx/sites-available/headscale
server {
    listen 443 ssl http2;
    server_name vpn.youraddress.com;

    ssl_certificate /etc/letsencrypt/live/vpn.youraddress.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/vpn.youraddress.com/privkey.pem;

    # Headscale API
    location / {
        proxy_pass http://127.0.0.1:8080;

        # Required for WebSocket
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Long-lived connections
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
    }

    # DERP relay (WebSocket)
    location /derp {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    # Tailscale control protocol (Noise)
    location /ts2021 {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Note: The server_url in Headscale's config.yaml must be set as HTTPS: server_url: https://vpn.youraddress.com:443

Systemd Service

# /etc/systemd/system/headscale.service
[Unit]
Description=headscale control server
Documentation=https://headscale.net
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=headscale
Group=headscale
ExecStart=/usr/local/bin/headscale serve --config /etc/headscale/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=30

# Security hardening
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
CapabilityBoundingSet=
AmbientCapabilities=

[Install]
WantedBy=multi-user.target

Starting the service:

sudo groupadd --system headscale
sudo useradd --system --gid headscale --home-dir /var/lib/headscale headscale
sudo chown -R headscale:headscale /etc/headscale /var/lib/headscale
sudo systemctl daemon-reload
sudo systemctl enable --now headscale
sudo systemctl status headscale

Command Reference

User Management

Node Management

Route Management

Pre-auth Keys

Other

Backup and Disaster Recovery

The most critical file in Headscale is the SQLite database. It contains all node keys, ACLs, and user registrations. Losing this means rebuilding the entire network from scratch.

Backup Script

#!/bin/bash
# /usr/local/bin/headscale-backup.sh
BACKUP_DIR="/backup/headscale"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p "$BACKUP_DIR"

# Stop the service for a consistent backup
systemctl stop headscale

# Backup
tar czf "$BACKUP_DIR/headscale-full-$DATE.tar.gz" \
    /var/lib/headscale/db.sqlite \
    /etc/headscale/config.yaml \
    /etc/headscale/acl.hujson

# Start the service
systemctl start headscale

# Clean up backups older than 30 days
find "$BACKUP_DIR" -name "headscale-full-*" -mtime +30 -delete

echo "Backup complete: $BACKUP_DIR/headscale-full-$DATE.tar.gz"

Daily backup with cron:

0 3 * * * /usr/local/bin/headscale-backup.sh

Restoration

systemctl stop headscale
tar xzf /backup/headscale/headscale-full-20260115_120000.tar.gz -C /
systemctl start headscale
headscale nodes list  # Verification

Comparison: Headscale vs Alternatives

Real-World Use Cases

1. Homelab VPN

Bring all your home servers (Jellyfin, Home Assistant, NAS) together in one network. From outside, you only connect one device to headscale, and access everything else through the mesh.

2. Remote Development

Secure connections between cloud CI/CD servers, development environments, and staging servers. Everything works as if it's all on localhost.

3. Site-to-Site VPN

Permanent VPN between two offices using subnet routers. Access a printer in one office directly from the other.

4. IoT Device Management

Secure access to IoT devices behind NAT (Raspberry Pis, cameras, sensors). No port forwarding headache.

5. Privacy-First VPN

You want to use Tailscale but don't want all your traffic going through third-party servers. Same convenience with full control with Headscale.

Security Best Practices

1. Don't use HTTP: Headscale should always run behind a reverse proxy with HTTPS.
2. Use expiring pre-auth keys: Don't create keys with infinite validity.
3. Limit traffic with ACL: Don't allow everyone to access everything.
4. Regular backups: Back up your SQLite database and config files.
5. Expire unused nodes: Clean up nodes that are no longer in use.
6. Use OIDC: If possible, integrate with your own authentication system.
7. Open STUN port in firewall: Don't forget to allow UDP 3478 (if using it).
8. Monitor logs: Track suspicious connection attempts.

Conclusion

Headscale is a great project that lets you use Tailscale's power on your own server. Especially:

• Full data control
• Scalable mesh architecture
• WireGuard performance
• Easy setup and management
• Active community

If Tailscale's "free up to 20 devices" policy works for you and you don't mind your data being on a third party, you can use Tailscale directly. But if you want control and your data to stay with you, Headscale is the right choice.

Resources

• Headscale Documentation
• GitHub Repo
• Tailscale CLI
• Headscale Discord

*In the next article, I'll set up and manage Headscale in a production environment, covering every detail from SSH to MagicDNS live.*MARKDOWN_EOF