meshvpn

Help

Troubleshooting

Start with sudo meshvpn doctor: it checks the daemon, Tailscale conflicts, firewalls, peers, direct paths, SSH logins, admins and updates, and says how to fix each problem. For one node that only connects through relays: meshvpn net why <node>. The rest of this page covers what people run into most.

SSH logins

Permission denied (publickey) when logging in to another node

Password-less logins use the SSH key of your account on the machine you connect from. Each node publishes its users' public keys (~/.ssh/id_ed25519.pub, id_ecdsa or id_rsa), and the target lets in the keys its owner allowed. If your account has no key there, there is nothing to allow. Create one on the machine you connect from:

$ ssh-keygen -t ed25519

Within about a minute your account shows up on the target in sudo meshvpn ssh (the permission matrix) as you@yourmachine. If "everyone" or your node is ticked there, the login works. meshvpn ssh list on your machine shows which keys it offers to others.

My account doesn't appear in the matrix, although I have a key

Keys are read from the accounts in /etc/passwd. Accounts from LDAP or SSSD aren't there, and on homes on NFS root may not be able to read ~/.ssh. A key with a passphrase and no .pub file next to it is skipped too. Allow the key directly on the target instead, independent of what your machine publishes:

$ cat ~/.ssh/id_ed25519.pub                                   # on your machine
$ sudo meshvpn ssh allow ssh-ed25519 AAAA... --as alice       # on the target
The login asks for a password or goes to the wrong account

Name the account: ssh alice@server.mesh. The account used when you don't name one (console, desktop) is set on the target with sudo meshvpn ssh default-user alice. A permission is always for a local account (--as alice): being allowed in as one account doesn't let you in as another.

Joining and connections

A new node can't join, or a node on the internet can't reach the network

A new node joins by dialing the addresses in the invite. If the network sits behind a company firewall, a node outside can't reach it, but the network can reach that node. Start the connection from inside:

$ sudo meshvpn join <invite> --name edge --endpoint 203.0.113.7:7870   # on the public node
$ sudo meshvpn connect 203.0.113.7                                   # on any node of the network

Until then, meshvpn status on the public node says it can't reach the network and prints this command.

Two nodes only talk through a relay

That works, but it's slower. meshvpn net why <node> tests the path between the two and says what's in the way (NAT type, a blocked port, no IPv6) and what to do. A node others can reach (an open port, or a public address) helps every pair.

Home router: the port can't be opened

If the router has no public IPv4 address itself (DS-Lite, carrier-grade NAT, most mobile networks), no port can be opened on it; meshvpn status says so. IPv6 hole punching usually still connects. Two nodes that both have only IPv4 behind carrier-grade NAT need a third node with a public address, such as a small VPS. See Home router and 5G.

A node shows up twice after reinstalling the machine

The reinstalled machine has a new identity, and the old one stays listed as offline. Once the new node has taken the name, the old one is forgotten automatically after 10 minutes. Remove it at once with sudo meshvpn forget <name>, or all offline nodes with sudo meshvpn forget --offline.

"… wants to make this network managed" in meshvpn status

The network was created open (init --open, or before version 0.7), and a node ran sudo meshvpn admin enable: it wants to be the admin, so that only it can invite and ban. From 0.32.1 on no node takes that over by itself. If it's right, accept it on each machine:

$ sudo meshvpn admin trust hub

If you don't know why that node wants to be admin, don't confirm it: your machine stays open and keeps working. meshvpn admin status shows who asked and which admins it named.

"address conflict" in meshvpn status

Two nodes got the same mesh address (it's a hash of the node's key). From 0.31.1 on both move to another address by themselves and restart; update older nodes with sudo meshvpn update.

On the machine

Tailscale is installed too, and mesh traffic doesn't arrive

Tailscale uses the same address range (100.64.0.0/10), and its firewall drops every packet from that range that doesn't arrive on tailscale0, so all incoming mesh traffic disappears. meshvpn status and doctor warn about it. Run sudo tailscale down or uninstall Tailscale on that machine.

Windows: other nodes can't reach a service on the PC

Windows Defender Firewall blocks connections into the machine from the mesh, as from any new network. Allow the service's port for the mesh range, in an administrator PowerShell:

New-NetFirewallRule -DisplayName "meshvpn: my app" -Direction Inbound -Protocol TCP -LocalPort 8000 -RemoteAddress 100.64.0.0/10 -Action Allow
Windows: a command seems to hang

PowerShell ISE can't run interactive console programs, so questions (meshvpn leave) and ssh's password prompt never show up. Use Windows Terminal or a normal PowerShell window.

Behind a proxy: the install or updates fail

The installer finds the proxy in https_proxy/http_proxy, /etc/environment, /etc/profile.d, apt's and dnf's settings and GNOME's proxy setting, and saves it in /etc/meshvpn/proxy.env for updates. Proxy auto-configuration (PAC) isn't supported: set https_proxy then. Mesh connections themselves never go through the proxy.

In a container, programs can't reach other nodes

Without /dev/net/tun or NET_ADMIN, meshvpn runs in userspace mode (meshvpn status shows mode: userspace). Programs reach the mesh through the SOCKS5 proxy: ALL_PROXY=socks5h://127.0.0.1:1055 curl http://hub.mesh:8000. ssh user@node.mesh works without setup. Only TCP and ping work there, no UDP; for everything, give the container --cap-add NET_ADMIN --device /dev/net/tun.

GPUs and training

Several people share a GPU node and get in each other's way

Reserve GPUs before using them: meshvpn gpu reserve server -n 2 --for 8h, or let meshvpn launch --gpus 2 server -- python train.py reserve and release them. meshvpn gpu list shows who has which GPU until when. Reservations are advisory: a job must use the CUDA_VISIBLE_DEVICES it got, or it still sees every GPU. Agree in the team that no GPU job starts without a reservation; gpu list shows GPUs that are busy without one.

3D in the desktop without root

sudo meshvpn desktop setup --gpu installs VirtualGL for everyone. Without sudo it installs it for your account only (in ~/.local/share/meshvpn), from 0.32.0 on. The packed desktop works without root too:

$ meshvpn desktop setup --gpu
$ meshvpn desktop setup --desktop mate
$ vglrun blender                      # in the desktop session

On NVIDIA that works as long as your account may open /dev/nvidia0 (ls -l /dev/nvidia*; often the video group). Other GPUs (Mesa) also need the system's Xvfb, which only root can install; setup says so. Without root there's no ffmpeg, so the stream uses the built-in encoder.

Distributed training (NCCL) hangs at the start

Use eval $(meshvpn net env --master gpu1 tag:gpu) or meshvpn launch: they set NCCL_SOCKET_IFNAME and the other variables, and use the LAN only if every pair of nodes shares one. Mixed paths, some pairs over the LAN and some over the mesh, make NCCL hang.

Not listed? sudo meshvpn doctor, the reference, or an issue on GitHub. Security problems please privately, see Security.