added: self-reported LAN endpoints for peers behind the same NAT #1

Merged

jak merged 6 commits from lan-endpoints into main 2026-11-07 00:33:21 -05:00

Conversation 0 Commits 6 Files changed 32 +1340 -441

jak commented 2026-10-20 17:21:35 -04:00

Owner

Copy link

Problem

Two peers behind the same router never learned an address that works between them. Every address they heard about came from someone outside their LAN, and outsiders only ever see the router's address, never a peer's LAN address. In the original scenario:

• The cord server saw the router's internal address, because it sits behind the same router.
• An outside VPS saw the router's public address.

Neither works if the router doesn't loop traffic between two of its own LAN devices back through itself, which many routers don't.

How it works

1. Each client reports its own LAN address

Every time a client reports the addresses it has seen other peers at, it also reports itself. Device.GetLocalEndpoint provides the address: it asks the OS which local address it would use to reach the server, then pairs that with the port WireGuard is actually listening on, including a port the OS picked.

Finding the address sends no packets and does no DNS lookup, because the server's address is already resolved. netaddr.ReachableFromLAN throws away addresses no LAN neighbor could use: loopback, link-local, multicast, and addresses inside the cord network itself.

2. The server stores it like any other sighting

A self-report is a normal row in the server's endpoint table. Peer B is listed as the witness of its own address. The table keeps one row per witness and peer, so each new self-report replaces the last one. The server's code for receiving reports didn't need to change.

3. The server shares it only with peers on the same network

When peer A asks the server for its peer list, the server checks whether A and B seem to be behind the same router. That's true when some third peer has seen both A and B coming from the same IP address in the last 15 minutes.

• If they match, A gets B's self-reported LAN address.
• If not, A gets every other address for B as usual; only the LAN address is held back.

A few details:

• Self-reports don't count toward the match. Only what other peers actually saw counts.
• Old self-reports aren't shared. One older than 15 minutes is held back too.
• The server is always a third peer. Every client keeps a connection to it, so there's always someone who has seen both A and B.
• Wire format: the peer list marks these addresses with a self_reported flag. Older clients ignore it.

How the check behaves in practice:

Situation	Does A get B's LAN address?
A and B behind the same router (including the original scenario)	Yes
A and B behind different routers	No
A and B seen at the same IP, but by different peers	No
A recently left the shared network, but an old sighting still shows the shared IP	No, because sightings older than 15 minutes don't count
A and B at different sites that happen to share one public IP, as with some mobile carriers	Yes, by mistake. A wastes one 90-second connection attempt
A and B behind a router that uses a different public IP for each connection	No match, so nothing changes from main

4. The client tries the most trustworthy addresses first

For each address it knows, the client stores how confident it is that the address works, and when it last had that evidence. There are three levels:

Level	Meaning
1, witnessed	another peer saw the peer at this address
2, self-reported	the peer reported it, and the server judged us to be on the same network
3, connected	this client has actually connected through it

When new information about a known address arrives:

• Stronger evidence replaces the level and time.
• Equal evidence updates the time.
• Weaker evidence is ignored.

The client tries addresses highest level first, newest first within a level. So a LAN address from the same network is tried right after addresses that have already worked.

In the code, PeerEndpoint is split in two: EndpointObservation is what gets saved, and EndpointCandidate is what the client reads back when choosing an address to try. The store methods are renamed to match.

Commits

1. added: self-reported LAN endpoints for peers behind the same NAT: the feature as first drafted.
2. refactor: folded the LAN endpoint lookup into the network report: removes lan.go and adds netaddr.ReachableFromLAN.
3. refactor: asked the WireGuard device for its LAN endpoint: replaces the runtime.Options.LocalAddr function with a device method, named GetLocalEndpoint as of commit 6. runtime.go and the test setup files are back to matching main.
4. changed: ranked client endpoints by confidence: replaces two timestamps and a flag with the confidence levels.
5. refactor: grouped the same-site helpers after the snapshot code: moves code within topology.go without changing it.
6. refactor: tidied the LAN endpoint change after review: moves the last inline SQL out of a transaction, gives functions verb names, simplifies the client migration, and drops two internal tests.

Commit 3 removes code that commit 1 added. Reviewing them together, or reviewing the whole diff, saves reading code that no longer exists.

Worth a look in review

• Upgrade servers before clients. An older server accepts self-reports from new clients but shares them with every peer that can see the sender, without the same-network check.
• Connected addresses expire. If a day passes without a successful connection through an address, the client drops it. If the server still mentions it, it comes back as "witnessed". To make that work, syncing with the server now deletes expired addresses before saving the new list.
• Client database migration. Client schema v2 recreates the endpoint table empty. The table is a cache that the client refills from the server on its first sync, which runs as soon as a network starts. Upgraded clients briefly forget which addresses had worked and relearn that within about a minute.
• Existing gap, now more common. Once A connects to B over the LAN, A's regular report says "B is at 192.168.0.3". The server passes that to everyone who can see B, with no same-network check. main already does this, but this PR makes it happen more often. The obvious fix breaks setups where the server itself is on the LAN, so I've left it for a follow-up.
• Likely conflicts with macos-support. That branch also edits internal/client/runtime/network.go, runtime.go and internal/wireguard/backend_userspace.go.

Testing

• make lint and make test-full pass.
• Server: a table test of the same-network check, the self_reported flag in the peer list, and storing a peer's report of itself with the peer as witness. Turning the check off makes all five "No" cases fail.
• Client storage: the evidence rules, try order, the starting address for a new peer, and cleanup, plus an expired connection coming back as witnessed. Deliberately breaking the evidence rules or the cleanup order makes at least one test fail.
• Client reporting: the client reports its LAN address with an OS-chosen or configured port, over IPv4 and IPv6. It reports nothing when it has no route, or when the address is loopback or inside the cord network.
• WireGuard: Device.GetLocalEndpoint with the mock backend, and parsing the listen port from the userspace backend's status.
• Flaky test on main: one race-detector run failed in TestConfirmInstall_ClosesTemporaryTunnelBeforePublishingNetwork, where the test's cleanup changes the mock while the client is still closing it. It fails the same way on unmodified main, once in 20 runs. The rerun passed.
• Not tested:
  • GetLocalEndpoint on the real kernel and userspace backends hasn't been run, since both need root. The OS address lookup inside it was checked by hand on a machine with several network interfaces, where it picked the Ethernet address.
  • The original three-machine scenario hasn't been tried on a real network.

🤖 Generated with Claude Code

## Problem Two peers behind the same router never learned an address that works between them. Every address they heard about came from someone outside their LAN, and outsiders only ever see the router's address, never a peer's LAN address. In the original scenario: - **The cord server** saw the router's internal address, because it sits behind the same router. - **An outside VPS** saw the router's public address. Neither works if the router doesn't loop traffic between two of its own LAN devices back through itself, which many routers don't. ## How it works ### 1. Each client reports its own LAN address Every time a client reports the addresses it has seen other peers at, it also reports itself. `Device.GetLocalEndpoint` provides the address: it asks the OS which local address it would use to reach the server, then pairs that with the port WireGuard is actually listening on, including a port the OS picked. Finding the address sends no packets and does no DNS lookup, because the server's address is already resolved. `netaddr.ReachableFromLAN` throws away addresses no LAN neighbor could use: loopback, link-local, multicast, and addresses inside the cord network itself. ### 2. The server stores it like any other sighting A self-report is a normal row in the server's `endpoint` table. Peer B is listed as the witness of its own address. The table keeps one row per witness and peer, so each new self-report replaces the last one. The server's code for receiving reports didn't need to change. ### 3. The server shares it only with peers on the same network When peer A asks the server for its peer list, the server checks whether A and B seem to be behind the same router. That's true when some third peer has seen both A and B coming from the same IP address in the last 15 minutes. - **If they match**, A gets B's self-reported LAN address. - **If not**, A gets every other address for B as usual; only the LAN address is held back. A few details: - **Self-reports don't count toward the match.** Only what other peers actually saw counts. - **Old self-reports aren't shared.** One older than 15 minutes is held back too. - **The server is always a third peer.** Every client keeps a connection to it, so there's always someone who has seen both A and B. - **Wire format:** the peer list marks these addresses with a `self_reported` flag. Older clients ignore it. How the check behaves in practice: | Situation | Does A get B's LAN address? | |---|---| | A and B behind the same router (including the original scenario) | Yes | | A and B behind different routers | No | | A and B seen at the same IP, but by different peers | No | | A recently left the shared network, but an old sighting still shows the shared IP | No, because sightings older than 15 minutes don't count | | A and B at different sites that happen to share one public IP, as with some mobile carriers | Yes, by mistake. A wastes one 90-second connection attempt | | A and B behind a router that uses a different public IP for each connection | No match, so nothing changes from `main` | ### 4. The client tries the most trustworthy addresses first For each address it knows, the client stores how confident it is that the address works, and when it last had that evidence. There are three levels: | Level | Meaning | |---|---| | 1, witnessed | another peer saw the peer at this address | | 2, self-reported | the peer reported it, and the server judged us to be on the same network | | 3, connected | this client has actually connected through it | When new information about a known address arrives: - **Stronger evidence** replaces the level and time. - **Equal evidence** updates the time. - **Weaker evidence** is ignored. The client tries addresses highest level first, newest first within a level. So a LAN address from the same network is tried right after addresses that have already worked. In the code, `PeerEndpoint` is split in two: `EndpointObservation` is what gets saved, and `EndpointCandidate` is what the client reads back when choosing an address to try. The store methods are renamed to match. ## Commits 1. `added: self-reported LAN endpoints for peers behind the same NAT`: the feature as first drafted. 2. `refactor: folded the LAN endpoint lookup into the network report`: removes `lan.go` and adds `netaddr.ReachableFromLAN`. 3. `refactor: asked the WireGuard device for its LAN endpoint`: replaces the `runtime.Options.LocalAddr` function with a device method, named `GetLocalEndpoint` as of commit 6. `runtime.go` and the test setup files are back to matching `main`. 4. `changed: ranked client endpoints by confidence`: replaces two timestamps and a flag with the confidence levels. 5. `refactor: grouped the same-site helpers after the snapshot code`: moves code within `topology.go` without changing it. 6. `refactor: tidied the LAN endpoint change after review`: moves the last inline SQL out of a transaction, gives functions verb names, simplifies the client migration, and drops two internal tests. Commit 3 removes code that commit 1 added. Reviewing them together, or reviewing the whole diff, saves reading code that no longer exists. ## Worth a look in review - **Upgrade servers before clients.** An older server accepts self-reports from new clients but shares them with every peer that can see the sender, without the same-network check. - **Connected addresses expire.** If a day passes without a successful connection through an address, the client drops it. If the server still mentions it, it comes back as "witnessed". To make that work, syncing with the server now deletes expired addresses before saving the new list. - **Client database migration.** Client schema v2 recreates the endpoint table empty. The table is a cache that the client refills from the server on its first sync, which runs as soon as a network starts. Upgraded clients briefly forget which addresses had worked and relearn that within about a minute. - **Existing gap, now more common.** Once A connects to B over the LAN, A's regular report says "B is at 192.168.0.3". The server passes that to everyone who can see B, with no same-network check. `main` already does this, but this PR makes it happen more often. The obvious fix breaks setups where the server itself is on the LAN, so I've left it for a follow-up. - **Likely conflicts with `macos-support`.** That branch also edits `internal/client/runtime/network.go`, `runtime.go` and `internal/wireguard/backend_userspace.go`. ## Testing - `make lint` and `make test-full` pass. - **Server:** a table test of the same-network check, the `self_reported` flag in the peer list, and storing a peer's report of itself with the peer as witness. Turning the check off makes all five "No" cases fail. - **Client storage:** the evidence rules, try order, the starting address for a new peer, and cleanup, plus an expired connection coming back as witnessed. Deliberately breaking the evidence rules or the cleanup order makes at least one test fail. - **Client reporting:** the client reports its LAN address with an OS-chosen or configured port, over IPv4 and IPv6. It reports nothing when it has no route, or when the address is loopback or inside the cord network. - **WireGuard:** `Device.GetLocalEndpoint` with the mock backend, and parsing the listen port from the userspace backend's status. - **Flaky test on `main`:** one race-detector run failed in `TestConfirmInstall_ClosesTemporaryTunnelBeforePublishingNetwork`, where the test's cleanup changes the mock while the client is still closing it. It fails the same way on unmodified `main`, once in 20 runs. The rerun passed. - **Not tested:** - `GetLocalEndpoint` on the real kernel and userspace backends hasn't been run, since both need root. The OS address lookup inside it was checked by hand on a machine with several network interfaces, where it picked the Ethernet address. - The original three-machine scenario hasn't been tried on a real network. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jak added 1 commit 2026-10-20 17:21:35 -04:00

added: self-reported LAN endpoints for peers behind the same NAT ... 032cef6244

Two peers behind one NAT could not reach each other. Every endpoint they
learned came from a witness outside the LAN, which only ever saw the NAT's
address, and many routers do not hairpin those mappings. No witness can
observe a peer's LAN address, so peers now report it themselves.

Each client reports a sighting of itself on its regular report. The address
is the one the host routes toward the coordination server from, and the
port is the WireGuard device's live listen port, so an OS-chosen port is
reported correctly. Loopback, link-local, and overlay addresses are never
reported. We reused the existing witness table: a self-report is a row
whose witness is the peer itself, and each new one replaces the last.

The server shares a self-report only with peers on the same site, meaning
one other witness saw both peers from the same IP within 15 minutes. The
window keeps a peer that moved away from matching on day-old sightings. The
rule only withholds self-reports, so no peer loses a candidate it had
before. Snapshot endpoints now carry 'self_reported', and clients rank
those endpoints after locally proven ones and ahead of everything else, so
a LAN endpoint is the first unproven candidate tried.

Client databases migrate to schema version 2 to store the mark. Servers
should be upgraded before clients: an older server stores self-reports but
hands them to every visible peer without the same-site check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jak added 3 commits 2026-11-05 00:08:33 -05:00

refactor: folded the LAN endpoint lookup into the network report ... 9705591697

The lookup lived in its own 'lan.go' with a private error type, two free
functions, and a method on 'Network'. We moved the address check into
'netaddr.ReachableFromLAN', where the other pure address helpers live and
where a table test covers it directly. The self-sighting now sits beside
the report that sends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

refactor: asked the WireGuard device for its LAN endpoint ... 84c888c737

The runtime took a 'LocalAddr' function in its options and passed it
through every network, only so tests could replace the host's route
lookup. Which local address the device sends from is a question about
the device's own socket, and the WireGuard backend is already where cord
meets the host. 'Device.LocalEndpoint' now answers it for a peer, pairing
the route toward that peer's endpoint with the live listen port.

The device looks up the server peer's endpoint it already resolved, so
the lookup does no DNS and needs no timeout. Tests set 'LocalAddr' on the
mock backend, and an unset address reports no route, so the existing
harnesses needed no changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

changed: ranked client endpoints by confidence ... 9862f1a107

Each endpoint row kept a server time, a local time, and a self-reported
flag. No Go code read them. They existed so SQL could sort and prune, and
every reader had to combine them to learn how sure the client was that an
endpoint worked.

We replaced them with one confidence level and the time of that evidence.
The levels are witnessed, self-reported, and connected. A stronger
observation replaces an endpoint's level and time, an equal one advances
the time, and a weaker one is ignored. 'PeerEndpoint' split into
'EndpointObservation' for writes and 'EndpointCandidate' for the ranked
list rotation reads, and the store methods were renamed to match.

A connected endpoint now falls back to witnessed once a day passes
without a handshake through it. Synchronization prunes before it merges
the snapshot, so the server's copy comes back as witnessed rather than
being dropped.

Client schema version 2 has not shipped, so we rewrote it as a table
rebuild. Endpoints with a local observation migrate as connected at that
time, and the rest as witnessed at their server time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jak added 1 commit 2026-11-05 00:12:02 -05:00

refactor: grouped the same-site helpers after the snapshot code ... faef7885ea

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jak added 1 commit 2026-11-07 00:24:03 -05:00

refactor: tidied the LAN endpoint change after review ... 25ed1d6a2a

'RecordConnectedEndpoint' ran its peer lookup inline. It now calls a
'sqlGetPeerIDTx' helper, so the transaction is only a list of helper calls.

Functions that do work now have verb names: 'witnessSelf',
'findRouteSource', 'collectSiteAddrs', and 'Device.GetLocalEndpoint',
which matches 'Device.GetPeerStatuses'. Some signatures and calls were
split one argument per line, with blank lines between steps.

We dropped both new internal tests. Client schema version 2 recreates the
endpoint table empty. The table is a cache that the first sync refills,
so converting old rows bought little and needed a white-box test. The
route lookup test covered a thin wrapper around one standard library
call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jak merged commit ef930fcb25 into main 2026-11-07 00:33:21 -05:00

jak deleted branch lan-endpoints 2026-11-07 00:33:21 -05:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

pollinator/cord!1

No description provided.