Skip to main content

WebSocket Routing in Production

The actual architecture​

Production runs this app as a Docker container on a GCE VM (confirmed directly from .github/workflows/{staging,production}-deployment.yaml — SSH onto the instance, docker run the image), fronted by nginx as a reverse proxy on that same VM. This is not Cloud Run, not GKE, and not App Engine — an older internal note speculated across all three before landing on the real answer, which is worth knowing if you find that note first: the nginx-on-GCE-VM architecture below is the one actually in use.

Why a new WebSocket endpoint needs nginx changes, not just code​

The backend registering a new WebSocket path (via the central router — see Architecture Overview) is necessary but not sufficient in production. nginx only proxies WebSocket upgrade requests (Connection: Upgrade, Upgrade: websocket headers) for paths that have an explicit location block configured to forward them — anything else falls through to the default location, which doesn't set the upgrade headers, and the connection fails (typically as a 404 or a silent timeout) even though the exact same code works fine locally, where there's no nginx in front of it at all.

This has happened at least once already: a second WebSocket endpoint (device) was added to the backend, initialized correctly, and worked in local dev — but nginx was never updated to add its location block, so it silently 404'd in production while a sibling endpoint (chat), added earlier with its own nginx block, kept working. Same code, same router, different nginx config — that's the tell if you see one WebSocket path work and a newer one not.

Adding a new WebSocket endpoint — the nginx block​

# [FEATURE] WebSocket endpoint - exact match, placed BEFORE the default location block
location = /v1/[path]/ws {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;

    # Upgrade headers - without these, this is indistinguishable from a broken proxy
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering off;
    proxy_cache off;
    proxy_request_buffering off;

    # Long timeouts - these are meant to be long-lived connections
    proxy_connect_timeout 7d;
    proxy_send_timeout 7d;
    proxy_read_timeout 7d;
}
sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/default.backup.$(date +%Y%m%d_%H%M%S)
# edit the file, add the block above
sudo nginx -t          # validate before reloading
sudo systemctl reload nginx

Checklist for a new WebSocket endpoint​

  • Gateway class + init function created, registered in src/app.ts
  • Path added to the central router (src/modules/websocket/websocket-router.ts)
  • nginx location block added — this is the step that's been missed before; don't assume backend code alone is enough
  • sudo nginx -t validated, sudo systemctl reload nginx applied on the actual production VM
  • Tested from an external machine with wscat, not just locally

Verifying a deployment picked up new WebSocket code​

# Confirm the router initialized
gcloud logging read "textPayload:WebSocket router setup" --limit 5 --format=json

# Watch upgrade requests arrive in real time while you connect
gcloud logging tail "textPayload:WebSocket Router OR textPayload:[YOUR-FEATURE] WS"

# Check specifically for errors
gcloud logging read "severity=ERROR AND textPayload:WebSocket" --limit 20 --format=json

If you see old-style log lines (e.g. a standalone "WebSocket server listening on path" message without the central router's own log lines around it), the deployment hasn't actually picked up the newer routing code — don't assume nginx is the problem before ruling this out.

Testing an upgrade directly​

curl -i -N \
  -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: test" \
  https://<host>/v1/<path>/ws

A working endpoint returns 101 Switching Protocols. A 404 means nginx has no matching location block (or the path is wrong); anything else (502/503) points at the app itself rather than nginx.

  • src/modules/websocket/websocket-router.ts — the central router all WebSocket paths register with
  • src/modules/chat/chat-websocket-init.ts, src/modules/device/device-websocket-init.ts — the two existing gateways