Monitoring and troubleshooting

Endpoints

The bridge listens on port 8092 and accepts nothing over HTTP.

The container ships a HEALTHCHECK against /healthz, so docker ps reports the same state without publishing the port at all.

What to alert on

Two gauges separate “quiet because nothing is happening” from “quiet because I am broken”, and they are the ones worth a Prometheus rule:

• wxmesh_feed_connected: 0 means the broker no longer has us.
• wxalerts_seconds_since_feed_message: climbing past a few hours means the subscription is wrong even though the connection is fine. -1 means nothing has ever arrived.

wxmesh_meshmonitor_reachable is the third worth watching, since a bridge that is reading the feed perfectly and cannot reach MeshMonitor is still silent.

The rest are counters: wxmesh_feed_messages_total, wxmesh_feed_tombstones_total, wxmesh_feed_reconnects_total, wxmesh_matched_total, wxmesh_unmatched_total, wxmesh_suppressed_duplicate_total, wxmesh_suppressed_interval_total, wxmesh_sent_total, wxmesh_send_failed_total, wxmesh_tx_disabled_total, wxmesh_queue_dropped_stale_total, wxmesh_bytes_sent_total, plus wxmesh_live_alerts, wxmesh_queue_depth and wxmesh_airtime_tokens as gauges.

When someone reports a missing alert

Check /live first. It answers the only question that matters: did the bridge ever know about the hazard.

• The hazard is there. A rule or the airtime budget dropped it, and /stats says which: unmatched, suppressed_duplicate, suppressed_interval, suppressed_other_county, or queue_dropped_stale.
• The hazard is not there. The subscription is wrong: the wrong county, or a marine zone with no county behind it, or a subscription that was refused.

Troubleshooting

Nothing is ever sent. Check dry_run. Then check that a rule matches, so set log_level: DEBUG and every unmatched alert is logged with its codes.

the API token cannot write to channel N. The token’s user needs write on channel_N in MeshMonitor’s permissions, or point the rule at another channel.

transmit is disabled on the MeshMonitor source. MeshMonitor returned 409, usually because its own airtime cutoff has engaged. Messages are dropped, never held for later, because a backlog that lands all at once when transmit returns is worse than the gap.

no channel named 'WX'. The bridge logs the channels MeshMonitor does have. Names match ignoring case and a leading hash, so WX, wx and #wx are the same channel; WX and WEATHER are not.

MeshMonitor returned no channels. Almost always the token: an under-privileged caller gets 200 [] rather than a 401. The bridge retries every 60 seconds in the background, so fixing the permission takes effect without a restart.

Alerts stopped after a network blip. Reconnects back off from 1 second to a ceiling of 2 minutes. Every reconnect replays the full retained set, so a hot reconnect loop is a bandwidth amplifier pointed at a broker other people depend on. The backoff is a requirement, not politeness. wxmesh_feed_reconnects_total climbing steadily is the tell.

An office subscription is louder than expected. It carries every alert that office issues, for every county in its forecast area. office_alerts: unzoned keeps only the alerts with no county behind them, which is the reason to subscribe to an office at all.

MeshMonitor compatibility

Verified against MeshMonitor 4.14.0. Notes for anyone reading MeshMonitor’s own API reference alongside this:

• The published reference documents toNodeId and channelIndex on the send route. The Meshtastic route actually reads destination and channel, and a request built from the docs is accepted with the channel silently defaulted to 0, because it transmits on the wrong channel rather than failing.
• A MeshCore source is a different endpoint entirely: POST /api/sources/{id}/meshcore/messages/send, with channelIdx. The source type is detected from /api/v1/sources and the right one is used.
• API tokens authenticate on /api/v1/*. On 4.14.0 the legacy /api/channels answered 200 [] for a token that /api/v1/sources/{id}/channels served in full, so channels are read through v1.
• MeshCore returns 200 {"success": false} when the device refuses a frame, so the HTTP status alone is not proof of a send.

The feed itself

Public, read-only, no signup. The credential is published because the broker enforces that it cannot publish.

URL       wss://mqtt.wxalerts.org/mqtt
Username  wxalerts
Password  wxalerts
Topics    wxalerts/nws/v1/same/{same}/{etn}              retained, QoS 1
          wxalerts/nws/v1/alert/{office}/{p}/{s}/{etn}

# Watch a county directly
mosquitto_sub -L "wss://wxalerts:wxalerts@mqtt.wxalerts.org:443/mqtt" \
  -t 'wxalerts/nws/v1/same/012113/#' -v

Office subscriptions deliberately use +/+/+ rather than #: raw text products are filed under the office as alert/{office}/Messages/{hash}, and # would drag the entire NWS product feed onto a LoRa bridge.