An Israeli airspace-security company runs drone-detection sensors from different vendors, each with its own protocol, login and data model. I led the design and build of a hub that ingests all of them, turns every detection into one event format, and streams it live to the operators' dashboard and to national drone-traffic platforms, each in the format it expects.
The company deploys detection sensors from more than one vendor. One is an RF sensor network that publishes threats through GraphQL subscriptions over WebSocket, with a separate details stream per active threat. The other pushes drone and Remote-ID tracks over its own WebSocket protocol with token-based sessions. Each has its own login and token-renewal rules, its own failure modes and its own idea of what a “detection” is.
On the other side, the operators' dashboard, a national drone-traffic (UTM) platform and a U-space service each expect a different payload, and some want a live stream while others want HTTPS pushes. Wiring every sensor directly to every consumer would mean N × M fragile integrations, each with its own reconnect logic.
The diagram is wider than the screen: swipe sideways to see all of it →
Both services run as containers. The ingest service only knows sensors; the distribution service only knows consumers. The queue between them means a slow or offline consumer never back-pressures a sensor connection, and either side can be redeployed without dropping the other.
Sensor connections are long-lived and expensive to re-establish; consumers come and go. SQS decouples the two, absorbs bursts and lets each side scale and deploy on its own. Trade-off: one more hop and at-least-once delivery, which the event IDs make visible.
Every detection becomes the same record (position, altitude, frequency, level, classification) with a typed, vendor-specific extension. Each consumer gets a mapper discovered at start-up and registered by key. Trade-off: the shared model is a common denominator, so vendor detail lives in the extension rather than the core.
Active providers and consumers, their transport (WebSocket or HTTPS) and mapper type live in a DynamoDB registry. Credentials and API keys live in Secrets Manager. Adding a consumer is a row and a secret, not a release. Trade-off: the registry is read at start-up, so a change takes a restart.
Each connection gets its own bounded channel. If a client falls behind, it loses the oldest batches instead of slowing every other client. For a live air picture, fresh beats complete. Trade-off: this path is not an audit log; v1 deliberately keeps no event store.
Most of the real work showed up only once real sensors were connected. A selection:
When nothing was detected for a while, the load balancer closed idle WebSocket connections and clients saw a dead feed.
The send loop waits for data with a 30-second timeout and sends a keep-alive frame when it expires, well inside the load balancer's idle window.
The RF network publishes the list of active threats on one subscription and the details on another, which must be re-issued with the current ID set every time that set changes.
Modelled the active IDs as an observable, debounced changes by 50 ms and used Switch so each new ID set tears down the old details subscription cleanly.
Under real sensor load the distribution side fell behind: each message was received, written to every socket and deleted one at a time, and the backlog reached about 186k.
Raised poller concurrency, deleted in batches of up to 10, and handed each received batch to an in-memory per-client channel so it goes out as one WebSocket frame. The backlog drained and stayed at zero.
Token refresh against one vendor kept failing in odd ways. The vendor returns HTTP 200 even when the call fails, with the real result in the body.
Check the body's error code, not just the status. If renewal fails, drop the session and log in again under a lock, so concurrent callers don't trigger several logins.
Exceptions were logged and swallowed inside the stream handlers. The worker looked healthy, but no events were arriving.
Errors now fault the stream, which drops the worker into its reconnect loop with a fixed back-off (3 s and 5 s per vendor). The browser client reconnects with exponential back-off from 1 s up to 15 s.
A U-space platform accepts events only as authenticated HTTPS posts, while the others subscribe to the stream.
Added a transport field to the consumer registry and a publisher that fans each batch out to every HTTPS target in parallel, caches each target's token and re-authenticates once on 401. One failing target can't block the others.
After token expiry, the GraphQL server reported authentication failures as an error payload on the details subscription. Nothing threw, so the worker never logged in again.
Detect the authentication error code in the payload and fault the stream deliberately, which triggers a full re-login.
Calls needed to take one batch of 10 events off the queue and deliver it to one connected client. Derived from the code before and after the May 31 rework, not a benchmark.
Free scoping call · 30 minutes · No commitment
Write a few lines: the process or system, what hurts today (time, errors, cost or an AI pilot that stalled) and who uses it. After a short call you get a written plan and a fixed quote within ≈3 days.
Opens a pre-filled email · reply within one business day
The model reads. The code counts.
All projects