Operations and product teams at a large Israeli TV and streaming provider can now ask questions in plain English or Hebrew about tens of millions of playback and platform log records. The agent answers with numbers they can trust, and says so when the data can't support an answer.
The provider's observability pipeline writes every playback, API call and backend event to OpenSearch. There are seven log sources, each with its own schema, and hundreds of fields that are often sparse. Answering “which device had the most licence errors yesterday?” meant writing Query DSL by hand. It also meant knowing traps nobody had written down: which fields are empty, which values are constants, and how much of the traffic is automated monitoring.
The diagram is wider than the screen: swipe sideways to see all of it →
The runtime's IAM role is mapped as an OpenSearch fine-grained-access backend role, so there is no database password to store or leak. The policy names the staging domain explicitly, and that keeps the agent away from production in the same account.
The model picks a tool and fills typed arguments. What counts as a playback, an error or a session, and which fields can be grouped or returned, is enforced in Python. When the deployment had to switch models, every guarantee held.
Tools return fixed sentences (must_state, caveats_to_state) that name the exact window checked, what was excluded and how sparse a field is. Provenance never depends on the model's own paraphrase.
The client needed per-subscriber activity. The tool takes one exact ID, accepts no wildcards, returns a minimal field set and writes an audit line on every call. Search can never return identifiers under any arguments.
When coverage grew from 3 to 7 sources we considered adding an orchestrator layer. We rejected it. A single agent with explicit routing rules (viewing, platform, measurement) already routed correctly and stayed easier to debug.
Most of the value came from problems that only showed up with real data and real users. A selection:
A public Lambda URL, then a CloudFront front end, both failed on organization-level resource control policies that weren't visible from inside the account. The policies had to be discovered by experiment.
Moved the web UI to App Runner. A plain HTTPS request to it is not an AWS API call, so the policy doesn't apply.
The reseller-program account blocked the model family the agent was designed for.
Moved to Amazon Nova Pro behind a single MODEL_ID variable. The IAM policy for switching back is already in place.
The agent's access role covered only 3 of 7 log sources. Several “this field is empty” findings were artefacts of looking in the wrong sources.
Widened access (read-only), profiled every field in all 7 sources, and corrected the agent's vocabulary to match.
A plain “total for this window” query crashed because OpenSearch omits the aggregations key. The model then told the user the data was “beyond retention”, which was false.
Fixed the crash and added a rule: on a tool failure, say that it failed and never invent a reason.
“Sessions today between 10:00 and 11:00” came back empty even though there were matching records. The model had no real date, and its guess was well-formed enough to pass validation.
Injected the current UTC time into every turn.
Fuzzy matching ranked service closer to “device” than deviceType. The agent grouped by the wrong field and reported success.
Matching now checks substring containment before fuzzy ratio. Real fields that a tool doesn't allow are refused by name.
Every question failed at runtime start. Container logs showed the agent module missing from the image.
Ruled out a wrong-account session, checked the image history in ECR, rebuilt from a verified source and restored service the same day.
The client's own dashboard showed about 4× more playbacks than the agent reported.
Traced the gap to documented exclusion of monitoring-probe traffic (about 72% of staging playback records) plus the difference between counting sessions and counting records. Explained it to the client, who then chose to include probes by default. We shipped that change.
A bigger time range returned fewer matches than a smaller one inside it. The OpenTelemetry trace showed that the model had truncated a midnight-crossing range to one day and then described the full range.
Added a rule: one call per calendar day, add the results, and state each call's real window. Verified live on the first deploy.
This is a replay of my open-source reference implementation running on synthetic data. It contains no client data or code. Each question shows the tool calls the agent makes, including refusals and retries. Numbers are computed live by the reference code.
streaming-log-agent-demo: 28 tests, runs offlineFree scoping call · 30 minutes · No commitment
Write a few lines: the process or system, what hurts today (time, errors, cost or an AI pilot that stalled) and who uses it. After a short call you get a written plan and a fixed quote within ≈3 days.
Opens a pre-filled email · reply within one business day
The model reads. The code counts.
All projects