Generative AI agents are finally moving from flashy demos to hard nosed production work, and few examples show that shift as clearly as Motorway’s collaboration with Amazon Web Services on dealer stock search. In a live automotive marketplace where every wrong answer can cost dealers time and money, cutting agent errors from roughly one in eight queries to about one in fifty is not just a nice improvement, it is the difference between a fun experiment and a trusted tool.
From manual filters to conversational search
For years, dealer stock search looked more like spreadsheet archaeology than modern software. Dealers typically sifted through bulk listings with static CSV files and rigid filters, piecing together complex queries by hand and often spending hours to find a handful of viable vehicles.
Dealer stock search meant digging through static CSVs and rigid filters, spending hours to unearth a few decent cars
The industry has seen search interfaces slowly improve, but most systems remained narrowly scripted and brittle, with limited ability to understand natural language or nuanced intent. Early chat-based assistants in retail and automotive tended to sit on top of existing search engines without deep integration into inventory systems, which made them feel helpful in simple cases but unreliable when queries became messy or context-heavy.
Motorway’s dealer stock search agent marks a clear step beyond that pattern. It lets dealers speak in everyday language about what they need, such as a diesel sport utility vehicle under a given budget near a specific location or a sporty automatic car that still works for a family and then translates those requests into structured searches over the underlying marketplace. The agent is built to serve a high volume environment, supporting roughly one thousand five hundred concurrent users during peak periods while still meeting the reliability targets Motorway set for enterprise use.
Inside Motorway’s agent architecture
At the core of the dealer stock search agent is an architecture that combines the Strands Agents software development kit for orchestration with Amazon Bedrock AgentCore for integration, runtime, and observability. This pairing gives Motorway a standardized way to define agents, connect them to tools, and deploy them into a managed execution environment without reinventing infrastructure for every new use case. Because Amazon Bedrock AgentCore runs on an ARM64 runtime, Motorway and AWS also had to address common build and configuration issues such as architecture mismatches, container port exposure, and ECR authentication errors to keep the agent stable under load.
The agent exposes eight distinct tools that work together to answer dealer queries. Some focus on structured filtering across more than eighty-nine vehicle attributes, while others provide semantic search over natural language descriptions using LanceDB and Amazon Titan Text Embeddings V2. This mix lets the agent bridge the gap between casual dealer language and the precise fields that drive inventory systems.
Crucially, each tool sits inside a clearly defined fault isolation boundary, which limits the blast radius of failures when something goes wrong. Tools follow strict versioned contracts for requests and responses, and every hop between them includes schema validation to catch malformed payloads and incompatible parameter changes before they cascade into user-visible errors.
On top of structural checks, Motorway and AWS introduced semantic validation rules, including consistency checks between fields and data freshness requirements, so that answers reflect current inventory rather than stale records.
The orchestration itself is intentionally deterministic. Instead of letting a model improvise arbitrary tool calls, the agent follows well-defined flows that constrain which tools it can invoke and in what sequence. This guardrail reduces spurious tool invocations, lowers the chance that the agent drifts away from the intended search path, and makes behavior easier for engineers and operations teams to reason about. Operational risks associated with agentic systems are thus minimized.
The three layer evaluation blueprint
What really differentiates this project is not just the architecture but the evaluation pipeline wrapped around it. Motorway and AWS adopted a three-layer framework that separately examines tool usage, reasoning quality, and final output quality before any agent version sees full production traffic.
The first layer focuses on whether the agent chooses the right tools, passes parameters correctly, and respects schema constraints. Deterministic graders check tool calls against expected behavior, acting like automated reviewers that either pass or fail each interaction with no ambiguity.
The second and third layers use a large language model as a judge to score reasoning coherence and response helpfulness, respectively, with human review used to calibrate edge cases and safety-sensitive scenarios. In Motorway’s setup, Claude Sonnet 4.6 serves as the judging model for these layers, providing structured evaluations that can be aggregated across test suites.
Initial test suites draw on twenty to fifty real user queries, including both successful interactions and refusal cases, so that the agent is measured on realistic behavior rather than synthetic prompts. The evaluation harness itself is relatively lightweight, with the sample suite costing on the order of between five and ten dollars in Amazon Bedrock inference charges, which makes it practical to run frequently during development.
Motorway defined high bar thresholds for promotion. Tool usage must show more than ninety-five percent pass rate, reasoning scores need to exceed eighty-five percent, and output quality must surpass ninety percent before a candidate agent is allowed to handle significant production traffic. If any layer fails, logs and metrics collected through Amazon CloudWatch and AgentCore observability features provide detailed traces, error patterns, and trend data that guide targeted fixes rather than guesswork.
There is an important trust dimension here. By making evaluation criteria explicit and tying them to real data, Motorway can explain to stakeholders how an agent version earned its place in production. The process also gives teams a shared language for discussing failures, such as a good tool choice with poor reasoning or strong reasoning paired with outdated inventory, and then deciding what to fix first.
What changed in production
Once these practices moved from lab tests to production-shaped validation, the effects were tangible. Tool usage accuracy exceeded ninety-five percent, meaning the agent almost always chose appropriate tools and parameters when handling dealer stock queries.
Reasoning and output quality scores consistently cleared the eighty-five and ninety percent thresholds, which reduced intent misinterpretation and sharpened the relevance of returned inventory results.
The headline metric is the drop in incorrect query results from roughly one in eight to about one in fifty, which translates to an error rate reduction from around twelve and a half percent to close to two percent. Combined with deterministic workflows, schema validation, and ongoing monitoring, this cut in errors gives operations teams much clearer visibility into each processed query and makes it easier to isolate and remediate failures before they accumulate.
Issue detection time also fell sharply. The evaluation pipeline and observability stack reduced the window for spotting problems from a few hours down to just minutes, which matters in a marketplace where stale or faulty search behavior can quickly influence pricing and dealer decisions.
Taken together, these numbers start to look like the reliability figures associated with mature software services rather than experimental agents. That shift is what makes this case study relevant well beyond Motorway’s business.
How this fits into the wider agentic landscape
Motorway’s work is part of a broader movement in automotive and adjacent industries toward agentic AI systems that are deeply embedded in operations rather than standing apart as optional assistants.
Cox Automotive, for example, has used Amazon Bedrock AgentCore and Strands Agents to launch seventeen enterprise solutions in under a year, spanning consumer car shopping, fleet service, auctions, and dealership operations. In one internal platform called FleetMate, Cox cut complex fleet repair estimate times from a range of eight to forty-eight hours down to roughly thirty minutes by structuring agent workflows in a standardized execution environment.
Motorway itself has gone further than dealer search in its AI journey. The company reports using AI coding agents with a system of five quality gates to support more than one hundred twenty engineers, leading to roughly two hundred fifty percent more deployments while maintaining production safety. Those gates include spec-driven planning, AI-assisted code review, deterministic tests, complexity checks, and final human review, echoing the layered evaluation strategy seen in the stock search agent.
When viewed against this backdrop, the dealer stock search collaboration with AWS looks less like a one-off win and more like a blueprint for how agentic AI can be industrialized. The pattern is clear. Start with a business-critical workflow, wrap it in deterministic orchestration and strict tool contracts, add semantic and safety checks, then insist on measurable evaluation thresholds before release. The technology choices are important, but the discipline around evaluation and observability is what really moves error rates and trust.
Opportunities and risks for businesses and society
For businesses, the upside of this approach is straightforward. Better search accuracy and faster issue detection improve operational efficiency, build dealer trust, and reduce the hidden cost of wrong or misleading answers in complex marketplaces. Organizations that adopt similar blueprints can expect gains in throughput and user satisfaction, especially when agents tap into rich structured data combined with semantic search capabilities.
There are broader societal implications as well. As AI agents become responsible for decisions that affect pricing, access to goods, and the distribution of opportunities, reliability becomes a matter of fairness and accountability, not just convenience. A dealer who consistently receives poorer search results because an agent misinterprets their style of queries is experiencing a form of algorithmic friction that can compound over time. Structured evaluation and transparent thresholds help surface these issues earlier, although they do not eliminate them.
Risks remain, and they deserve explicit acknowledgment. Relying on an LLM as judge introduces the possibility of systematic biases or blind spots in evaluation, especially around nuanced ethical or safety questions. Metrics like pass rates and quality scores can encourage optimization toward what is easy to measure, potentially underweighting rare but high-impact failures. Continuous human oversight, domain-specific safety evaluators, and periodic audits of evaluation criteria are necessary complements to automated pipelines.
Another limitation is that deterministic workflows and strict schemas can sometimes slow innovation or restrict flexibility. When business requirements change rapidly, teams must update contracts, validators, and orchestration graphs in lockstep, which demands strong engineering discipline and cross-functional communication. The Motorway and Cox examples suggest that this investment is worthwhile, but smaller organizations may need guidance and tooling to avoid getting stuck in complexity.
Practical lessons for teams building agents today
Several lessons from this case study apply directly to teams considering their own agentic deployments.
First, treat tool selection and parameter passing as a first-class design challenge rather than an implementation detail. The use of versioned contracts, schema validation at every boundary, and fault isolation around tools is what makes Motorway’s agent robust in practice.
Second, invest in a layered evaluation pipeline early. Separating tool usage, reasoning quality, and output quality lets teams pinpoint where a candidate agent is failing and focus improvements accordingly, instead of relying on a single blended score that obscures root causes.
Third, combine structured determinism with selective flexibility. Deterministic flows reduce chaos and make agents auditable, but they can still be designed to accommodate variations in user intent, especially when supported by semantic search tools and well-chosen embeddings.
Finally, use observability not just for debugging but for learning. The logs, traces, and metrics collected from evaluation and production traffic can reveal patterns in user behavior and failure modes that inform future features, safety guardrails, and business processes.
Looking ahead
The Motorway and AWS collaboration shows that agentic AI can clear the bar of enterprise reliability when treated as a software engineering discipline rather than a novelty project. Cutting error rates down to around two percent in a high-volume automotive marketplace hints at a future where conversational agents quietly handle complex operational tasks with predictable performance and clear accountability.
As more organizations in transportation, retail, finance, and public services adopt similar blueprints, the conversation about AI will likely shift from whether agents can work at all to how they should be evaluated, governed, and integrated into human workflows. The balance between deterministic structure and adaptive intelligence, and between automated evaluation and human judgment, will define how trustworthy these systems feel to both businesses and end users.
The Motorway case suggests that the path forward is not magic but method. Careful architecture, layered evaluation, and honest measurement can turn generative agents from risky experiments into dependable infrastructure, provided teams remain clear-eyed about the limits of metrics and the need for ongoing human oversight.
Conclusion
The story of Motorway and AWS quietly marks an inflection point in how serious teams are starting to treat AI agents. It shows that the biggest gains no longer come from chasing the newest model, but from building disciplined evaluation pipelines around the models already in use. In Motorway’s case, that discipline cut agent error rates from roughly one in eight queries to about one in fifty and shrank detection times from hours to minutes.
From experimental agents to accountable systems
For the last two years, AI agents have moved from intriguing demos to production systems that handle real work in sales, support, logistics, and engineering. Yet their real world performance has lagged behind the hype. Independent analyses estimate that agents can fail between 70 and 95 percent of the time in realistic production settings, especially when tasks are repeated or involve long, multi step workflows.
This gap between promise and reality comes from familiar causes. Agents are non deterministic, they make silent errors that look plausible, and they interact with messy external systems. Traditional software testing only partially applies. As a result, many teams have shipped agents with minimal evaluation, discovered brittle behavior in production, and then struggled to debug what went wrong.
In parallel, a second trend has been emerging. Reliability focused teams have started to borrow ideas from site reliability engineering and observability, applying them to AI. They combine structured logging, error taxonomies, circuit breakers, and layered evaluation to move failure rates from the high tens of percent into something closer to traditional software reliability. The Motorway and AWS collaboration sits squarely in this second camp.
What Motorway and AWS actually built
Motorway is a used car marketplace where dealers depend on accurate stock search to find the right vehicles quickly. A dealer stock search agent that returns wrong or incomplete results does not just annoy users. It can directly hurt dealer trust and revenue.
To address this, Motorway partnered with AWS to build a dealer stock search agent and a surrounding evaluation pipeline using the Strands Agents SDK and Amazon Bedrock AgentCore. Strands handles structured evaluation in development environments, while AgentCore Evaluations monitors the agent in live traffic. Together they form an end to end loop from test scenario design to production incident detection.
The results are unusually concrete for an AI case study. According to AWS and Motorway, the team:
- Reduced incorrect results from about one in eight queries to roughly one in fifty, which corresponds to an error rate drop from around 12.5 percent to about 2 percent.
- Cut the time to detect issues from a few hours down to a few minutes, thanks to continuous monitoring and targeted alerts.
- Achieved a 96 percent task completion rate and 94 percent context retention in evaluation, while reducing monthly incidents from around 12 to about 2.
Those numbers are not the outcome of a single clever prompt or a brand new model. They are the product of a carefully designed evaluation strategy that treats the agent as a production system rather than a novelty experiment.
A layered evaluation strategy, not a single gate
A key insight in the Motorway blueprint is that no single test can capture agent quality. Instead, the team designed three layers of evaluation, each tuned to a different aspect of behavior.
First, they defined strict thresholds. The agent must use tools correctly in more than 95 percent of relevant situations, maintain coherent reasoning in more than 85 percent of cases, and deliver high quality answers more than 90 percent of the time. Those are aggressive targets, similar to the reliability levels seen in advanced error handling deployments that push overall success rates toward 99 percent.
Second, they matched each layer with the right evaluator.
- For tool use and basic correctness, they rely on deterministic checks. For example, if the agent should call a particular search tool when a dealer asks for stock in a city, the evaluation verifies that this call actually happens. This style of checking is closer to traditional assertions in software tests and removes ambiguity.
- For reasoning quality and answer usefulness, they use a powerful model, Claude Sonnet 4.6, as a judging system. This evaluator scores whether the agent followed a sensible chain of thought and whether the final answer aligns with the intent of the query.
- For edge cases and safety sensitive situations, they still involve human reviewers. Humans calibrate the automatic judgments, especially where even a small number of failures would be unacceptable.
Third, they explicitly address non deterministic behavior. Instead of evaluating a single run of the agent, they use a pass at k metric that measures how often the agent can succeed within a few attempts. This is better aligned with how agents are often used in practice, where they may be allowed to retry or explore multiple solution paths.
Crucially, all of this is wired into a continuous loop. Strands provides standardised scenarios and metrics during development, while AgentCore Evaluations applies similar checks to real traffic and raises alerts when failure rates or behavior patterns drift. That continuity is what allows Motorway to collapse detection time from hours to minutes.
Why evaluation and observability now matter more than new models
On the surface, this story might look like another case of a company adopting a new AWS capability. The deeper lesson is more important. The largest improvements in Motorway’s metrics came from evaluation design, not from switching to a completely different model family.
Across the industry, there is growing evidence that robust error handling and monitoring can transform reliability without changing the underlying model. One detailed study found that adding retries with backoff, circuit breakers, and graceful degradation improved agent reliability from 87 percent to over 99 percent in a production setting, a fourteenfold reduction in failures. Others emphasize that comprehensive audit logging, real time alerts on error thresholds, and outcome validation are now non optional for serious deployments.
Motorway’s pipeline reflects these lessons. It turns every failure into data. Query transcripts, tool calls, evaluator scores, and incidents are all captured and fed back into model prompts, tools, and system design. When combined with clear business metrics such as incident counts and task completion rates, this creates a virtuous cycle where the agent steadily converges toward acceptable performance.
This shift also fits the broader pattern observed in research from Perplexity Sonar and others. As models plateau in raw capability gains from one release to the next, systematic evaluation, observability, and feedback loops increasingly determine which teams can safely deploy agents at scale.
Implications for engineering teams
For engineering leaders, the Motorway blueprint suggests several practical lessons.
First, define success in business terms before touching prompts or models. Motorway cared about incorrect dealer stock search results, incident counts, and resolution times, so those metrics framed the entire evaluation design. Many struggling agent projects never achieve this clarity and remain stuck in subjective judgments of quality.
Second, treat agents as distributed systems rather than black boxes. That means instrumenting every tool call, tracking error taxonomies, and setting thresholds and alerts on anomaly patterns. Production guides now consistently recommend practices such as retry with backoff, circuit breakers for fragile dependencies, model fallback chains, and timeouts to prevent agents from hanging workflows. Motorway’s use of AgentCore Evaluations is an example of this mindset applied to inventory search.
Third, embrace layered testing. Unit tests alone do not capture the behavior of an agent that reasons, calls tools, and interacts with live data. Successful teams combine deterministic checks, adversarial scenarios, model based judging, and targeted human review, just as Motorway did with its three layer design.
Finally, invest early in incident analysis. Postmortems that replay failed agent runs, classify error types, and update evaluation scenarios are emerging as a standard practice. Motorway’s reduction from roughly 12 incidents per month to about 2 indicates not only better runtime behavior but also a stronger feedback loop around failures.
Implications for businesses and users
From a business perspective, Motorway’s results highlight a path toward trustworthy automation that does not rely on blind faith in AI. An error rate near 2 percent in dealer stock search, combined with fast detection and remediation, is the kind of number operational teams can reason about and incorporate into risk calculations.
For users, the benefit shows up as consistency. Dealers are less likely to waste time on missing or incorrect vehicles, and support teams will encounter fewer unexplained agent failures. That reliability is crucial to maintaining trust in the marketplace. Once users experience erratic behavior from an AI driven feature, they tend to avoid it even after improvements, which is why early investment in evaluation pays off disproportionately.
At a societal level, this kind of work helps answer a central question: can AI systems be made predictable enough for high stakes workflows without requiring perfect models. Motorway’s example suggests that careful engineering can make agents not only powerful but also accountable, as long as organizations are willing to invest in measurement, governance, and human oversight.
Risks, limitations, and what remains uncertain
It is important to keep this achievement in context. A dealer stock search agent, while meaningful, is still a constrained domain. Error tolerances are different from healthcare, finance, or safety critical infrastructure. Success in one domain does not automatically prove that the same approach will yield similar error rates elsewhere.
There are also risks in relying heavily on model based evaluators. If the judging model shares biases or blind spots with the primary agent, some classes of failure may go under detected. Human review mitigates this, but only for the sampled cases that reviewers actually see.
Another limitation is cost and complexity. Building and maintaining a pipeline with Strands, AgentCore, model based judges, and human calibration is a substantial engineering investment. Smaller teams may find it challenging to reproduce the full setup. The wider ecosystem is still converging on standards and tooling that can lower the barrier to entry.
Finally, metrics like pass at k and task completion are necessary but not sufficient. They do not fully capture user trust, long term behavior drift, or adversarial robustness. Many teams are still experimenting with richer metrics that connect agent performance directly to user satisfaction and business outcomes.
The road ahead
Despite those caveats, the Motorway and AWS work is a strong signal of where serious AI engineering is heading. The focus is moving from single model benchmarks to holistic reliability engineering that spans development, deployment, and operations.
Over the next two years, expect to see more companies adopting similar blueprints:
- Clearly defined task specific metrics and error budgets for every agent.
- Shared evaluation frameworks that run the same scenarios in development and production.
- Automated quality gates that combine deterministic checks, model based scoring, and human review.
- Rich observability that treats agent behavior as data to be analyzed, not anecdotes to be debated.
The deeper lesson is simple but easy to overlook. For production AI agents, reliability is no longer an afterthought. It is a product feature in its own right. Motorway’s collaboration with AWS shows that by investing in structured evaluation and observability, organizations can turn fragile prototypes into dependable systems and transform every failure into an opportunity to improve. reddit
