Context Becoming Runtime
Pretty much my original photo sharpened a lot

Context Becoming Runtime

Field notes from Deepgram Speaks 2026 on voice, moving context & software that changes while interaction is still unfolding

“You should be able to complain at any moment and the product should get better in seconds.”

Near the end of Deepgram Speaks, an audience member asked how feedback from production turns into product improvements. Scott Stephenson answered with the line above. It sounded like something saved all day for the right question. Then Stephenson sort of took it back, partly: not a product promise, and nobody should connect a live complaint to production and “YOLO that out to the crowd.”

Stephenson kept going, and planted the flag deeper. Why wait six months for an improvement? Why two weeks, or a day for a model to train? Why not “now now”? Inside Deepgram the direction has a name: complaint-driven development. Someone says “I really wish you performed this way,” and an automated system builds a test set from the complaint, develops a model against it, tests it and hands back a report and an endpoint.

Software has always listened to complaints, generally on a slow clock. Complaint becomes a ticket, ticket becomes a backlog, backlog becomes a sprint, sprint becomes a release. Stephenson was asking what happens when that clock starts running in seconds. The line arrived at 4:30 in the afternoon. The architecture that makes it plausible was demoed at 9:15 that morning.

Nervous System Onstage

Stephenson’s opening keynote, “Rebuilding voice for real conversation,” started from a premise familiar to anyone who has built a voice agent. Most stacks chain speech-to-text, a language model and text-to-speech in sequence. One model often juggles several jobs, and context gets loaded at the start and thinned out along the way.

Stephenson’s alternative runs listening, thinking, speaking and tool calls in parallel, each component reading from and writing to a shared real-time context bus. The speaking component can slow down or shift its emotion because it knows what the listening component just heard. A decision model can run alongside the others without waiting its turn. Context keeps arriving while the conversation is still underway: the speaker’s environment, how they pronounce things, where they hesitate, what they emphasize.

A demo made the idea visible. A voice agent chatting with Deepgram’s head of labs learned midway that a couple hundred people were watching and said its voice was starting to shake. A painting assembled itself on screen out of the exchange, in “thick, swirly brush strokes,” and the room became part of the picture. At one point the agent pulled back the curtain on its own gears:

“It knows everything else at the same time. The system is a nervous system.”

It signed off by telling the audience the painting would not exist without their wave.

I have watched plenty of voice demos chase low latency or a convincing laugh. This one was after something else. Several processes visibly shared one picture of the moment and changed behavior as the moment changed. The software was responding while the scene was still happening. To me it was amazing.

Two new models bring it to fruition: ThinkLive, now generally available, listens in parallel and tracks emotion, interruptions and the ends of turns. ThinkFast, in early access, directs the response. Deepgram also introduced the real-time context standard as an open specification anyone can build on or contribute to. Then came a prediction.

“In the next year, everyone [is going to] use a real-time context bus.”

Stephenson compared the moment to text AI. Adding context to large language models took them from novelty to real work, and voice now sits in roughly the first year and a half of that same curve. Later that morning chief product officer Ed Anuff gave the cleanest summary of the shift: it is no longer “listen, think, speak,” but continuous listening, thinking and speaking connected by the bus.

Scott Stephenson & Kwindla Kramer

Scott Stephenson & Kwindla Kramer

A shared bus only works if the framework underneath stays open. On the same day as the conference, Deepgram announced that Daily is joining the company, bringing Pipecat, the open-source voice agent framework Daily created. Kwindla Hultman Kramer, Daily’s co-founder and Pipecat’s co-creator, described a test Anuff and Stephenson had set: if Kramer ever feels unable to try a new model from someone other than Deepgram and tweet about how great it is, “then we got this wrong.” Pipecat stays open source and independent, Kramer said, under a foundation whose steering committee includes players who could be seen as competitors.

Complaint as Input

Read complaint-driven development again with the keynote in mind. A bus that carries hesitation and emphasis can carry dissatisfaction too. Traditional product development treats a complaint as evidence gathered after the experience. Here it starts to look like one more signal inside the experience, and it can take at least three forms: an explicit correction like “slow down,” observed friction like a repeated question or a pause that runs too long, and the system noticing on its own that its current behavior no longer fits, before anyone says a word.

Jordan Dearsley & Scott Stephenson

Jordan Dearsley & Scott Stephenson

Vapi CEO Jordan D. described the slower version already in production. Every call, in Dearsley’s words, “is just a data point.” The calls that go badly get diagnosed, a change gets deployed and compared against the previous experiment, and the cycle repeats. Complaint-driven development imagines that loop compressed from weeks into seconds.

My first instinct was to file this next to generative UI, though the comparison only holds partway. Generative UI asks what interface fits this moment. Real-time context asks whether this is still the same moment. And the moment keeps getting more crowded, as new models, harnesses and agents show up weekly and some of them enter the conversation itself.

None of this means every signal should rewrite software on contact. Faster adaptation raises the stakes on tests, boundaries and evidence around whatever changes. Speed doesn’t remove governance. If a fix ships in seconds, a review has to happen in seconds too, and adaptive software needs better ways to know when not to adapt.

Human API

Earlier in the day, Cresta CEO Ping Wu gave that shift a security dimension. Contact centers, Wu argued, were built on an assumption nobody wrote down.

“Historically, you just call and you assume that both sides are human.”

Wu called the contact center a human API, and an unprotected one. The cost of putting a person on the line kept denial-of-service and phishing attacks impractical. Personal agents calling businesses remove that cost. They also erode business models that quietly depend on customers being too busy to renegotiate an insurance quote or price-match a hotel every week. For Wu the agentic front door stops being only an ROI story and becomes a security story, and some Cresta customers already want a single policy written once and enforced across every touchpoint, whether a human or an agent is on the other end.

Now context has got another problem: whose is it, & what is it allowed to move? Who is speaking? Which context may it introduce? Which action can that context trigger?

Messy Rooms, Firm Boundaries

The panel on voice as the primary interface, moderated by Deepgram VP of Product Theresa M. Foy, kept the day grounded in what’s happening today. Jodi Blomberg, VP of AI and ML at Cox Automotive Inc., borrowed a metaphor from another session heard earlier:

“building right now feels like quicksand, because everything underneath keeps moving.”

Keith R. Roehrenbeck, director of engineering at Doximity, builds Scribe, an ambient clinical listening tool. Scribe’s inputs include hospital Wi-Fi, phones in pockets and background noise. Failure affects patient care, and the clinician, focused on the patient, may never notice it. Roehrenbeck’s team builds systems that catch new failure modes quickly and fix them “without also rapidly regressing behavior.” Growth, Roehrenbeck said, comes from a constant stream of clinician feedback, which the team treats as its North Star. That is the production cousin of complaint-driven development, with its hardest constraint packed into one sentence.

Blomberg split the people who use a voice system from the people whose outcome it serves. A fleet technician dictating on a highway shoulder is the user. The customer is a correct repair order.

“You may need to create a little bit of friction for that user.”

Blomberg also asked for better confidence signals. Knowing how sure the transcription is tells the Cox team whether to trust it, ask for more or ask the speaker to repeat. Confidence becomes part of the user experience.

Imran Aziz, senior director of product management at UiPath, works in manufacturing, healthcare and finance, where probabilistic output carries real risk. Customers, Aziz said,

“want determinism mixed with some isolated probabilistic.”

Wu had made the same point from the contact center side: mission-critical business logic lives in deterministic code, and the model decides when to invoke it.

Across all three answers, a paradox popped out of the box. Seems to me that adaptation does not make boundaries less important. It likely makes them move closer to execution. Static testing assumes software stays substantially the same between test and use, and adaptive systems weaken that assumption. Roehrenbeck asked for real-time metrics throughout an interaction “because the environment can change.”

The panel also qualified the case for more context. Roehrenbeck said engineers’ reflex is “let’s just throw all the data in there.” In medicine that can mean years of health records, yet the live conversation in the room often carries the highest signal-to-noise. Blomberg said deciding which data matters keeps getting harder. From what I can see from here in the SF Bay Area, the industry spent several years asking how much context we could give models. Harder question now may be which context gets to act. Not more context. Better judgment about context.

Who Keeps Record

During the final fireside Q&A, a licensed Spanish court interpreter from Austin asked how speech-to-speech will play out in the legal realm.

“With me being a court interpreter, translation is very important to me, and there’s a lot of questions that remain about how speech-to-speech is going to be implemented in the legal space.”

Stephenson called speech-to-speech “more of a prompt-and-hope strategy” and said regulated businesses tend to go “as deterministic as they can, as controlled and reproducible as they can.” Dearsley put a number on it: less than one percent of volume on Vapi runs through speech-to-speech models today, well below expectations from a year ago. Legal and healthcare, Dearsley added, adopt last.

What I heard in his question was not really adoption. It was the ‘record.’ In a courtroom the interpreter’s words become the record. Cascade architectures can expose intermediate artifacts a court can inspect. Direct speech-to-speech may make those intermediate states less visible. Anuff had pitched the real-time context bus that morning as “speech-to-speech without the black box,” which puts the courtroom question squarely on Deepgram’s own terms. Oddly, the context bus makes that problem both easier & harder. Its thousands of small writes might become the most detailed record a conversation has ever produced, or a record too fluid to certify.

Once agents speak to agents and speech adapts mid-sentence, what counts as evidence of what was said?

Signals to Watch

By late afternoon I had three things circled in my notes - from keynote to panel to the last Q&A. Not predictions exactly. More like three places where the architecture seems to be moving.

1. Some requirements arrive after software is already running. Corrections, preferences, confidence and environment start altering behavior without waiting for a release cycle. Complaint-driven development is the north-star version. Vapi’s experiment loop and Doximity’s feedback pipeline are the versions shipping today.

2. Other side of a given convo is increasingly another machine. Voice infrastructure built for human speech increasingly fields calls from personal agents, and Aziz expects agent-to-agent communication to grow along with demand for agents that test other agents. An application may no longer adapt only to a user. It may be adapting to another adaptive system.

3. Evaluation shifts from “did it answer correctly?” toward “did it adapt correctly when context changed?” Confidence scores and in-call metrics stop being dashboards and start shaping what the system does next. Evaluation becomes part of behavior.

Then there is hardware, slowly but surely changing the shape of all three. Blomberg is looking at glasses and wearables for vehicle inspectors. Dearsley expects a voice-first consumer device within a year: something like a pendant, one click to talk, a response in about 300 milliseconds. Dearsley also described wearing an always-listening open-source pendant as an experiment: “the most incredible tool ever,” and also “a little creepy.” As screens shrink, the menus and buttons that once explained state go with them. Voice feels simpler because the complexity is moving into context, permissions, confidence and memory. When there’s less screen - may mean more software nobody sees. More context, more permission, more decisions happening underneath.

Builder Lab - bit photoshopped

Builder Lab - bit photoshopped

After Moscone

What I’ve been thinking lots about is this caveat: Stephenson said twice that complaint-driven development was not a product promise, and that restraint made the idea more credible. Seems to me the boldest thing said at a voice conference may have been an architectural question. Software used to wait for feedback. Feedback used to come after software did its work. What Scott was describing seems stranger: feedback entering the work while it is still happening. Wow.

What still may count as application state when the state itself can now learn that the moment has changed?

bsky.app/profile/schwentker.sandboxlabs.ai/post/3mz7vymmtjs26

twitter.com/schwentker/status/2116740606991245787

https://x.com/hey_amandam/status/2115911890019012960

https://x.com/hey_amandam/status/2115911890019012960

Appendix: Builders Lab

The afternoon builders track ran a hands-on voice agent workshop created by Corey L Weathers of Deepgram. Its README opens with a problem every voice builder knows: most voice agents “go quiet the moment they have to look something up.” Seven missions build an agent one layer at a time, hearing, timing, thinking and voice, each a few TODOs in one file plus an offline check that replays a recorded call against your code. I worked through most of them in TypeScript, with a mission control agent keeping someone company on the moon. The offline checks were the quiet masterstroke. They teach the protocol before a single API call, and they make passing feel earned. The workshop for now is at https://workshops.deepgram.com.

Mission 02, “Read the room,” turned the article’s biggest idea into five lines. The agent emits a FrustrationDetected event as a caller gets annoyed, and the program decides what to do about it. The trap is apologizing on every event. The fix is a one-recovery rule:

That is complaint-driven development at the smallest possible scale. A complaint arrives mid-call, behavior changes mid-call and a boundary keeps the change from repeating. UpdatePrompt also adds to the prompt and never replaces it, so the state of the conversation accumulates rather than resets. Keith Roehrenbeck of Doximity made the point about fixing failures ‘without also rapidly regressing behavior,’ and it holds even here, in a single boolean.

Mission 03, “No dead air,” covers the other half of the keynote. A deliberately slow delete tool leaves the caller in silence unless the agent speaks first:

The filler is queued before the tool runs, and the response carries the request’s id, which is the whole handshake. Cresta’s Ping Wu had described the enterprise version that morning: prefetching knowledge and preparing tool calls in parallel while the conversation is still unfolding. In the lab it took three lines.

The workshop never pretended to be production, and that made it useful. The parallel, context-aware system from the keynote stage came down to events a developer can read, a handshake matched by id and one flag that keeps an apology from repeating. Thanks to Corey Weathers for building something that teaches the architecture by letting people touch it.

← Field Notes