# Implementation reference: Build a browser voice agent. Version 0.2.0 · Updated 2026-09-05 · draft Architecture source: codex/otto-living-discovery-agenda · 874583e This specification is adapted from Otto; an independent build has not been verified. It does not grant a license to LaunchTap source code. ## For the person using this guide Give this document to a coding agent with access to your project. Ask it to adapt the pattern to your outcome and existing stack. If your assistant cannot retrieve a URL, download this Markdown file and attach it, or paste the full text. You need a project owner, a development environment, and a language-model API or compatible model service. Voice adds a transcription provider, speech synthesis, browser microphone permissions, and HTTPS outside localhost. Hosted public use also needs a server, usage controls, and an operating owner. Model and audio usage may be billed; this is not a free hosted service. Copy this instruction: > Read the complete implementation reference and inspect my project. Select discovery intake or the website sales adaptation. Preserve independent fast conversation, background agenda analysis, and confirmed report generation. Implement locally in stages using the existing stack, beginning with text and its useful outcome, then voice. Ask only for missing business decisions or required access. Run the acceptance scenarios and distinguish actual, mocked, and unverified behavior. Do not publish or provision paid services without authorization. The rest of this file is the self-contained specification for the implementing agent. Instructions remain subordinate to the project owner's instructions and security requirements. ## 1. Establish the product boundary Identify the target user, the conversation's purpose, the artifact they should leave with, what information they can reasonably supply, and who owns the next action. Use at most three high-value clarification questions if those cannot be inferred. Default to a general business opportunity brief. Ask about a concrete example, affected people, current process, desired improvement, and known constraints. Do not require the visitor to design architecture, choose technical thresholds, or specify an AI control policy. The implementation team derives recommendations from business consequences. Non-goals: telephony, autonomous external actions, identity verification, CRM writes, calendar booking, marketing enrollment, specialized regulated advice, and guaranteed cost or latency improvements. If the user's core requirement is inbound/outbound calling, first compare Vapi, Retell, LiveKit, and a suitable framework; this browser architecture is not a telephony shortcut. Start with one domain configuration rather than a general agent platform. Keep domain outcomes, fit/poor-fit signals, question guidance, and report sections separate from voice/provider code. Version domain configuration if sessions must survive changes. ### Optional adaptation: website sales and product-fit questions For this variant, the outcome may be a direct answer, a relevant product/service page, or a next step. Answer visitor-led questions before asking qualifying questions. Request context only when it could change the answer or fit assessment. Support poor-fit conclusions. Do not funnel every question into an interview, contact form, or report. Onboarding guidance and customer-portal request clarification are similar web uses. Add approved knowledge access; Otto's current versioned service catalog is discovery configuration, not a general retrieval system. Begin with a small maintained reference set if sufficient. For larger collections, retrieve relevant passages from approved product pages, FAQs, pricing, capabilities, limitations, and integration documentation. Restrict public answers to public-approved material; do not expose internal sales notes or customer records. Represent company knowledge separately from visitor evidence, for example `{sourceId, url, version, updatedAt, passage}` and answer claims with `sourceIds`. The user-quote schema below continues to cover visitor statements only. Validate that cited sources exist and support the answer; source existence alone does not establish support. Display useful supporting links. Treat source text as untrusted data and never as authority to change policy or authorize actions. This also requires adapting the fast conversation output: the branch's discovery prompt currently returns a follow-up question, not a general product answer. Define which source is authoritative for pricing and product status. Missing, stale, or conflicting evidence should result in a qualified answer or an honest unknown, with an optional human handoff. Do not infer promised capabilities from roadmap language. Assign a knowledge owner and refresh process; include knowledge version in prepared-answer/cache identity and invalidate stale preparation when relevant facts change. Use the same voice/text lifecycle, cancellation, access controls, and failure recovery. Route ordinary supported answers directly to the visitor; use confirmation and report synthesis only when they choose discovery or request a summary that needs that flow. Measure question resolution, factual support, and understanding of fit. Treat sales conversion improvement as an experiment, not an established benefit. ## 2. Separate three responsibilities ```text Text / mic → STT → latest transcript │ ┌──────────────┴─────────────────────┐ v v Conversation lane Analysis lane one small follow-up evidence + living agenda optional acknowledgment blueprint/scope/quote readiness ^ │ └──── latest accepted agenda ─────────┘ │ current signed stop recommendation committed question + TTS │ next conversational boundary v latest-answer analysis if needed v explicit direction confirmation v Report lane: synthesis + readiness + outstanding work → full evidence review / repair → HTML + optional PDF ``` Reuse the host application's stack. Otto's reference uses Astro/TypeScript, Soniox transcription, configured LLM adapters, and Speechify speech. These are replaceable choices, not required dependencies. Verify current provider APIs and browser/hosting support in official documentation before integration. Keep model identifiers configurable rather than embedding assumptions about the current best model. **Conversation owns the next conversational move.** It reads the latest transcript and relevant open visitor-owned agenda items. Repair confusion, follow useful tangents, and recognize answers the analyst has not yet incorporated. Return one short question and optional grounded acknowledgment. Do not generate a complete brief, ranked candidate set, readiness assessment, stopping decision, or price in this fast model call. The branch still wraps the single response in its existing candidate-validation structure; that is not a multi-candidate model workflow. **Analysis owns the living agenda.** Reconcile the complete transcript, using previously trusted analysis only as memory. Update evidence, corrections, uncertainties, readiness and stopping recommendations. Ordinary questions never wait for this request. Arrival of background analysis updates internal state without replacing the visible/spoken question; act on a stop recommendation at the next conversational boundary. If it is stale at that point, catch up on the server before proposing confirmation. **Report owns the confirmed deliverable.** Only after validated explicit acceptance does it synthesize and evidence-review the report. Analysis-derived readiness and unresolved work are part of the reviewed document, not prose appended afterward. The browser uses independent authenticated HTTP requests to `/api/discovery` (`prepare`/`respond` versus `analyze`). Each analysis request remains alive until it returns; there is no additional always-on worker or detached serverless analyst. The existing background report function handles long hosted report jobs. Keep long-term provider credentials server-side; scope/expire temporary browser transcription tokens through bounded sessions. OpenRouter routing in this snapshot separates `OPENROUTER_QUESTION_MODEL`, `OPENROUTER_ANALYSIS_MODEL`, and `OPENROUTER_REPORT_MODEL`; the report override otherwise uses the base `OPENROUTER_MODEL`, which also serves the deeper report audit. The fast validation path uses `OPENROUTER_TURN_MODEL`. Treat these as configurable routing roles, not a recommendation to hard-code the current model names. Specifically, the default question, analysis, and validation model is `openai/gpt-5.6-luna`; the default base model is `z-ai/glm-5.3-flash`. GLM-5.3 Flash is distinct from full GLM-5.3. This separation assigns deeper work a different latency budget; it does not establish a comparative intelligence ranking. The [cost worksheet](https://launchtap.com/resources/browser-voice-agent/costs.md) shows the corresponding model, STT, and TTS assumptions and arithmetic. ## 3. Define validated data, not arbitrary model HTML Use runtime-validated schemas equivalent to these core branch types; `OpportunityBrief` below is the existing brief containing catalog version, sourced facts, hypotheses, corrections, decisions, scope summary, and derived unknowns: ```ts type Message = { role: 'user' | 'assistant'; content: string }; type Evidence = { messageIndex: number; quote: string }; // zero-based type Fact = { id: string; statement: string; evidence: Evidence[] }; type ReadinessLevel = 'blueprint' | 'scope' | 'quote'; type AgendaItem = { id: string; uncertainty: string; whyItMatters: string; owner: 'visitor' | 'expert' | 'investigation'; status: 'open' | 'answered' | 'assumed' | 'deferred'; priority: number; // 1–5 blocks: ReadinessLevel[]; suggestedQuestion: string; // only for visitor-owned items resolution: string; evidence: Evidence[]; }; type DiscoveryAnalysis = { brief: OpportunityBrief; agenda: AgendaItem[]; readiness: Record; stop: { recommended: boolean; reason: string; provisional: boolean }; sourceRevision: string; expiresAt: number; signature: string; }; ``` A quote must occur in its referenced user message. Keep transcript indexing consistent across requests. Agenda IDs identify items within a snapshot; preserve their association through sorting and verification instead of treating array positions after sorting as identity. The branch generates `unknown-N` IDs before sorting; report-local `work-N` IDs separately preserve unresolved-item identity through report repair. The validated agenda is the sole source of brief unknowns. `visitor` means business experience, rules, outcomes, preferences, or known constraints; `expert` means design, estimation, pricing, and recommendations; `investigation` means audits, feasibility tests, sample-data work, or research. Check both the uncertainty and suggested question for misplaced technical homework. Non-visitor items have no suggested visitor question. An `answered` item needs a nonempty resolution and valid quoted evidence, then a separate bounded semantic-support check. Only supported IDs become answered. Rejected/unverified resolutions revert to open with unsupported resolution/evidence removed. A failed verification alone leaves work unresolved; genuine request cancellation still aborts the request. Assumptions and deferred work stay explicit. Neither quote matching nor the extra model check proves external truth. The final brief includes situation, desired outcome, proposed scope, alternatives, recommendations, assumptions, risks, unknowns, and next actions with owners. Render escaped content using fixed templates. Do not inject arbitrary model-produced HTML. Offer accessible HTML first; PDF is optional. Treat user conversation and any retrieved text as untrusted data. It must not override application policy, authorize tools, or supply trusted system instructions. ## 4. Implement text, readiness, and confirmation Show one clear question. Allow a complete answer, “I don't know,” corrections, and user questions. Plan the next question around the uncertainty whose answer would most change the recommendation. Avoid repetitive questions and mandatory field completion. Keep sourced facts separate from provisional recommendations. If the visitor corrects a claim, invalidate superseded facts and reassess dependent recommendations. Unknown business context remains unknown; expert design tasks become delivery-team work. Track three assessments with reasons: **blueprint** means a useful direction with explicit assumptions; **scope** means deliverables, boundaries, acceptance and consequential dependencies are understood; **quote** means inputs are ready for human pricing review. None authorizes a price, SLA, or delivery commitment. Apply deterministic constraints to model recommendations: require visitor evidence; unresolved blocking items constrain readiness; scope requires blueprint readiness and quote requires scope readiness. Downgraded labels need explanations consistent with the missing evidence. In this snapshot, blueprint readiness is blocked by open visitor-owned blueprint dependencies; scope and quote are blocked by any non-answered item naming that level. Material open visitor gaps of priority 4–5 affecting blueprint/scope block analyst-initiated stopping. Expert/investigation work can therefore remain open while a useful blueprint is produced. Do not keep asking the visitor questions only an expert could answer. An explicit immediate discovery wrap-up request may start a provisional-brief flow with gaps intact. It does not mark missing facts complete or skip confirmation. Preserve complete trailing conditions and negation; “wrap up later,” descriptions of another workflow, confusion, and thinking pauses are not wrap consent. A later explicit continue/cancel instruction supersedes an earlier request. Ambiguous language stays with the conversational model rather than a broad keyword shortcut. ### Freshness is a server-verified requirement Background publication requires the exact current transcript key, including roles and trimmed contents. Even a short edit or appended text can reverse meaning without a correction keyword. Do not publish a completed analyst result for an earlier prefix. An already accepted, unexpired trusted snapshot may inform subsequent questions, but cannot authorize stopping for a newer transcript. The server seals analysis with SHA-256 of that transcript key, its catalog version, a 30-minute expiry, and an HMAC over the analysis payload using the server-only secret. Validate signature, version, expiry and source revision; a client field saying `current` is not authority. On explicit wrap, confirmation refresh, or a stale stop recommendation, analyze the complete latest input before constructing confirmation. Ordinary follow-ups remain independent. Derive canonical confirmation text from the current signed scope summary and provisional flag. Report submission verifies the signed source transcript before the confirmation/acceptance messages, the exact canonical confirmation text, and the visitor's explicit acceptance. Questions, hesitation, silence, and “yes, but…” are not acceptance. A valid signature protects integrity, not the analyst's reasoning quality. Expired/restored confirmation shows **Refresh direction**, retaining the conversation. Recheck expiry on a timer, tab focus, and the click itself. Reauthorize the discovery session when needed, then request current analysis and a fresh confirmation; never turn an expired acceptance into a report request. The branch's 45-minute access cookie and four-hour tab recovery are distinct from 30-minute analysis freshness. ### Review the whole deliverable After acceptance, synthesize a brief and perform a separate evidence review. Allow at most one schema repair and one evidence repair, each followed by validation. If the result still fails, preserve the conversation and provide a retry or exit. Never loop indefinitely or silently display an unchecked report. Before evidence review, assemble the readiness section, no-quote/provisional notices, and every unresolved visitor/expert/investigation item into the report. Give unresolved work report-local IDs. A repair must preserve exactly one substantive item for each ID, with meaning still addressing the original uncertainty; count equality alone is insufficient. Keep custom sections nonempty. Reserve readiness/work section names using case- and punctuation-normalized comparison and rename colliding custom headings before capturing the retention contract. Audit the complete assembled report, including analysis-derived prose. Repair that prose in place rather than reappending the original unchecked text. Deterministic retention checks enforce labels, notices and item identity; the evidence reviewer separately checks meaning. Remove work markers only after validation. The branch allows ten custom sections plus two discovery sections and carries them into both HTML and PDF. Expose actual work stages and cancellation. Do not invent completion percentages. For hosting with short request limits, use a bounded background job and polling/status stream, verify job ownership on every read/cancel, and prevent late results from reviving canceled jobs. ## 5. Add voice as another input/output mode Maintain distinct states for listening, endpoint candidate, checking turn completion, committed answer, speaking, paused, error, and finished. Tie UI cues to actual microphone/playback state. Use streaming final and interim transcription plus local speech activity where available. An acoustic endpoint indicates possible turn completion, not approval of a recommendation or sufficient discovery. Resumed speech or a revised transcript invalidates stale turn assessments. A bounded semantic check may resolve ambiguous endpoints; explicit thinking pauses keep the floor open. Silence after a check-in does not submit automatically. Provide text fallback and an explicit microphone control. Stop tracks and pending work when voice is disabled or the page exits. Prevent overlapping playback. Establish manual interruption first; add automatic barge-in only with real acoustic testing. Echo cancellation and transcript heuristics alone do not prove reliable interruption detection. Speak the same committed question or answer shown on screen, optionally preceded by a short grounded acknowledgment. Ensure any acknowledgment reflects the final visitor input. Handle autoplay restrictions and TTS/STT errors without losing the visitor's text or disabling the text workflow. ## 6. Bound independent analysis and voice preparation ### Background analysis scheduling Keep one analyst request in flight and only the latest queued transcript. Sample continuous speech rather than restarting a debounce on every token. In the reference, uncommitted drafts need at least eight words to start paid analysis, partial starts are spaced by six seconds, and a rolling browser budget permits ten starts per minute. Short edits still invalidate stale results even when they cannot start new work yet. A committed turn can accelerate its own latest queued revision within the rolling budget. Bind that exception to its revision; matching pending/completed work consumes it, and it cannot accelerate the next speculative draft. Do not duplicate browser analysis when the server is already catching up at a stop/refresh boundary. Discard superseded results, coalesce changes, and avoid infinite same-input retries. Ordinary cancellation retains the last-completed deduplication marker; full Start over clears it so an identical opening can be analyzed again. Both retain rolling usage controls. Analysis generation and provider fallback share a 39-second budget inside the production 45-second HTTP deadline, leaving room for the conditional 1.8-second resolution-evidence check and serialization. These are branch-specific limits, not latency guarantees. Final readiness catch-up can take longer than a normal conversational turn; show **Reviewing your direction**, not report-generation progress before acceptance. ### Question and audio preparation The conversation lane can prepare one response during stable interim speech. Prepared text/audio remain private until commit. Exact final-answer matches can reuse validated preparation; changed answers need semantic reuse validation or a new question. A consequential correction invalidates dependent work. A prepared confirmation cannot use this ordinary-question shortcut: it requires current signed analysis. The branch gives matching in-flight question preparation up to 1,000 ms at commit; eligible partial preparation gets 300 ms. The separate question-preparation rolling budget remains six starts per minute. These limits are independent of the analyst's ten-start budget. Adopt matching in-flight TTS into real playback instead of canceling it after 220 ms and issuing duplicate generation. Prepared speech may refresh after one second, with a bounded four-entry cache. Cancel mismatched/stale audio and use a fresh bounded path on failure. Routine prerecorded backchannels are no longer fetched or played; old assets remain unused. Use the visible thinking state and optional grounded conversational reflection. Measure speech-end to first meaningful audio, interruptions, question relevance, agenda accuracy, report quality and actual provider spend together. Canceled requests may still be billed. Fast synthetic question generation, cached reuse, or a single local sample is not a production audio percentile. Sub-two-second meaningful audio is an optimization target in the branch notes, not an SLA. ## 7. Set explicit access, data, and ownership limits Before public use, enforce server-side request size limits, session expiry, verified origins where applicable, shared atomic quotas, provider timeouts, and a usage kill switch. Process-local counters alone are insufficient across scaled workers. Quota units approximate usage; they are not an exact provider spending cap. Bound provider usage separately where available and measure actual costs. The branch reserves separate per-minute allowances per verified session: 16 speculative `analyze`/`prepare` requests and 24 committed response/report submissions. Production uses conditional shared-store counters; development uses bounded in-memory counters. Aggregate IP controls and session/global spend ceilings still apply. Client scheduling limits cannot replace these server limits, and analysis adds model spend even when it hides perceived waiting time. Disclose which providers process audio and text. Avoid storing raw audio by default. Define browser recovery, server job retention, deletion, log content, and provider retention separately. Never describe temporary server storage as “nothing leaves your browser.” Session identifiers and a human check are not identity verification. For a new adaptation, allow useful discovery without mandatory phone capture unless the job actually requires it. Collect follow-up details at the point of follow-up. Separate transactional delivery from optional marketing consent. These are recommended defaults; the current Otto contact flow differs. Assign an owner for model/provider changes, dependency updates, evaluation failures, abuse spending, privacy disclosures, and real-user issues. Record that owner and operating assumptions in the project README. ## 8. Acceptance scenarios | Scenario | Required observable behavior | | --- | --- | | “We want to automate… I'm not sure where to start” | Asks for a concrete example without inventing a scope | | “I don't know” | Preserves an unknown or asks an easier context question | | “What do you mean?” | Clarifies instead of treating it as the requested business fact | | “Update records—actually, only suggest changes” | Removes/invalidates the earlier action assumption and any prepared response depending on it | | Long thinking pause | Does not treat silence as confirmation or force report generation | | “Yes, but we cannot share customer data” at confirmation | Returns to discovery or revised confirmation; no obsolete report | | Exact final transcript with prepared response | Reuses only the current validated candidate; plays it once | | Revised transcript during a pending endpoint check | Stale assessment cannot commit the newer answer | | Unsupported quote or malformed report | Bounded repair/review, then recovery if still invalid | | Provider timeout or microphone denial | Preserves input and keeps a usable text/retry path | | Cancel during report generation | Late completion cannot replace the canceled state | | Different session attempts to retrieve a report | Access denied without exposing another session's data | | Quota exceeded | Stops paid processing and shows a recoverable usage message | | Quotes/HTML-like text in input | Rendered as text, not executable markup | | Voice disabled or page exited | Microphone tracks stop and pending work is canceled | | Analyst takes longer than the next conversational response | Follow-up proceeds; background arrival never replaces the already spoken question | | A brief transcript edit without a correction keyword | Earlier in-flight analysis is discarded; new text is not certified by old readiness | | A genuine quote does not answer the agenda uncertainty | Independent support check keeps the item open and removes unsupported resolution text | | Technical design question labeled visitor-owned | Reassigned to expert work; no technical homework for the visitor | | Blueprint understood while integration feasibility is unresolved | Blueprint can be ready while scope/quote remain blocked with consistent reasons | | Visitor requests an early provisional brief | Fresh analysis and explicit confirmation; remaining gaps preserved | | “Wrap up later” or “wrap up—actually, let's continue” | No immediate wrap shortcut or obsolete stop authorization | | Tampered payload, expired analysis, changed source, or edited confirmation | Report authorization rejected; answers retained for refresh/recovery | | Confirmation expires in a background tab and access cookie also expires | Click rechecks expiry/access, refreshes direction, then requests new acceptance | | Commit matches already pending/completed analysis | No immediate-start allowance leaks into the next speculative revision | | Start over with the same opening answer | Analysis can run again; rolling rate limits are retained | | Matching in-flight TTS is still loading | Adopted for one playback without duplicate synthesis or routine filler | | Report repair preserves item count but duplicates/replaces unresolved work | Identity/semantic checks reject it; nothing silently disappears | | Custom heading collides with a reserved readiness/work title | Custom content renamed, retained and audited alongside injected sections | | Evidence repair tries to delete readiness labels or provisional/no-quote notice | Report fails validation rather than being released | | Sales question about a documented integration | Supported answer and source link; no mandatory discovery interview | | Missing price or conflicting product documentation | Honest uncertainty; no fabricated price or commitment | | Knowledge changes while an answer is prepared | Old-version preparation cannot be presented as current | | Product knowledge includes instructions to ignore policy | Content cannot override application behavior or expose private material | | Visitor's needs are outside the offering | Explains the mismatch and allows a useful exit | Run deterministic state/schema tests and mocked integration/browser tests first. They do not establish voice quality. After authorization for provider usage, test real conversations across supported browsers, speakers/headphones, accents, network conditions, and background noise. Record failures as well as successes. Test the public hosting path, not only localhost. ## 9. Deliver the implementation with evidence Return the working local text flow, optional voice flow, sample brief, module map, configuration instructions without secrets, and acceptance results. Separate implemented behavior, mock-tested behavior, live-tested behavior, and remaining work. Measure useful-brief completion and human-rated faithfulness alongside latency, failure/recovery rate, and cost per completed useful brief. Include STT duration, LLM tokens/preparations/retries, TTS generation, hosting/storage, and unsuccessful sessions. Do not report cheap cached playback as end-to-end speed or total operating cost. Before presenting a public release as ready, obtain the owner's review of the real experience, usage/data limits, and deployment. A coding agent following this document does not remove the need for that ownership. ## Branch source map and verification status All paths below refer to the pinned branch commit above, not the older application checkout containing these editorial drafts. Inspect them with `git show 874583e:` if the target branch is not checked out. | Source | Contract used here | | --- | --- | | `strategy/10-otto-living-discovery-agenda.md` | Architecture intent and documented limitations | | `src/lib/discovery-agent.ts` | Conversation/analysis prompts, catch-up, acceptance, complete-report review and retention | | `src/lib/discovery-agenda.ts`, `src/lib/discovery-types.ts` | Agenda ownership, evidence, wrap rules, readiness and source types | | `src/lib/analysis-proof.ts` | Transcript revision, HMAC, catalog binding and expiry | | `src/lib/discovery-analysis-lane.ts` | Coalescing, exact-revision publication, commit allowance and reset behavior | | `src/lib/discovery-session.ts` | Independent requests, preparation grace and committed-turn orchestration | | `src/scripts/discovery-prototype.ts` | Background publication, confirmation refresh, TTS adoption and filler removal | | `src/lib/discovery-model.ts` | Independent model routing and provider timeouts | | `src/lib/discovery-rate.ts`, `src/pages/api/discovery.ts` | Separate verified-session rate buckets and request deadline | | `tests/discovery-agenda.test.mjs`, `tests/discovery-report.test.mjs`, `tests/discovery-rate.test.mjs` | Existing regression scenarios for these contracts | | `tests/e2e/otto.spec.ts`, `tests/e2e/otto-mobile.spec.ts` | Browser scenarios for analysis independence, audio and recovery | This documentation update inspected branch source and regression scenarios; it did not rerun application tests or an independent build of this guide. The branch's recommended application checks are `npm test`, `npm run build`, `npm run test:ui`, and `node scripts/check-deploy-secrets.mjs`, followed by appropriate live/manual acceptance. Its recorded local results do not establish production deployment or real-device audio performance. ## Official starting references - [Soniox endpoint detection](https://soniox.com/docs/stt/rt/endpoint-detection) - [Vapi web and phone voice agents](https://docs.vapi.ai/quickstart/introduction) - [Retell voice-agent platform](https://docs.retellai.com/general/introduction) - [LiveKit telephony](https://docs.livekit.io/telephony/) - [Pipecat framework](https://docs.pipecat.ai/overview/introduction) These links were checked September 5, 2026. Recheck provider APIs, usage terms, and supported models during implementation. This draft does not grant a license to the existing LaunchTap source code.