fix(amd): start SIP detection timeout after answer - #2226
fix(amd): start SIP detection timeout after answer#2226rosetta-livekit-bot[bot] wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 5df65ae The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const trackGateAbort = this.trackGateAbort; | ||
| const publicationTimeout = setTimeout(() => { | ||
| if (this.trackGateAbort !== trackGateAbort || this.settled) return; | ||
| this.settleParticipantMissing('timed out waiting for participant audio track'); | ||
| trackGateAbort.abort(); | ||
| }, TRACK_PUBLICATION_TIMEOUT_MS); |
There was a problem hiding this comment.
🔴 Outbound calls can be abandoned as "no participant" five seconds after dialing starts
The five-second budget for the caller's audio to arrive is started when detection begins (setTimeout(...) at agents/src/voice/amd.ts:652-656) instead of after the call is answered, so a call that is still ringing is given up on and reported as having no participant.
Impact: Outbound calls that ring for more than a few seconds are classified as uncertain/participant_missing and the agent stops listening even though the callee later answers.
Ordering of track-publication bound vs. SIP answer gate
The PR's stated design is "start AMD before creating a SIP participant so no audio is missed" and "the detection timeout begins only after listening starts; SIP settings bound the pre-answer phase". However gateListening() (agents/src/voice/amd.ts:631-716) arms TRACK_PUBLICATION_TIMEOUT_MS = 5_000 immediately at execute() time and only waits for the SIP sip.callStatus === active attribute after the publication resolves. So the 5s bound is applied to the pre-answer phase, which is exactly what the fix intends to leave unbounded (Python bounds track publication only after the call is answered).
In examples/src/telephony_amd.ts:105 detector.execute() is invoked before sip.createSipParticipant(..., { waitUntilAnswered: true, timeout: 45 }) (examples/src/telephony_amd.ts:131-136). If the SIP participant does not join and get its audio track subscribed within 5s (trunk setup latency, no early media until answer, ring window up to 45s), settleParticipantMissing('timed out waiting for participant audio track') fires and the whole AMD run resolves as uncertain/participant_missing while the call is still ringing. The abort also tears down the track listener, so late audio is never observed.
Note the previous code armed detectionTimer in execute() as a global backstop; that backstop was removed here (startDetectionTimer() now only runs from startListening()), so the 5s publication timer is the only thing bounding the pre-answer phase.
Prompt for agents
In agents/src/voice/amd.ts gateListening(), the 5-second TRACK_PUBLICATION_TIMEOUT_MS timer is started as soon as execute() runs, i.e. before the SIP call is placed/answered. Since AMD is now documented (and used in examples/src/telephony_amd.ts) to be started before createSipParticipant, this bound elapses while the call is still ringing and settles the run as uncertain/participant_missing. The upstream Python fix bounds track publication only after the answer gate. Restructure the gate so the pre-answer phase is unbounded (relying on participant disconnect / SIP dial timeout to settle participant_missing), and only apply the 5s publication budget once the target participant exists and, for SIP participants, once sip.callStatus is active — e.g. wait for the participant to join, then wait for the SIP answer attribute, then wait (bounded) for the subscribed audio publication before startListening().
Was this helpful? React with 👍 or 👎 to provide feedback.
| const detector = new voice.AMD(session, { | ||
| participantIdentity, | ||
| }); | ||
| const detection = detector.execute(); |
There was a problem hiding this comment.
🟡 Example agent can crash with an unhandled error while a call is being dialed
The detection run is kicked off and left without any error handler (detector.execute() at examples/src/telephony_amd.ts:105) while the code awaits the outbound dial, so a failure during that window becomes an unhandled error that can terminate the process.
Impact: If detection fails (for example the session closes or the classifier errors) while the phone is still ringing, the example agent process can crash instead of logging the failure.
Unhandled rejection window between execute() and the awaits
const detection = detector.execute(); at examples/src/telephony_amd.ts:105 has no attached handler until await detection at line 167 or the finally at line 195. The intervening await sip.createSipParticipant(...) can take up to 45 seconds. AMD.execute() rejects on aclose() (AMD closed) and on LLM classification errors (scheduleLLMClassification → rejectRun). A rejection occurring during that window has no handler at the time Node checks the microtask queue, producing an unhandledRejection (fatal by default in modern Node).
Attaching a no-op catch (or a logging catch) immediately after creating the promise avoids this while still allowing await detection later.
| const detection = detector.execute(); | |
| const detection = detector.execute(); | |
| // Keep a handler attached while we dial so an early failure isn't an unhandled rejection. | |
| detection.catch(() => {}); |
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6580 to the JS AMD implementation.
Summary
uncertainwithparticipant_missingwhen participant audio never arrives or the participant disappearsSource diff coverage
examples/telephony/amd.py: adapted toexamples/src/telephony_amd.ts. Registers room cleanup before dialing, starts AMD before creating the SIP participant, uses the JS SDKtimeout: 45option, handles call failure, and resolves the answered participant directly fromroom.remoteParticipants.livekit-agents/livekit/agents/utils/participant.py: ported toagents/src/utils.ts.waitForTrackPublicationrejects when the requested participant disconnects and removes the listener during cleanup.livekit-agents/livekit/agents/voice/amd/classifier.py: adapted intoagents/src/voice/amd.tsbecause JS combines classifier and detector. The fallback settlement path is generalized and supports immediate settlement before listening.livekit-agents/livekit/agents/voice/amd/detector.py: adapted intoagents/src/voice/amd.ts. Detection timing starts instartListening; track publication is bounded to 5 seconds; track timeout, participant disappearance, and SIP answer failure settleparticipant_missing.tests/test_amd_classifier.py: ported toagents/src/voice/amd.test.ts. Source-equivalent regressions cover timer arming, pre-track disconnect, publication timeout, post-subscription disappearance, SIP answer failure, and pre-listening settlement.No source files were omitted and no target infrastructure gap remains.
Validation
pnpm test agents/src/voice/amd.test.ts(36 passed)pnpm build(40/40 workspace tasks passed)pnpm lint(passed; existing warnings only)cue-clivoice-mode run emittedamd_predictionwithAMD_HUMANfor a live-person utteranceFull touched-package suites were also run. They reproduce failures already present on
mainand unrelated to this port:pnpm test agents: 1582 passed, 5 skipped, 2 failed inremote_session.test.ts;mainforwardsagent_false_interruptionbut its expected event set omits it.pnpm test examples: 95 passed, 2 skipped, 2 failed in existing task/judge flows, plus 3 unrelated leaked async errors.Ported from livekit/agents#6580
Original PR description
Carrier early media can publish audio before answer, exhausting AMD’s detection budget during ringback and misclassifying late answers.
Start the budget only when listening begins, and settle immediately if the participant disappears before audio arrives. This builds on #6386 by @dexhunter, credited as a commit co-author.
Fixes livekit/agents#6187. Closes livekit/agents#6187.