You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c970d63
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: fern/observability/simulations-advanced.mdx
+24Lines changed: 24 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -318,6 +318,29 @@ Variables provide values for the [**dynamic variables**](/assistants/dynamic-var
318
318
319
319
Use variables to test specific inputs, such as a customer name or account tier, without editing the assistant or squad.
320
320
321
+
### Simulate a call from a specific phone number
322
+
323
+
Assistants that identify callers by phone number, for example by passing `{{customer.number}}` to an account lookup tool, need a known caller number to test against. Set `customer.number` in the scenario's `targetOverrides.variableValues`. The **Variables** tab only accepts flat names, so set this value through the API:
For a transient scenario, set the same `targetOverrides` on the inline `scenario` in `POST /eval/simulation/run`.
339
+
340
+
- The value replaces the whole `customer` variable, so also set any other `customer` fields the assistant references, such as `customer.name`.
341
+
- Only `{{customer.number}}` changes. The `call.customer` object in server messages does not, so a tool that reads the caller from the message payload won't see the override.
342
+
- To test the conversation without a real lookup, [mock the tool](#mock-tool-responses) instead.
343
+
321
344
## Troubleshooting
322
345
323
346
| Issue | What to check |
@@ -328,6 +351,7 @@ Use variables to test specific inputs, such as a customer name or account tier,
328
351
| There is no audio or recording | Check whether the run used chat mode. Use voice mode to test or record audio. |
329
352
| Start or end webhooks do not trigger | Confirm the corresponding hook and its URL are configured on the scenario. |
330
353
| A tool mock does not apply | Confirm that `toolName` matches the configured tool name exactly and that `enabled` is `true`. |
354
+
| A tool looks up the wrong caller | Set `customer.number` in the scenario's variable values. See [**Simulate a call from a specific phone number**](#simulate-a-call-from-a-specific-phone-number). |
331
355
| A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. |
332
356
| Latency limits show as skipped | Check whether the run used chat mode or the assistant under test uses GPT-Live. Both skip latency limits. |
0 commit comments