The goal
We wanted a real business phone line that never goes to a generic voicemail, costs little to run, and does useful work on the call instead of just taking messages. We also wanted to own it: the phone system, the call logic and the data.
How it works
A call to our public number arrives over a SIP trunk from a VoIP carrier and lands on a FreePBX/Asterisk phone system running in a virtual machine on our own server. A phone tree lets callers pick where to go, including a real desk phone and a set of AI agents.
When a call reaches an AI agent, Asterisk streams the audio to a set of containers on our server:
- Listen. Speech recognition tuned for phone audio turns the caller's words into text as they speak.
- Think. A language model (Anthropic's Claude) decides what to say and whether to take an action, following the agent's written instructions.
- Speak. A natural text-to-speech voice sends the reply back down the line.
The whole loop typically completes in well under a second once the system is warm, which is what makes the conversation feel natural rather than robotic.
Doing real work on the call
Talking isn't enough. The agents have a small set of tools they can use during a call:
- Transfer. Hand the caller to a person or another line. Transfers go through an allowlist, so the agent can only send calls to destinations we've approved. That's a toll-fraud guard as much as a feature.
- Take a message. Capture the details and send them to our automation platform (n8n), which files and forwards them.
- Hang up cleanly when the conversation is done.
- Control building devices. The internal line can turn lights and devices on and off through Home Assistant, by voice, over the phone.
What we learned
Latency is everything. A one-second pause feels natural. A three-second pause makes callers talk over the agent. Choosing fast models and streaming every step mattered more than any single feature.
Guardrails belong in the phone system, not just the prompt. The agent's instructions say where it may transfer calls, but the phone system enforces it too. Never trust a language model alone with an action that costs money.
Old hardware has opinions. Our desk phone is a vintage analog handset on an adapter. Its message-waiting lamp only works with one specific signaling type, which is the kind of detail that never shows up in a spec sheet.
Write down every step. Adding a new agent touches the containers, the configuration, the phone system's dial plan and its admin screens. Skipping one breaks things in confusing ways, so the process is documented as a checklist.
What this means for clients
The same building blocks power the AI phone agents and dispatch systems we build for businesses. Some clients run it on their own hardware; others prefer a hosted setup. Either way, the accounts are in your name and the configuration is documented.