How We Built a Bilingual AI Voice Assistant in Laravel (Arabic + English) — Part 1 of 4

The Voice Assistant Problem

A customer opens the app, taps the mic, and says in Arabic: "أبي رحلة للمطار" ("I want a ride to the airport").

Three things need to happen in about 2 seconds:

1. The audio gets converted to text.
2. The text gets sent to an AI that understands the request and calls the right booking tool.
3. The AI's response gets converted back to audio and played back.

Each step requires a different specialist API. No single service does all three well — especially in Arabic with Gulf dialect.

This 4-part series walks through exactly how we built this in a Laravel 12 modular app using:

• Deepgram Nova-3 — speech-to-text, Arabic + English code-switching
• Anthropic Claude Sonnet — reasoning and action through tool use
• Cartesia Sonic-3 — text-to-speech at ~40ms first-chunk latency

Why These Three Providers?

Deepgram was the only provider with a model that handles Gulf dialect reliably (UAE, Saudi, Kuwaiti accents) and supports multi-language mode — so a user can switch between Arabic and English mid-sentence without re-configuring anything.

Anthropic Claude is the best model for structured tool use. In a booking assistant, the AI doesn't just answer questions — it calls real functions: get_fare_estimate, book_ride, cancel_booking. Claude's tool use API is reliable, well-documented, and works correctly with a two-turn pattern (more on that in Part 3).

Cartesia Sonic-3 produces natural-sounding Arabic voice at ~40ms first-chunk latency — fast enough to feel responsive in a mobile app. Google TTS and Amazon Polly both have noticeably more robotic Arabic output.

The Full Pipeline

Flutter App
    │
    │  POST /api/v1/customer/assistant/chat
    │  { audio_base64, module_context, session_id, location_context }
    ▼
AssistantController::chat()
    │
    ├─ 1. Rate limit check
    ├─ 2. Get or create session
    │
    ├─ 3. If voice: SpeechToTextService → transcript + language detected
    │     If text:  detect language from Unicode ranges
    │
    ├─ 4. Save user message to DB
    │
    ├─ 5. Build Claude context (user name, GPS, active booking, last fare)
    ├─ 6. Load conversation history (last 10 turns)
    │
    ├─ 7. LanguageModelService::complete()
    │       └─ If Claude calls a tool:
    │             ├─ Execute tool (fare estimate, book ride, track driver...)
    │             └─ LanguageModelService::resumeWithToolResult()
    │
    ├─ 8. TextToSpeechService::generateSpeech() → MP3 audio base64
    │
    ├─ 9. Save assistant message to DB
    ├─ 10. Log costs (STT/LLM/TTS) + increment rate limit counters
    │
    └─ Response: { response_text, audio_base64, transcript, action_taken, session_id }

One request, one response, everything in sequence. There is no streaming in the current implementation — the Flutter app shows a loading indicator, then plays the full audio when it arrives.

Database Schema

Eight migrations power the system. Here's what each table stores:

The Config File

At Modules/Assistant/config/assistant.php. All API keys come from .env — never hardcode them.

<?php

return [
    'deepgram' => [
        'api_key'  => env('DEEPGRAM_API_KEY'),
        'base_url' => 'https://api.deepgram.com/v1',
        'model'    => 'nova-3',
        'language' => 'multi',       // handles Arabic + English code-switching
        'options'  => [
            'smart_format'  => true, // auto punctuation
            'utterances'    => true, // sentence boundary detection
            'endpointing'   => 300,  // ms silence before finalising
            'keyterms'      => [     // vocabulary boost for ride-booking
                'حجز', 'مطار', 'Downtown', 'Marina', 'cancel', 'track',
                'تاكسي', 'رحلة', 'السايق', 'الغي',
            ],
        ],
        'price_per_second_usd' => 0.00011944,
    ],

    'anthropic' => [
        'api_key'       => env('ANTHROPIC_API_KEY'),
        'base_url'      => 'https://api.anthropic.com/v1',
        'model'         => env('ASSISTANT_LLM_MODEL', 'claude-sonnet-4-6'),
        'max_tokens'    => 1024,
        'temperature'   => 0.3,
        'api_version'   => '2023-06-01',
        'price_per_input_token_usd'  => 0.000003,
        'price_per_output_token_usd' => 0.000015,
    ],

    'cartesia' => [
        'api_key'  => env('CARTESIA_API_KEY'),
        'base_url' => 'https://api.cartesia.ai',
        'model'    => 'sonic-3',
        'voices'   => [
            'ar' => env('CARTESIA_ARABIC_VOICE_ID'),
            'en' => env('CARTESIA_ENGLISH_VOICE_ID'),
        ],
        'output_format' => [
            'container'   => 'mp3',
            'bit_rate'    => 128000,
            'sample_rate' => 44100,
        ],
        'price_per_char_usd' => 0.000065,
    ],

    'rate_limits' => [
        'minute' => ['max_requests' => 10, 'max_voice_requests' => 5,   'max_cost_usd' => 0.50],
        'hour'   => ['max_requests' => 60, 'max_voice_requests' => 30,  'max_cost_usd' => 5.00],
        'day'    => ['max_requests' => 200,'max_voice_requests' => 100, 'max_cost_usd' => 20.00],
    ],

    'session' => [
        'expiry_minutes' => 30,
        'max_turns'      => 20,
        'max_audio_sec'  => 30,
    ],

    'context' => [
        'max_history_turns' => 10,
    ],
];

.env keys to add:

DEEPGRAM_API_KEY=your_deepgram_key
ANTHROPIC_API_KEY=your_anthropic_key
CARTESIA_API_KEY=your_cartesia_key
CARTESIA_ARABIC_VOICE_ID=your_arabic_voice_id
CARTESIA_ENGLISH_VOICE_ID=your_english_voice_id
ASSISTANT_LLM_MODEL=claude-sonnet-4-6

The Input DTO

The controller receives one DTO that normalises all inputs — whether from voice or text. Place at Modules/Assistant/app/DTOs/AssistantMessageDTO.php:

<?php

namespace Modules\Assistant\DTOs;

use Modules\Assistant\Enums\InputTypeEnum;
use Modules\Assistant\Enums\LanguageEnum;
use Modules\Assistant\Enums\ModuleContextEnum;
use Modules\Assistant\Http\Requests\AssistantChatRequest;

class AssistantMessageDTO
{
    public function __construct(
        public readonly int               $userId,
        public readonly InputTypeEnum     $inputType,       // voice | text
        public readonly ModuleContextEnum $moduleContext,   // booking | cargo | general
        public readonly ?string           $sessionId,       // null = start new session
        public readonly ?string           $audioBase64,     // set when inputType = voice
        public readonly ?string           $text,            // set when inputType = text
        public readonly LanguageEnum      $language,        // ar | en | mixed
        public readonly ?string           $contextEntityId,
        public readonly ?string           $contextEntityType,
        public readonly ?array            $sessionMetadata,
    ) {}

    public static function fromRequest(AssistantChatRequest $request): self
    {
        // If audio is present, it's a voice message regardless of what else was sent
        $inputType = $request->filled('audio_base64')
            ? InputTypeEnum::Voice
            : InputTypeEnum::Text;

        return new self(
            userId:            $request->user()->id,
            inputType:         $inputType,
            moduleContext:     ModuleContextEnum::from($request->input('module_context', 'general')),
            sessionId:         $request->input('session_id'),
            audioBase64:       $request->input('audio_base64'),
            text:              $request->input('text'),
            language:          LanguageEnum::from($request->input('language', 'ar')),
            contextEntityId:   $request->input('context_entity_id'),
            contextEntityType: $request->input('context_entity_type'),
            sessionMetadata:   $request->input('session_metadata'),
        );
    }
}

What's Next

Part 2 dives into the Deepgram integration — how we configure Nova-3 for Arabic Gulf dialect, how multi-language mode handles code-switching between Arabic and English, and how we normalise the detected language before passing it to the rest of the pipeline.