Show your agent's mascot in a Flutter app with Wixzel Voice

What do you need before you start?

• A Wixzel Voice agent and an API key with credit, kept on your server. Any agent works: every agent has a mascot, starting with the default green one.
• Flutter 3.27 or newer.
• A server you control that can call the API. The example uses Node.js and the TypeScript SDK.
• A recording plugin that streams 16-bit PCM, ideally 8 kHz mono with echo cancellation, and a player that accepts raw PCM, such as record and flutter_pcm_sound.

How do you build this on Wixzel Voice, step by step?

1. 01Add the packages

   wixzel_voice_flutter draws the mascot; wixzel_voice is the Dart SDK it builds on:

   pubspec.yamlCopy

   dependencies:
     wixzel_voice: ^0.8.0
     wixzel_voice_flutter: ^0.1.0
     http: ^1.2.0

   The package depends on sensors_plus, so the mascot's eyes can follow the phone's tilt, and on visibility_detector, so it animates only while it is on screen. It does not record or play audio.

2. 02Give the app the agent and a session, from your server

   Your API key never goes in the app. Your server fetches the agent and mints a one-minute, single-use session, and the app asks your server for both:

   server.mjsCopy

   import express from 'express';
   import { WixzelVoice } from 'wixzel-voice';
   
   const wixzel = new WixzelVoice({ apiKey: process.env.WIXZEL_API_KEY });
   const app = express();
   const AGENT_ID = process.env.AGENT_ID;
   
   // Put your own sign-in check in front of both routes.
   app.get('/agent', async (req, res) => {
     res.json(await wixzel.agents.retrieve(AGENT_ID));
   });
   
   app.post('/session', async (req, res) => {
     res.json(await wixzel.realtime.createSession({ agent_id: AGENT_ID, max_duration_seconds: 300 }));
   });
   
   app.listen(3000);

   The agent JSON carries mascot.spec, the small spec the mascot is drawn from. Minting a session costs nothing; billing starts when the app connects.

3. 03Draw the mascot

   Agent.fromJson reads what your server sent, and AgentMascotView draws it:

   agent_face.dartCopy

   import 'dart:convert';
   
   import 'package:flutter/widgets.dart';
   import 'package:http/http.dart' as http;
   import 'package:wixzel_voice/wixzel_voice.dart';
   import 'package:wixzel_voice_flutter/wixzel_voice_flutter.dart';
   
   const backendUrl = 'https://api.example.com';
   
   Future<Agent> loadAgent() async {
     final res = await http.get(Uri.parse('$backendUrl/agent'));
     return Agent.fromJson(jsonDecode(res.body) as Map<String, dynamic>);
   }
   
   Widget agentFace(Agent agent, MascotController controller) =>
       AgentMascotView(agent: agent, controller: controller, size: 220);
   
   // In a list, draw one frame and run no animation:
   Widget agentRow(Agent agent) => AgentMascotView(agent: agent, size: 40, still: true);

   An agent with no spec gets the default look. A near-black or near-white mascot stands on a stage of its own colour, so it does not disappear into a dark or light screen; turn that off with stage: false. A tap plays an expression, and MascotController plays moves by name, such as mascot.play('hello').

4. 04Link the mascot to a call

   Connect with the Dart SDK's RealtimeConnection and hand it to MascotRealtimeLink. The link makes the mascot think while the call connects, wave hello, listen while the user talks, and move its mouth with the agent's audio:

   talk.dartCopy

   import 'dart:async';
   import 'dart:convert';
   import 'dart:typed_data';
   
   import 'package:http/http.dart' as http;
   import 'package:wixzel_voice/wixzel_voice.dart';
   import 'package:wixzel_voice_flutter/wixzel_voice_flutter.dart';
   
   /// Your audio plugins, behind a small interface.
   abstract class PcmPlayer {
     void add(Int16List samples); // 16-bit PCM, 8 kHz mono
     void clear();
   }
   
   class MascotCall {
     MascotCall(this.mascot, this.player);
   
     final MascotController mascot;
     final PcmPlayer player;
     RealtimeConnection? _conn;
     MascotRealtimeLink? _link;
     StreamSubscription<Int16List>? _mic;
   
     /// [micFrames] is 16-bit PCM at 8 kHz in 160-sample (20 ms) frames.
     Future<void> start(Stream<Int16List> micFrames) async {
       final res = await http.post(Uri.parse('https://api.example.com/session'));
       final session = RealtimeSession.fromJson(jsonDecode(res.body) as Map<String, dynamic>);
   
       final conn = RealtimeConnection.forSession(session);
       _conn = conn;
       _link = MascotRealtimeLink(mascot, conn);
       conn.events.listen((event) {
         switch (event) {
           case RealtimeAudio(:final mulaw):
             player.add(mulawToPcm16(mulaw)); // the mouth follows this audio through the link
           case RealtimeAudioClear():
             player.clear(); // the user spoke over the agent
           case RealtimeEnded():
             unawaited(end());
           default:
         }
       });
       await conn.ready;
       _mic = micFrames.listen((frame) {
         conn.sendAudio(pcm16ToMulaw(frame));
         _link?.micAudio(frame); // it leans in while the user talks
       });
     }
   
     Future<void> end() async {
       await _mic?.cancel();
       await _link?.dispose(); // leaves the mascot idle; does not end the call
       await _conn?.end();
       _mic = null;
       _link = null;
       _conn = null;
     }
   }

   Show it with AgentMascotView(agent: agent, controller: mascot), the same controller the call drives. Dispose the controller when the screen goes away.

   Turn on the platform's echo cancellation in your recorder, or the agent hears itself through the speaker and interrupts itself.

5. 05Make the lip-sync exact, if your player schedules audio

   By default the link assumes your player starts each chunk about 120 ms after it arrives. If your player knows exactly when a chunk plays, tell the link instead:

   exact_timing.dartCopy

   import 'dart:typed_data';
   
   import 'package:wixzel_voice/wixzel_voice.dart';
   import 'package:wixzel_voice_flutter/wixzel_voice_flutter.dart';
   
   MascotRealtimeLink exactLink(MascotController mascot, RealtimeConnection conn) =>
       MascotRealtimeLink(mascot, conn, estimatePlayback: false);
   
   // In your player, each time it schedules a chunk, on the player's own clock:
   void scheduled(MascotRealtimeLink link, Int16List pcm, double playAt, double now) {
     link.agentAudio(pcm, playAt: playAt, currentTime: now, sampleRate: 8000);
   }

What does this build cost on Wixzel Voice?

Which Wixzel Voice limits apply to this build?

• The package does not record or play audio. Your app supplies both through plugins of its choice.
• Phone-quality audio: G.711 µ-law at 8 kHz in both directions.
• Sessions count toward the account's concurrent-call limit, five by default.
• Flutter 3.27 or newer. Tilt needs an accelerometer; without one the mascot looks around on its own.
• Changing a mascot's look is an API call from your server (agents.updateMascot), not from the app, because it needs your API key.