AI/TLDR

Multimodal AI · TRACK 02/04

Speech & Voice

Whisper-style transcription, neural voices, and the realtime voice agent stack.

13 ARTICLESbeginner → intermediate
// THE TRACK
01 · START HERESpeech-to-Text (ASR)Understand how modern ASR models like Whisper convert audio into text, and transcribe your first recording with a few lines of code.BEGINNER
AI Text-to-SpeechUnderstand how modern neural TTS works, what separates ElevenLabs Flash from OpenAI TTS, and how to add lifelike voice to any app in minutes.BEGINNERNeural Voice SynthesisLearn how neural TTS generates natural-sounding voices, what makes them sound human, and which APIs and open models to start with.BEGINNERVoice CloningLearn how models clone a voice from a short sample, the legitimate uses, and the consent, legal, and fraud lines you should never cross.INTERMEDIATESpeaker DiarizationUnderstand how diarization separates speakers in audio, why overlapping speech breaks it, and how to add speaker labels to your transcripts.INTERMEDIATERealtime Voice APIsUnderstand what a realtime voice API is, how streaming audio works, and why it powers natural talk-back voice agents.INTERMEDIATESpeech-to-Speech ModelsLearn how end-to-end speech-to-speech models work and why skipping the text middleman preserves tone, emotion, and timing.INTERMEDIATEReducing TTS LatencyLearn where latency hides in a voice agent and how streaming and chunking make AI speech feel instant.INTERMEDIATEOpenAI WhisperYou will understand what Whisper is, how it transcribes and translates speech, and why it became the open baseline for speech-to-text.INTERMEDIATEfaster-whisperYou will understand what faster-whisper is, how a CTranslate2 runtime speeds up Whisper with less memory, and how it pairs with WhisperX for timestamps and diarization.INTERMEDIATEElevenLabsYou will understand what ElevenLabs is, how its text-to-speech and voice cloning work, and why it is the market leader for AI voice.BEGINNERKokoro TTSYou will understand what Kokoro TTS is, why a tiny open-weight model is a breakout, and how it runs text-to-speech on modest edge hardware.INTERMEDIATEOpenAI Realtime APIYou will understand what the OpenAI Realtime API is, how speech-to-speech works in a single model, and how it powers low-latency voice agents that can call tools.INTERMEDIATE