In plain English
An AI meeting notes app takes a recording of a meeting and hands you back a clean, readable summary: what was decided, who agreed to do what, and the key points raised. You stop scribbling notes during the call and stop re-listening to a 50-minute recording afterwards. You drop in the audio file, and a few seconds later you have a tidy page of decisions and action items.
The trick is that no single model does the whole job. The work splits into two stages handled by two different kinds of model. First a speech-to-text (transcription) model listens to the audio and writes down every word that was said — turning sound into a wall of raw text. Then a language model reads that text and rewrites it into a short, structured summary.
Think of it like a court reporter plus an editor. The court reporter (speech-to-text) types out everything, word for word, including the rambling and the "ums." The editor (the LLM) then reads that full transcript and produces the version a busy person actually wants: the gist, the conclusions, the follow-ups. Neither could do the other's job, but chained together they turn a recording into notes.
Why it matters
Meetings produce a lot of talk and very little written record. People forget who promised what, decisions get lost, and someone has to re-watch the recording to find that one number. An app that reliably captures decisions and action items saves real time for an entire team, every single day. That is why "AI notetaker" products are now everywhere.
For a builder, this project is valuable for a deeper reason: it teaches you to chain two model types that work in completely different ways. A chatbot project only touches one model. Here you learn how the output of one model becomes the careful input of another, and all the messy real-world problems that live in between.
- Handling a different input type. Unlike a chatbot or a chat-with-PDF app, the raw input here is audio. You learn to accept an upload, send bytes to a transcription API, and get text back.
- Long inputs that overflow the prompt. A one-hour meeting can run 8,000–10,000 words. Long transcripts can blow past a model's context window, forcing you to split and recombine — a problem every serious AI app eventually faces.
- Structured output, not chat. You don't want a friendly paragraph; you want fields: a list of decisions, a list of action items with an owner. Coaxing a model into clean, parseable structure is a core production skill.
- A real, shippable product. Transcribe-and-summarize is the backbone of a category of commercial tools. Building a small version teaches the whole shape of the bigger one.
If you've already built a chatbot and a chat-with-PDF tool, this is the natural next step on the beginner project ladder: same APIs you already know, plus the two genuinely new skills — audio input and multi-stage chaining.
How it works
The whole app is a short, linear pipeline. Audio comes in one end, structured notes come out the other, and each box in between is one clear job you can build and test on its own.
Stage 1 — Transcription (audio to text)
You send the audio file to a speech-to-text service. It returns the spoken words as text. Good APIs can also add timestamps (when each line was said) and sometimes speaker labels ("Speaker 1", "Speaker 2") — a feature called diarization, which is gold for meeting notes because it lets you attribute decisions to a person. Popular options include OpenAI Whisper (runnable yourself or via API), AssemblyAI, and Deepgram.
Stage 2 — Summarization (text to structured notes)
Now you have a long transcript. You put it inside a prompt that tells the LLM exactly what to extract, and you ask for the result in a fixed shape — ideally JSON — so your app can render it reliably. The prompt is the heart of the app; spend your time here.
You are a meeting-notes assistant. Read the transcript and return JSON
with exactly these keys:
"summary": a 3-4 sentence overview
"decisions": a list of concrete decisions that were made
"action_items": a list of {"task": ..., "owner": ...} objects
Only use information present in the transcript. If an owner is not
stated, set "owner" to "unassigned". Do not invent anything.
Transcript:
<<< full transcript text goes here >>>The model reads the transcript and writes the JSON. Your app parses it and shows three neat sections. That instruction — only use information present in the transcript — is what keeps the model from inventing action items nobody agreed to.
Handling long transcripts that overflow the context window
A short stand-up fits in one prompt and you're done. A two-hour strategy session does not. When the transcript is bigger than what you can comfortably send in one call, you need a strategy to summarize it in pieces and stitch the pieces back together. The standard pattern is called map-reduce.
- Split. Cut the transcript into chunks that each fit the model comfortably (e.g. ~10–15 minutes of talk). Add a little overlap between chunks so a sentence split across a boundary isn't lost.
- Map. Summarize each chunk on its own, pulling out that chunk's decisions and action items.
- Reduce. Feed all those partial summaries into one final call that merges them, removes duplicates, and produces the single clean notes page.
A minimal pipeline in code
Here is the whole idea end to end in Python. It transcribes a file, then asks an LLM for structured notes. It's deliberately small — no web UI, no map-reduce — but it shows the two-stage chain that everything else builds on.
import json
from anthropic import Anthropic
client = Anthropic(api_key="sk-ant-...")
def transcribe(audio_path):
# Stage 1: audio -> text. Use any speech-to-text API here
# (Whisper, Deepgram, AssemblyAI). It returns the full transcript.
return speech_to_text(audio_path) # -> str
SYSTEM = (
"You are a meeting-notes assistant. Return JSON with keys "
"'summary' (string), 'decisions' (list of strings), and "
"'action_items' (list of {'task','owner'}). Use ONLY the "
"transcript. If an owner is unstated, use 'unassigned'."
)
def summarize(transcript):
# Stage 2: text -> structured notes.
msg = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1500,
system=SYSTEM,
messages=[{"role": "user",
"content": f"Transcript:\n{transcript}"}],
)
return json.loads(msg.content[0].text)
def meeting_notes(audio_path):
transcript = transcribe(audio_path) # stage 1
return summarize(transcript) # stage 2
notes = meeting_notes("standup.mp3")
print(notes["summary"])
for item in notes["action_items"]:
print(f"- {item['task']} ({item['owner']})")How this project compares to a chatbot or chat-with-PDF
All three beginner projects call an LLM, but each adds one new muscle. Seeing them side by side makes the meeting-notes app's specific lessons clear.
| Project | Input type | Models used | New skill it teaches |
|---|---|---|---|
| Chatbot | User text, turn by turn | One LLM | Calling an LLM and keeping chat history |
| Chat with PDF | A document + questions | Embeddings + LLM (RAG) | Retrieval: finding the right passage to answer from |
| Meeting notes | An audio recording | Speech-to-text + LLM | Chaining two model types; long-input map-reduce |
The defining difference is the audio front-end. Chat-with-PDF starts from text you can read directly; meeting notes starts from a sound file you must transcribe first. That extra stage is also the most common place for the pipeline to go wrong, because a bad transcript poisons everything downstream.
Going deeper
The two-stage pipeline above is the honest core. Everything below is about making it accurate, fast, and pleasant to use once the basics work.
Better transcripts beat a better summary prompt. Garbage in, garbage out applies twice over here. Use a model with diarization so each line carries a speaker, pass any known vocabulary (people's names, product names, acronyms) as a hint if your API supports it, and clean obvious filler before summarizing. A transcript that knows who said what lets the LLM assign action items to real owners instead of guessing.
Grounding and hallucination. Even with a perfect transcript, a summarizer can invent a decision or assign a task to the wrong person. Instruct the model to quote the supporting line for each action item, keep timestamps so a user can jump to the moment in the recording, and treat the output as a draft a human edits, not gospel. This is the same grounding discipline that RAG systems live by.
Latency and user experience. Transcribing an hour of audio takes real time, often longer than a user will happily stare at a spinner. Process in the background and show progress, or stream partial results as each chunk finishes. The patterns in designing for LLM latency apply directly — a slow pipeline feels fast when the UI keeps the user informed.
Cost. You pay twice: once for transcription (usually billed per minute of audio) and once for summarization (billed per token of transcript). Long meetings make both add up. Trim silence, cache transcripts so you never re-transcribe the same file, and pick a cheaper, faster model for the map step while reserving a stronger one for the final reduce.
Privacy. Meeting audio is sensitive — it can contain salaries, strategy, and personal data. Be explicit about which third-party APIs the audio passes through, whether they retain data, and whether you need consent from everyone recorded. For some users, a self-hosted transcription model (running Whisper locally) is the only acceptable option, even at the cost of speed.
Where to go next. Add a web UI for upload and display, store past meetings in a database, wire it to a calendar so it joins and records calls, or expose the transcript for follow-up questions. When you're ready to ship, weigh your deployment options — background transcription jobs change which hosting model fits best.
FAQ
How do I build an AI meeting notes app?
Build it as a two-stage pipeline. First send the meeting audio to a speech-to-text model (like Whisper, Deepgram, or AssemblyAI) to get a full transcript. Then pass that transcript to a language model with a prompt asking it to extract a summary, decisions, and action items as structured JSON. Your app renders that JSON as the finished notes.
Which is better for transcription, OpenAI Whisper or an API like Deepgram or AssemblyAI?
Whisper is open-source and can run on your own hardware, which is great for privacy and cost control but needs setup. Hosted APIs like Deepgram and AssemblyAI are faster to integrate, handle long-file chunking for you, and add features like speaker diarization out of the box. Start with a hosted API to learn, then consider self-hosting Whisper if privacy or cost demand it.
How do I summarize a meeting that is too long for the model's context window?
Use a map-reduce approach. Split the transcript into overlapping chunks that each fit the model, summarize every chunk on its own (the 'map' step), then feed all those partial summaries into one final call that merges and de-duplicates them (the 'reduce' step). If the whole transcript already fits the context window, just send it in one call — only chunk when you actually overflow.
How do I get the model to return action items in a structured format?
Tell it the exact shape you want in the prompt — for example, return JSON with a 'decisions' list and an 'action_items' list of task-and-owner objects — and instruct it to use only the transcript. Many APIs offer a JSON or structured-output mode that guarantees valid JSON. Always wrap the parse in error handling and re-ask once if parsing fails.
Do I need a vector database for a meeting notes app?
Not for the basic transcribe-and-summarize flow — you feed the transcript straight into the prompt. You'd add embeddings and a vector database (RAG) only if you want users to search across many past meetings or chat with a very long transcript that won't fit in one prompt.
How accurate are AI meeting summaries?
They're good enough to save real time but not perfect. Accuracy depends mostly on transcript quality — poor audio, crosstalk, or accents cause errors that flow into the summary, and the model can still mis-assign an action item. Treat the output as an editable draft, keep timestamps so users can verify against the recording, and ask the model to ground each action item in the transcript.