Audio summary

Audio Summary Skill

This skill uses the Gemini API to analyze audio files and produce friendly, natural-language summaries. Gemini understands speech, identifies multiple speakers, detects non-speech sounds (music, laughter, ambient noise), and handles accented or colloquial speech well.

When to use which mode

• Summary mode (default): A friendly narrative description of what's in the recording — who's speaking, what they discuss, the mood, memorable moments. Best for voice messages, meetings, interviews, and podcasts.
• Transcription mode: A verbatim or near-verbatim text of the speech. Use this when the user explicitly asks for a transcript, "word for word", or needs to quote specific passages.
• Combined mode: Both a summary and a transcription. Use when the user asks for both, or when the content seems important enough to warrant the full text.

When in doubt, default to summary mode and offer to also transcribe if useful.

Step-by-step workflow

1. Locate the audio file

The user may give you a path, drop a filename, or refer to something like "the audio file in my Downloads folder." Resolve the path before proceeding. If ambiguous, ask.

2. Confirm the API key is available

This skill requires the GEMINI_API_KEY environment variable. Check it with echo "${GEMINI_API_KEY:+present}" — if empty, stop and tell the user the key is missing and how to set it (export GEMINI_API_KEY=... in their shell rc, then restart the session). Never print the key value.

If the user manages secrets through a password manager (1Password CLI, pass, etc.), suggest they wire it through their shell rc rather than hardcoding it.

3. Call the Gemini API

Use Python to encode the audio and call the API:

import base64, json, os, urllib.request

key = os.environ["GEMINI_API_KEY"]

# Encode audio
with open(audio_path, 'rb') as f:
    audio_b64 = base64.b64encode(f.read()).decode()

# MIME type mapping
mime_types = {
    '.m4a': 'audio/mp4', '.mp4': 'audio/mp4',
    '.mp3': 'audio/mpeg', '.wav': 'audio/wav',
    '.aac': 'audio/aac', '.ogg': 'audio/ogg',
    '.flac': 'audio/flac', '.webm': 'audio/webm',
}
ext = os.path.splitext(audio_path)[1].lower()
mime_type = mime_types.get(ext, 'audio/mp4')

# Call API
url = f'https://generativelanguage.googleapis.com/v1beta/models/gemini-3-flash-preview:generateContent?key={key}'
payload = {
    'contents': [{
        'parts': [
            {'inline_data': {'mime_type': mime_type, 'data': audio_b64}},
            {'text': prompt}
        ]
    }]
}
req = urllib.request.Request(
    url, data=json.dumps(payload).encode(),
    headers={'Content-Type': 'application/json'}
)
with urllib.request.urlopen(req, timeout=120) as r:
    result = json.load(r)
    print(result['candidates'][0]['content']['parts'][0]['text'])

4. Prompt template

For summary mode:

Listen to this audio and write a friendly, natural summary of what is being discussed.
Write it as if you are describing the conversation to someone who has not heard it.
Mention the people referenced by name, the topics covered, the mood and tone, and
any memorable moments or notable quotes. If there are non-speech sounds (music,
laughter, background noise), mention those too.

For transcription mode:

Please transcribe this audio as accurately as possible. Include timestamps every
30 seconds or at natural breaks. Indicate non-speech sounds in [brackets] (e.g.
[laughter], [music], [background noise]). If there are multiple speakers, label
them Speaker 1, Speaker 2, etc. (or use names if clearly stated in the audio).

5. Present the output

Return the summary or transcription as clean prose in the conversation. If the audio contains names, quotes, or notable details, light formatting (bold for names, italics for direct quotes) is fine, but keep it readable and natural.

If a word looks like it may have been misheard — especially proper nouns, brand names, or acronyms — flag it briefly rather than passing it through silently.

File size note

The inline base64 approach works well for files up to ~10MB. For larger files, use the Gemini File API instead (upload first, then reference by URI). For most voice messages and short recordings, inline is fine and simpler.

Model

Use gemini-3-flash-preview. It delivers the best combination of audio comprehension, natural language output, and speed (~10s for a 3-minute recording). If it returns a 503 (high demand), retry once before reporting the error to the user.