Podcast Generation (Microsoft)

tl;dr: a full-stack recipe for shipping an Azure OpenAI Realtime audio feature, not a generic “make a podcast” prompt. SKILL.md gives a working Python FastAPI WebSocket client (Azure OpenAI’s AsyncOpenAI with websocket_base_url), a React frontend that decodes base64 WAV to a blob and plays it, the Realtime API event lifecycle (response.output_audio.delta / response.output_audio_transcript.delta / response.done), and PCM-to-WAV conversion via scripts/pcm_to_wav.py. Listed in the README’s Core table with 8 test scenarios under tests/scenarios/podcast-generation/.

What it actually does

The skill activates when the agent needs to wire text-to-speech, podcast-from-text, or any audio narrative generation on top of Azure OpenAI Realtime Mini. The flow SKILL.md documents end-to-end:

1. Configure env vars for the Realtime endpoint (no /openai/v1/ in the base URL — the SDK appends it).
2. Open a WebSocket to wss://<endpoint>/openai/v1 via AsyncOpenAI(websocket_base_url=...).
3. Send a session config with output_modalities: ["audio"] and a narrator instruction.
4. POST the text as a user input_text item, call response.create(), and stream events.
5. Collect base64 audio deltas + transcript deltas, concatenate, and convert PCM (24 kHz, 16-bit, mono) to WAV.
6. Return base64 WAV to a React frontend that decodes it to a Blob and plays it.

The skill assumes you want a streaming round-trip, not a one-shot HTTP TTS call.

Environment

AZURE_OPENAI_AUDIO_API_KEY=your_realtime_api_key
AZURE_OPENAI_AUDIO_ENDPOINT=https://your-resource.cognitiveservices.azure.com
AZURE_OPENAI_AUDIO_DEPLOYMENT=gpt-realtime-mini

Endpoint must NOT include /openai/v1/ — the SDK appends that path itself when converting https:// → wss://.

Install (runtime deps)

The skill itself installs via the microsoft/skills wizard; the runtime deps below are what SKILL.md actually imports:

# Backend (Python)
pip install openai fastapi uvicorn websockets

# Frontend (React)
npm install react
# No extra SDK — base64 + Blob + Audio are native browser APIs.

Backend (FastAPI + AsyncOpenAI WebSocket)

This is the only complete runnable example SKILL.md ships:

from openai import AsyncOpenAI
import base64

# Convert HTTPS endpoint to WebSocket URL
ws_url = endpoint.replace("https://", "wss://") + "/openai/v1"

client = AsyncOpenAI(
    websocket_base_url=ws_url,
    api_key=api_key
)

audio_chunks = []
transcript_parts = []

async with client.realtime.connect(model="gpt-realtime-mini") as conn:
    await conn.session.update(session={
        "output_modalities": ["audio"],
        "instructions": "You are a narrator. Speak naturally."
    })

    await conn.conversation.item.create(item={
        "type": "message",
        "role": "user",
        "content": [{"type": "input_text", "text": prompt}]
    })

    await conn.response.create()

    async for event in conn:
        if event.type == "response.output_audio.delta":
            audio_chunks.append(base64.b64decode(event.delta))
        elif event.type == "response.output_audio_transcript.delta":
            transcript_parts.append(event.delta)
        elif event.type == "response.done":
            break

pcm_audio = b"".join(audio_chunks)
wav_audio = pcm_to_wav(pcm_audio, sample_rate=24000)
return base64.b64encode(wav_audio).decode("ascii")

Frontend (browser playback)

const base64ToBlob = (base64, mimeType) => {
  const bytes = atob(base64);
  const arr = new Uint8Array(bytes.length);
  for (let i = 0; i < bytes.length; i++) arr[i] = bytes.charCodeAt(i);
  return new Blob([arr], { type: mimeType });
};

const audioBlob = base64ToBlob(response.audio_data, "audio/wav");
const audioUrl = URL.createObjectURL(audioBlob);
new Audio(audioUrl).play();

No third-party audio library is required. The whole playback path is atob → Uint8Array → Blob → URL.createObjectURL → new Audio().

Voice options

Pass the chosen voice in the session.update instructions or via a per-request override; the SKILL.md covers the session-config form, not a per-utterance voice switch.

Realtime API event lifecycle

Only four event types matter for this skill:

Anything else from the Realtime stream is not part of this skill’s contract; ignore or log it.

Audio format

• Input: text prompt (string).
• Output: PCM, 24 kHz, 16-bit, mono.
• Storage / transport: base64-encoded WAV (wrap with scripts/pcm_to_wav.py).
• Browser: Blob("audio/wav") + URL.createObjectURL.

Gotchas

• Endpoint path is your responsibility. If you paste a URL with /openai/v1/ already in it, the WebSocket handshake fails — the SDK appends /openai/v1 itself. SKILL.md calls this out explicitly.
• Realtime ≠ chat completions. The AsyncOpenAI(websocket_base_url=...) form and client.realtime.connect(...) are different from client.chat.completions.create(). Don’t mix the two surfaces.
• Audio is base64, not binary. Deltas arrive as base64 strings — base64.b64decode(event.delta) per chunk, then concatenate.
• WAV header is not free. PCM chunks don’t play in a browser until you wrap them in a WAV header. SKILL.md points to scripts/pcm_to_wav.py (not inlined) for the wrapper.
• Transcript is delta-only. Stitch transcript_parts yourself; the skill does not join them.
• sandbox_test untested. We read SKILL.md and the README test-count table (8 scenarios), we did not run the FastAPI + WebSocket flow in the sandbox.

References (per SKILL.md)

• references/architecture.md — Full-stack architecture detail.
• references/code-examples.md — Production patterns beyond the single FastAPI example.
• scripts/pcm_to_wav.py — PCM-to-WAV header wrapper.
• Azure OpenAI Realtime API docs: https://learn.microsoft.com/azure/ai-services/openai/realtime-audio-quickstart

Related

• Cloud Solution Architect (microsoft/skills) — same Microsoft Core bundle, broader procedural depth for Azure services.
• MCP Builder (microsoft/skills) — same Microsoft Core bundle, when you want the audio feature exposed as a tool.
• Skill Creator (microsoft/skills) — same Microsoft Core bundle, for authoring skills like this one.
• Refactoring UI Audit — different vendor, the operator-authored skill this catalog uses as the gold-quality bar.