Guide: Video and Audio Input

Guide: Video and Audio Input#

A video file can contain both pictures and sound. QSTN supports both VideoInput and AudioInput.

For the Gemma 4/vLLM setup below, video input becomes timestamped image frames. The default video path does not supply the soundtrack to the model. We add it separately in the second run. This is backend-specific: other models or configurations may handle video audio differently. See vLLM’s Gemma 4 implementation.

In the future, models might support both video and audio input in one. For now we show how to get an LLM to answer questions related to both a video and the corresponding audio.

1. Get the clip#

We use Cat Jumpscare.webm by Panini!, released under CC0. The clip is about ten seconds long. Downloading it once lets both runs use the same file.

Run this notebook in a Python 3.12 environment with QSTN, vllm[audio], pandas, and Jupyter.

python -m pip install qstn "vllm[audio]" pandas jupyter

You also need FFmpeg on your system path to extract the soundtrack. The model uses a local GPU; the first use may download its weights.

import subprocess
from pathlib import Path
from tempfile import gettempdir
from urllib.request import Request, urlopen

import pandas as pd
from IPython.display import Video, display

from qstn.inference import AudioInput, VideoInput
from qstn.logger import configure_logging
from qstn.parser import raw_responses
from qstn.prompt_builder import LLMPrompt
from qstn.survey_manager import conduct_survey_sequential
from qstn.utilities import create_one_dataframe, placeholder

configure_logging(level="WARNING", force=True)
pd.set_option("display.max_colwidth", None)

media_dir = Path(gettempdir()) / "qstn_video_audio_guide"
media_dir.mkdir(exist_ok=True)
video_path = media_dir / "cat.webm"
video_url = "https://upload.wikimedia.org/wikipedia/commons/0/07/Cat_Jumpscare.webm"

if not video_path.exists():
    request = Request(video_url, headers={"User-Agent": "QSTN documentation tutorial"})
    with urlopen(request, timeout=60) as response:
        video_path.write_bytes(response.read())

print("Video ready:", video_path.name)

Video ready: cat.webm

Here is the video!

preview_path = media_dir / "cat_preview.mp4"
subprocess.run(
    ["ffmpeg", "-v", "error", "-y", "-i", str(video_path),
     "-map", "0:v:0", "-map", "0:a:0", "-shortest",
     "-vf", "scale=480:-2", "-c:v", "libx264", "-crf", "32",
     "-preset", "fast", "-c:a", "aac", "-b:a", "48k",
     "-movflags", "+faststart", str(preview_path)],
    check=True,
)
Video(filename=str(preview_path), embed=True, width=480)

2. Simple QSTN setup#

Does the model see what is happening in the video and does it hear the human laugh in the end?

questions = pd.DataFrame(
    [
        {   "questionnaire_item_id": 1,
            "question_content": "Describe what is happening in the video."
        },
        {
            "questionnaire_item_id": 2,
            "question_content": "What sound does the human make at the end?",
        },
    ]
)

questionnaire = LLMPrompt(
    questionnaire_name="cat_clip",
    questionnaire_source=questions,
    system_prompt=(
        "Use the video when it is available and use the soundtrack when it is attached. "
        "For human reactions, include audible sounds even when the person is off-camera."
    ),
    prompt=f"{placeholder.PROMPT_QUESTIONS}",
)

_ = questionnaire.add_video(VideoInput(video_path, label="Cat clip"))

VideoInput(source, label=...) works like ImageInput: the source can be a local path, an HTTP(S) URL, or a base64 data URL. Omitting item_id shares the video with both questions.

3. Load a small Gemma 4 model#

We load google/gemma-4-E2B-it once and reuse it. This small instruction-tuned model supports vision and audio. We allow one video and one audio attachment per request so the same model configuration works for both runs.

The short context, one request at a time, and eager execution keep this demonstration manageable on a 16 GB GPU. Thinking is disabled in both runs so the output stays brief.

from vllm import LLM

model = LLM(
    model="google/gemma-4-E2B-it",
    max_model_len=4096,
    max_num_seqs=1,
    enforce_eager=True,
    limit_mm_per_prompt={"image": 0, "video": 1, "audio": 1},
    seed=42,
)

4. Run with video only#

With video input only, the model only sees the frames of the video, but does not get the sound input, therefore it can only answer one of the questions.

video_results = conduct_survey_sequential(
    model,
    questionnaire,
    max_tokens=1280,
    chat_template_kwargs={"enable_thinking": False},
    print_progress=False,
)
video_answers = create_one_dataframe(raw_responses(video_results))
display(video_answers[["question", "llm_response"]])

INFO 09-14 13:31:46 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
WARNING 09-14 13:31:47 [video.py:348] Failed to grab frame 309 during video loading. This frame will be skipped.
WARNING 09-14 13:31:47 [video.py:370] Video loading completed with 1 broken/unreadable frames. Expected 32 frames but only loaded 31 frames.
WARNING 09-14 13:31:47 [video.py:408] Video loading completed with 1 broken/unreadable frames. Expected to sample 32 frames but only loaded 31 frames.
WARNING 09-14 13:31:47 [video.py:594] opencv video loading: expected 32 frames but got 31.

[opus @ 0x4054db00] Error parsing Opus packet header.

WARNING 09-14 13:32:15 [video.py:348] Failed to grab frame 309 during video loading. This frame will be skipped.
WARNING 09-14 13:32:15 [video.py:370] Video loading completed with 1 broken/unreadable frames. Expected 32 frames but only loaded 31 frames.
WARNING 09-14 13:32:15 [video.py:408] Video loading completed with 1 broken/unreadable frames. Expected to sample 32 frames but only loaded 31 frames.
WARNING 09-14 13:32:15 [video.py:594] opencv video loading: expected 32 frames but got 31.

[opus @ 0x731ae000] Error parsing Opus packet header.
question llm_response
0 Describe what is happening in the video. The video shows a sequence of clips of a cat interacting with its environment, primarily on a light-colored, shaggy carpet in front of a dark piece of furniture.\n\nHere is a breakdown of what is happening across the clips:\n\n* **00:00 - 00:03:** The cat is actively playing or stretching, with its body extended and tail raised in various poses, moving across the carpet.\n* **00:04:** The cat is shown lying down, seemingly grooming or relaxing.\n* **00:05:** The cat is seen in another playful, stretched posture.\n* **00:06:** The cat is moving or stretching again.\n* **00:07:** A close-up shot shows the cat's face, looking towards the camera, with its whiskers visible.\n* **00:08 - 00:09:** The cat appears to be lying down or resting near the base of the furniture, possibly settled in for a nap or observing.\n\nOverall, the video captures playful and relaxed moments of a cat in its home environment.
1 What sound does the human make at the end? Since the provided video clips only contain footage of the cat and no audio overlay or explicit human speech is visible or clearly audible in the context of the video description, **I cannot tell you what sound the human makes at the end.**

5. Add the soundtrack at the same place#

To have audio of the video we will have to extract the audio ourselves.

We duplicate the questionnaire and append AudioInput directly after the global video. We only need to add_audio.

audio_path = media_dir / "cat.wav"
subprocess.run(
    ["ffmpeg", "-v", "error", "-y", "-i", str(video_path),
     "-vn", "-ac", "1", "-ar", "16000", str(audio_path)],
    check=True,
)

with_audio = questionnaire.duplicate()
with_audio.add_audio(AudioInput(audio_path, label="Soundtrack of the cat clip"))
print("Attachment order:", [type(media).__name__ for media in with_audio.get_media()])

Attachment order: ['VideoInput', 'AudioInput']

Now the model has audio evidence as well as frames. Now it can detect the laugh at the end of the video.

audio_results = conduct_survey_sequential(
    model,
    with_audio,
    temperature=0,
    max_tokens=1280,
    chat_template_kwargs={"enable_thinking": False},
    print_progress=False,
)
audio_answers = create_one_dataframe(raw_responses(audio_results))
display(audio_answers[["question", "llm_response"]])

WARNING 09-14 13:32:17 [video.py:348] Failed to grab frame 309 during video loading. This frame will be skipped.
WARNING 09-14 13:32:17 [video.py:370] Video loading completed with 1 broken/unreadable frames. Expected 32 frames but only loaded 31 frames.
WARNING 09-14 13:32:17 [video.py:408] Video loading completed with 1 broken/unreadable frames. Expected to sample 32 frames but only loaded 31 frames.
WARNING 09-14 13:32:17 [video.py:594] opencv video loading: expected 32 frames but got 31.

[opus @ 0x641c4740] Error parsing Opus packet header.

WARNING 09-14 13:32:22 [video.py:348] Failed to grab frame 309 during video loading. This frame will be skipped.
WARNING 09-14 13:32:22 [video.py:370] Video loading completed with 1 broken/unreadable frames. Expected 32 frames but only loaded 31 frames.
WARNING 09-14 13:32:22 [video.py:408] Video loading completed with 1 broken/unreadable frames. Expected to sample 32 frames but only loaded 31 frames.
WARNING 09-14 13:32:22 [video.py:594] opencv video loading: expected 32 frames but got 31.

[opus @ 0x4546db00] Error parsing Opus packet header.
question llm_response
0 Describe what is happening in the video. This video shows a cat engaging in playful behavior on a light-colored, shaggy carpet.\n\nHere is a breakdown of what happens:\n\n* **00:00 - 00:03:** The cat is actively playing, stretching, and arching its back, with its tail raised. It appears to be moving around energetically on the carpet.\n* **00:04:** The cat is seen lying down, seemingly resting or settling down.\n* **00:05:** The cat is seen in a more relaxed or curious posture, perhaps looking around.\n* **00:06:** The cat is moving or stretching again.\n* **00:07:** A close-up shot shows the cat's face, looking directly at the camera with a slight expression.\n* **00:08 - 00:09:** The cat is seen lying down again, perhaps settling down for a nap or just relaxing.\n\nOverall, the clip captures a sequence of playful activity followed by moments of relaxation for the cat.
1 What sound does the human make at the end? At the very end of the video (around 00:09), there is a sound that sounds like a **chuckle or a light laugh**.

6. Final Comparison#

comparison = pd.DataFrame({
    "Question": video_answers["question"],
    "Video only": video_answers["llm_response"],
    "Video + audio": audio_answers["llm_response"],
})
display(comparison)
Question Video only Video + audio
0 Describe what is happening in the video. The video shows a sequence of clips of a cat interacting with its environment, primarily on a light-colored, shaggy carpet in front of a dark piece of furniture.\n\nHere is a breakdown of what is happening across the clips:\n\n* **00:00 - 00:03:** The cat is actively playing or stretching, with its body extended and tail raised in various poses, moving across the carpet.\n* **00:04:** The cat is shown lying down, seemingly grooming or relaxing.\n* **00:05:** The cat is seen in another playful, stretched posture.\n* **00:06:** The cat is moving or stretching again.\n* **00:07:** A close-up shot shows the cat's face, looking towards the camera, with its whiskers visible.\n* **00:08 - 00:09:** The cat appears to be lying down or resting near the base of the furniture, possibly settled in for a nap or observing.\n\nOverall, the video captures playful and relaxed moments of a cat in its home environment. This video shows a cat engaging in playful behavior on a light-colored, shaggy carpet.\n\nHere is a breakdown of what happens:\n\n* **00:00 - 00:03:** The cat is actively playing, stretching, and arching its back, with its tail raised. It appears to be moving around energetically on the carpet.\n* **00:04:** The cat is seen lying down, seemingly resting or settling down.\n* **00:05:** The cat is seen in a more relaxed or curious posture, perhaps looking around.\n* **00:06:** The cat is moving or stretching again.\n* **00:07:** A close-up shot shows the cat's face, looking directly at the camera with a slight expression.\n* **00:08 - 00:09:** The cat is seen lying down again, perhaps settling down for a nap or just relaxing.\n\nOverall, the clip captures a sequence of playful activity followed by moments of relaxation for the cat.
1 What sound does the human make at the end? Since the provided video clips only contain footage of the cat and no audio overlay or explicit human speech is visible or clearly audible in the context of the video description, **I cannot tell you what sound the human makes at the end.** At the very end of the video (around 00:09), there is a sound that sounds like a **chuckle or a light laugh**.