Indexing Long Videos with the Gemini API: Timestamps, Visuals, and Gap Auditing
An engineering workflow for indexing long video recordings with the Gemini API: uploading through the Files API, generating agentic drafts in the Interactions API, validating structured rows programmatically, and targeted inspection of coverage gaps.
Contents
Long-form video recordings—technical talks, workshops, architecture review calls, and screencasts—pack high-density information across spoken audio and on-screen visuals. Standard high-level summaries only capture broad themes, forcing engineers to scrub through footage manually whenever they need a specific terminal command or configuration parameter.
Gemini multimodal models can analyze audio and video streams simultaneously, generating structured, timestamped event indexes. However, the raw output from an initial model inference pass is only a draft candidate set, not a production-ready reference outline. Building a dependable index requires a disciplined pipeline: uploading once through the Files API and reusing the URI across requests, extracting draft intervals, strictly validating output schema, auditing suspicious coverage gaps, and performing targeted calibration.
Architecture: Delivery Methods and Processing Modes
In current Google Video Understanding documentation, primary implementations center around the Interactions API and the google-genai library. While the traditional generate_content method remains supported for backward compatibility, the Interactions API provides clearer control over multimodal processing parameters.
1. Video Delivery Methods
- Files API (Recommended for long videos): Best suited for recordings spanning from several minutes to several hours. The file is uploaded once, indexed on the server, and referenced by URI across repeated requests without re-transmitting raw bytes.
- Google Cloud Storage (GCS): Ideal for existing video archives already hosted in Google Cloud infrastructure.
- Inline Data: Transmitting raw bytes directly inside the request payload. Documentation outlines varying payload constraints across environments; for stable handling of hour-long media, relying on the Files API or Cloud Storage is a practical choice, avoiding streaming raw video inline.
2. Processing Modes: Static vs. Agentic
- Static Processing:
By default, the model samples frames at a discrete rate of 1 frame per second (1 FPS). Each second converts into tokens, accumulating substantial context load over a 60-minute duration. Keep in mind: 1 FPS sampling may miss brief visual events (such as sub-second window switches or fleeting tooltips) and cannot guarantee capturing every micro-interaction. - Agentic Video Understanding:
The model navigates the video dynamically, pulling specific frames and audio segments on demand. This sharply reduces the volume of processed context tokens.
Limitation: The agentic heuristic relies on audio cues and semantic triggers. If a presenter executes actions silently on screen (such as typing a command or inspecting a diagram) without speaking, the heuristic may treat that interval as idle background and skip requesting detailed visual frames.
Step 1. Video Upload and Status Polling
Files pushed to the Files API are not available for inference immediately; the server must unpack the container and index audio-visual tracks. Your application must poll the resource until its state becomes ACTIVE.
import time
from google import genai
client = genai.Client()
video_path = "tech_workshop_60min.mp4"
video_file = client.files.upload(file=video_path)
while not video_file.state or video_file.state.name != "ACTIVE":
if video_file.state and video_file.state.name == "FAILED":
error_info = getattr(video_file, "error", None)
raise RuntimeError(
f"Файл перешел в статус FAILED. Детали: {error_info}. "
"Проверьте кодеки, целостность контейнера или повторите попытку."
)
time.sleep(5)
video_file = client.files.get(name=video_file.name)
print("Видео готово к обработке.")Step 2. Requesting a Draft Index via the Interactions API
For long recordings, invoke client.interactions.create with "processing": "agentic". The prompt enforces a strict tabular output: timestamp range MM:SS - MM:SS, event type (visual, speech, hybrid), a concise speech claim (CLAIM), and on-screen activity (ACTION).
index_prompt = """
Ты — инструмент технической индексации видеозаписей.
Сформируй хронологический индекс событий для всей 60-минутной записи от 00:00 до 60:00.
Требования к структуре ответа:
1. Выведи результат построчно в формате TSV с разделителем |.
2. Каждая строка должна содержать ровно 5 полей:
START_TIME (MM:SS) | END_TIME (MM:SS) | TYPE (visual/speech/hybrid) | CLAIM | ACTION
3. Не объединяй слишком длинные интервалы в одну запись; фиксируй смену слайдов, терминал, ошибки и выводы спикера.
4. Если речи не было, в поле CLAIM укажи NONE. Если на экране не было динамики, в ACTION укажи STATIC.
5. Выводи исключительно строки данных без вводных слов и пояснений.
"""
interaction = client.interactions.create(
model="gemini-3.8-flash",
input=[
{
"type": "video",
"uri": video_file.uri,
"mime_type": video_file.mime_type,
"processing": "agentic"
},
{
"type": "text",
"text": index_prompt
}
]
)
raw_index_output = interaction.output_textOn Agentic Navigation Steps:
The presence ofprocessing_callentries insideinteraction.stepsconfirms that the model dynamically navigated the video timeline rather than ingesting a continuous stream. However, seeing these calls confirms only navigation activity—it does not guarantee output completeness across all relevant timeline events.
Step 3. Example Output Structure (Hypothetical Draft)
Below is an illustrative hypothetical excerpt from the model output demonstrating the expected data schema:
00:00 | 02:15 | hybrid | Вводное слово, обзор повестки миграции на шардированный кластер | Титульный слайд доклада, окно спикера
02:16 | 05:40 | visual | NONE | Переключение на схему архитектуры сервиса заказов в Miro
05:41 | 09:12 | speech | Пояснение причин отказа от распределенных транзакций в пользу Saga | Статичная схема Miro, курсор неподвижен
27:30 | 29:10 | visual | NONE | Открытие консоли, запуск сценария развертывания реплик БД
29:11 | 32:45 | hybrid | Разбор сценария split-brain при потере сетевой связности | Вывод логов etcd в терминале, подсветка таймаутов
54:10 | 57:25 | speech | Ответ на вопрос о допустимой задержке репликации данных | Финальный слайд с контактами спикера
57:26 | 60:00 | hybrid | Подведение итогов и демонстрация ссылки на репозиторий | Показ QR-кода на экране, завершение созвонаStep 4. Markup Validation and Local Search
Raw LLM responses cannot be trusted as structured datasets without verification. A resilient parser must not silently discard corrupt records; instead, it should isolate them for manual editing. The validator enforces: exactly five fields, valid event types (visual, speech, hybrid), and well-formed MM:SS timestamps within video boundaries.
import csv
import re
from typing import List, Dict, Tuple
TIME_PATTERN = re.compile(r"^(\d{2}):([0-5]\d)$")
ALLOWED_TYPES = {"visual", "speech", "hybrid"}
def time_to_seconds(t_str: str) -> int:
match = TIME_PATTERN.match(t_str)
if not match:
raise ValueError(f"Некорректный формат времени: {t_str}")
m, s = map(int, match.groups())
return m * 60 + s
def parse_and_validate_tsv(
raw_text: str,
max_duration_sec: int = 3600
) -> Tuple[List[Dict[str, str]], List[Dict[str, str]]]:
valid_rows = []
malformed_rows = []
lines = [line.strip() for line in raw_text.strip().splitlines() if line.strip()]
for idx, line in enumerate(lines, start=1):
cols = [c.strip() for c in line.split("|")]
if len(cols) != 5:
malformed_rows.append({
"line": idx,
"content": line,
"error": f"Ожидалось 5 полей, получено {len(cols)}"
})
continue
start_str, end_str, event_type, claim, action = cols
if event_type.lower() not in ALLOWED_TYPES:
malformed_rows.append({
"line": idx,
"content": line,
"error": f"Недопустимый тип события: {event_type}"
})
continue
try:
s_sec = time_to_seconds(start_str)
e_sec = time_to_seconds(end_str)
except ValueError as err:
malformed_rows.append({
"line": idx,
"content": line,
"error": str(err)
})
continue
if s_sec > e_sec:
malformed_rows.append({
"line": idx,
"content": line,
"error": f"Время начала ({start_str}) больше времени окончания ({end_str})"
})
continue
if e_sec > max_duration_sec:
malformed_rows.append({
"line": idx,
"content": line,
"error": f"Таймкод {end_str} выходит за хронометраж ({max_duration_sec} сек)"
})
continue
valid_rows.append({
"start": start_str,
"end": end_str,
"start_sec": s_sec,
"end_sec": e_sec,
"type": event_type.lower(),
"claim": claim,
"action": action
})
return valid_rows, malformed_rows
def search_index(
rows: List[Dict[str, str]],
keyword: str = None,
event_type: str = None
) -> List[Dict[str, str]]:
results = []
for r in rows:
if event_type and r["type"] != event_type.lower():
continue
if keyword:
kw = keyword.lower()
if kw not in r["claim"].lower() and kw not in r["action"].lower():
continue
results.append(r)
return results
valid_index, errors = parse_and_validate_tsv(raw_index_output, max_duration_sec=3600)
if errors:
print(f"Обнаружено некорректных строк: {len(errors)}. Требуется ручная правка:")
for err in errors:
print(f"Строка {err['line']}: {err['error']} -> {err['content']}")
else:
print(f"Все строки валидны. Записей в индексе: {len(valid_index)}")Step 5. Coverage Auditing and Identifying Gaps
Before publishing the index, audit the timeline for blind spots:
- Spot-check benchmark zones:
Manually check the start of the video (introduction and title slides), the midpoint (where live demos or architecture debates typically peak), and the conclusion (Q&A and closing notes). - Analyze timestamp intervals:
There is no universal threshold for acceptable spacing between index entries; tolerance depends on the use case. In a dense screencast, a 90-second gap could mean a missing configuration step. In an overview lecture, a single thesis spanning 5 minutes may be completely reasonable. If an interval looks inconsistent with the presentation’s pace, flag it as suspicious. - Inspect silent visual events:
If a speaker presented on screen without narration, agentic mode may have glossed over that segment. Route such windows to targeted static review.
Step 6. Targeted Inspection of Suspicious Zones via Static Clip
Re-evaluating a suspicious segment does not require re-running the entire 60-minute video. Clip slicing restricts static-mode analysis to precise second boundaries (start_offset and end_offset).
Method Limitation:
Static mode at 1 FPS provides a fixed sampling grid, helping uncover actions missed during the broad agentic pass. However, it does not guarantee absolute recall: visual changes occurring faster than one second can still fall between sampled frames.
start_sec = 1200
end_sec = 1380
targeted_inspection = client.interactions.create(
model="gemini-3.8-flash",
input=[
{
"type": "video",
"uri": video_file.uri,
"mime_type": video_file.mime_type,
"processing": {
"type": "static",
"start_offset": start_sec,
"end_offset": end_sec,
"fps": 1.0
}
},
{
"type": "text",
"text": (
"Хронологически опиши изменения на экране. "
"Зафиксируй команды в терминале, смену окон и системные сообщения."
)
}
]
)
print("Результат точечного досмотра отрезка:")
print(targeted_inspection.output_text)Integrate the newly extracted entries into the validated index either manually or through a semi-automated review stage.
Once you have resolved all entries in errors and verified timestamps against the source recording, export the final index. The code snippet below deliberately halts if the parser reports remaining errors; ensure valid_index contains your verified fixes before running it.
if errors:
raise ValueError("Исправьте строки из errors и повторите проверку перед экспортом")
with open("video_index.csv", "w", newline="", encoding="utf-8") as output_file:
writer = csv.DictWriter(
output_file,
fieldnames=["start", "end", "type", "claim", "action"],
extrasaction="ignore",
)
writer.writeheader()
writer.writerows(valid_index)The resulting CSV allows users to search for specific claims or screen actions and jump directly to relevant segments in the source recording. It remains a draft until a human editor confirms both which events were selected and their event boundaries, not merely sampling boundaries.
Troubleshooting and Edge Cases
FAILEDstatus during Files API upload:
Avoid diagnosing the issue without API diagnostics. Failures can stem from unsupported container formats, malformed file headers, or transient infrastructure errors. Inspect thefile.errorattribute via the SDK, verify local playback withffprobe, standardize streams viaffmpeg(-c:v libx264 -c:a aac) if needed, and retry.401 Unauthorizedor network drops:
A 401 error explicitly indicates authentication failure (an invalid or missing key, or an unconfiguredGEMINI_API_KEYenvironment variable), not an expired processing session. To avoid client HTTP connection timeouts during extended calls, enable streaming viastream=True.- Timestamps exceeding total duration:
This drift can occur as context complexity grows. Counteract it with strict prompt instructions and programmatic validation (e_sec > max_duration_sec) in your parser. - TSV structural drift:
When formatting breaks, route malformed lines tomalformed_rowsand supply 1–2 few-shot reference lines in the system prompt.
Pre-Publishing Verification Workflow
- Verify asset readiness: Confirm the file has reached
ACTIVEstatus in the Files API. - Generate initial draft: Create the baseline index using
agenticprocessing via the Interactions API. - Execute programmatic validation: Ensure all rows contain five required fields, valid types, and
MM:SSformat. Isolate invalid lines for correction. - Audit coverage: Inspect benchmark zones (start, middle, end) and evaluate timestamp density against the pacing of the talk.
- Conduct targeted inspection: Re-examine suspicious gaps or silent screen sections using static clips (
start_offset/end_offset). - Perform manual spot calibration: Check each event’s actual start time against the relevant source audio/video as the use case requires; do not demand a matching visual change, since an audio-only event can occur.
For request schemas, processing modes, and sampling constraints, consult the official Gemini Video Understanding documentation.
