Extracting JSON from Emails: Why Schema Validity Does Not Guarantee Truth and How to Protect Automation

The Structured Outputs mode in modern LLMs solves a fundamental engineering problem: it guarantees that model responses strictly adhere to a JSON Schema contract or Pydantic model. However, a valid schema guarantees only the document’s syntactic shape, not the factual truth of its contents.

The official Google Gemini API documentation on Structured Outputs explicitly warns that even when output is syntactically correct, developers must validate values on the application side and handle semantic discrepancies. A schema shields your parser from JSONDecodeError exceptions, but it is powerless against hallucinations, faulty inferences, and overlooked modal qualifications.

Below is an end-to-end scenario covering incoming customer support email processing: from passing raw text to the API through application-level semantic validation and safe field routing.

---

1. Input Data: An Ambiguous Support Email

Consider a synthetic, illustrative support email intentionally designed with typical natural language ambiguities: conditional deadlines and unverified assumptions.

email_text = """От: alex.m@partner-example.com
Тема: Проблема с выгрузкой отчетов и планы по релизу

Привет! У нас со вчерашнего дня сбоит генерация PDF в биллинге.
Мы хотели бы закрыть этот вопрос как можно быстрее, в идеале к концу недели,
но если ваш релиз 2.4 задерживается, то крайний срок сдвигается на следующий вторник.
Кстати, затронут ли европейский кластер платежей eu-central-1, мы пока точно не знаем — коллеги еще перепроверяют трейсы."""

This text contains two critical nuances:

1. Conditional deadline: The due date hinges on an external factor (a delay in release 2.4) and is phrased relatively (“next Tuesday”) rather than as an absolute calendar date.
2. Unconfirmed fact: The eu-central-1 cluster is mentioned merely as an open question and hypothesis pending trace verification. The string is formatted without internal line breaks, enabling exact substring matching.

---

2. Pydantic Schema and Invocation via the Interactions API

Let us define the extraction contract using Pydantic and invoke the model via the official Google GenAI SDK interface.

from typing import Optional
from google import genai
from pydantic import BaseModel, Field


class TicketExtraction(BaseModel):
    issue_summary: str = Field(
        description="Краткая суть проблемы."
    )
    deadline_iso: Optional[str] = Field(
        default=None,
        description="Однозначно зафиксированный срок в формате YYYY-MM-DD. Если дата не определена точно или зависит от условий, вернуть null."
    )
    deadline_context: Optional[str] = Field(
        default=None,
        description="Оговорки, условия или контекст, сопровождающие срок."
    )
    affected_cluster: Optional[str] = Field(
        default=None,
        description="Точно подтвержденный затронутый кластер или сервис. Если факт не подтвержден или выражено сомнение, вернуть null."
    )
    raw_quote: Optional[str] = Field(
        default=None,
        description="Точная цитата из текста, обосновывающая извлеченные факты."
    )


client = genai.Client()

prompt = f"""Извлеки параметры инцидента из письма.
Строго следуй правилу: если автор выражает сомнение или значение зависит от условий,
записывай в соответствующее поле null, а контекст выноси в deadline_context.

Письмо:
{email_text}"""

interaction = client.interactions.create(
    model="gemini-3.8-flash",
    input=prompt,
    response_format={
        "type": "text",
        "mime_type": "application/json",
        "schema": TicketExtraction.model_json_schema(),
    },
)

live_extraction = TicketExtraction.model_validate_json(interaction.output_text)

---

3. Illustrative Failure: Syntactic Success with Semantic Failure

The snippet below illustrates a hypothetical, typical logical failure rather than a guaranteed output of any specific API run. It demonstrates how a model can return formally valid JSON while ignoring the semantic constraints set in the prompt.

{
  "issue_summary": "Сбой генерации PDF в биллинге",
  "deadline_iso": "следующий вторник",
  "deadline_context": "в идеале к концу недели, но если релиз 2.4 задерживается, то следующий вторник",
  "affected_cluster": "eu-central-1",
  "raw_quote": "затронут ли европейский кластер платежей eu-central-1, мы пока точно не знаем"
}

From the perspective of a JSON Schema validator and the Pydantic library, this object is flawless: all keys are present, and the types match. However, for downstream automation, it contains critical errors:

1. Why string types permit arbitrary text: The Optional[str] field in the generated schema becomes an anyOf construct or type union (string and null). To the schema validator, the value "следующий вторник" is a completely valid string. Textual instructions inside description impose no constraints at the parser level, allowing any non-empty string to pass validation.
2. Turning a hypothesis into an established fact: The model extracted the eu-central-1 identifier while ignoring the questioning, speculative nature of the statement.
3. The unreliability of instruction-driven null values: Even strict prompt instructions to return null under uncertainty often yield to entity extraction tendencies when an entity name explicitly appears in the source text.

---

4. Limitations of Naive Checks: Quotes, Stop Words, and Infrastructure Registries

Developers frequently attempt to compensate for LLM semantic unreliability using heuristics. Here is why basic techniques fail to provide guarantees:

1. Quote presence verification (raw_quote in email_text): In our example, the phrase "затронут ли европейский кластер платежей eu-central-1, мы пока точно не знаем" is an exact, continuous substring of the email, so this check evaluates to True. However, literal presence in the text proves only that the quote was not hallucinated; semantically, it proves the exact opposite: the author is unsure whether an outage actually occurred.
2. Heuristic stop-word matching: Searching for marker words (such as particles like "ли", or phrases like "возможно", "не знаем") is fragile. Short markers easily produce false positives within other words or across sentences that carry entirely different modal meanings.
3. Validation against an infrastructure allowlist (Allowed Clusters): Verifying eu-central-1 against an internal server registry only confirms that such a cluster exists (entity identification). It does not prove that an outage or incident actually occurred on that node (failure event). Without external trusted evidence (such as monitoring signals or confirmed alerts), any cluster mention in an incoming email remains an unverified hypothesis.

---

5. Application Validation Pipeline and Safe Routing

To prevent failures in internal downstream systems, applications must implement a second validation layer. This layer independently verifies date formats, screens out questionable entities, and routes incomplete or ambiguous data to a human reviewer.

Python’s date.fromisoformat() helper does not guarantee strict compliance with the YYYY-MM-DD format alone (modern Python versions accept extended ISO strings). Therefore, format checks must combine a strict regular expression with subsequent calendar parsing.

To demonstrate this validator in action, we use a separate demo_extraction object initialized from our hypothetical invalid JSON.

import re
from datetime import date
from typing import Any, Optional
from pydantic import BaseModel, Field


class ValidationResult(BaseModel):
    is_safe_for_automation: bool
    confirmed_deadline: Optional[date] = None
    confirmed_cluster: Optional[str] = None
    review_reasons: list[str] = Field(default_factory=list)
    downstream_payload: dict[str, Any] = Field(default_factory=dict)


def validate_and_route_ticket(
    extracted: TicketExtraction,
    source_text: str
) -> ValidationResult:
    reasons: list[str] = []

    if extracted.raw_quote and extracted.raw_quote not in source_text:
        reasons.append("Указанная цитата отсутствует в исходном письме.")

    parsed_date: Optional[date] = None
    if extracted.deadline_iso:
        if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", extracted.deadline_iso):
            reasons.append(
                f"Значение deadline_iso не соответствует строгому формату YYYY-MM-DD: '{extracted.deadline_iso}'."
            )
        else:
            try:
                parsed_date = date.fromisoformat(extracted.deadline_iso)
            except ValueError:
                reasons.append(
                    f"Значение deadline_iso содержит недопустимую календарную дату: '{extracted.deadline_iso}'."
                )
                parsed_date = None

    if parsed_date and extracted.deadline_context:
        reasons.append(
            f"Срок {parsed_date.isoformat()} сопровождается оговорками ('{extracted.deadline_context}') и требует ручного согласования."
        )
        parsed_date = None

    validated_cluster: Optional[str] = None
    if extracted.affected_cluster:
        reasons.append(
            f"Кластер '{extracted.affected_cluster}' упомянут в письме как предположение; без внешнего подтверждения из мониторинга требуется проверка оператором."
        )

    needs_manual_review = len(reasons) > 0 or parsed_date is None or validated_cluster is None

    safe_payload = {
        "summary": extracted.issue_summary,
        "deadline": parsed_date.isoformat() if parsed_date else None,
        "cluster": validated_cluster,
        "routing_status": "MANUAL_REVIEW" if needs_manual_review else "AUTO_APPROVED",
    }

    return ValidationResult(
        is_safe_for_automation=not needs_manual_review,
        confirmed_deadline=parsed_date,
        confirmed_cluster=validated_cluster,
        review_reasons=reasons,
        downstream_payload=safe_payload,
    )


hypothetical_bad_json = """{
  "issue_summary": "Сбой генерации PDF в биллинге",
  "deadline_iso": "следующий вторник",
  "deadline_context": "в идеале к концу недели, но если релиз 2.4 задерживается, то следующий вторник",
  "affected_cluster": "eu-central-1",
  "raw_quote": "затронут ли европейский кластер платежей eu-central-1, мы пока точно не знаем"
}"""

demo_extraction = TicketExtraction.model_validate_json(hypothetical_bad_json)
decision = validate_and_route_ticket(demo_extraction, email_text)

---

6. Final State: Manual Review and Field Isolation

After application validation executes, the decision object transitions into a strictly controlled state:

{
  "is_safe_for_automation": false,
  "confirmed_deadline": null,
  "confirmed_cluster": null,
  "review_reasons": [
    "Значение deadline_iso не соответствует строгому формату YYYY-MM-DD: 'следующий вторник'.",
    "Кластер 'eu-central-1' упомянут в письме как предположение; без внешнего подтверждения из мониторинга требуется проверка оператором."
  ],
  "downstream_payload": {
    "summary": "Сбой генерации PDF в биллинге",
    "deadline": null,
    "cluster": null,
    "routing_status": "MANUAL_REVIEW"
  }
}

Field Triage by Trust Level

Field	Routing Status	Rationale
summary	Draft Ticket	The billing PDF generation issue is explicitly stated. This field is safe to populate as a preliminary ticket description in an issue tracker.
cluster	Blocked (null)	The cluster mention is an unconfirmed assumption by the author. Automated assignment to an infrastructure on-call rotation is blocked to avoid false alarms.
deadline	Blocked (null)	The unstructured phrase is not a calendar date, and the timeline itself is conditional. Automatically attaching this to a hard SLA is blocked.
routing_status	Operator Queue	The MANUAL_REVIEW status indicates that automated triggers are suspended, routing the ticket to a tier-1 agent alongside the itemized list of review reasons.

Human verification is required prior to any SLA calculation, priority escalation, or routing to specialized engineering teams. The application can generate a draft ticket containing the reported issue summary, but critical incident attributes remain unassigned until explicitly confirmed by an operator.

Summary

Structured Outputs ensure schema-level format stability, protecting downstream code from unexpected syntax errors. However, adherence to a schema contract does not equal data truth. Safely integrating LLMs into business processes requires a two-tiered architecture: syntactic enforcement at the API boundary, strict regex and type validation within application code, and mandatory quarantine of uncertain fields into human-in-the-loop review queues.