Dify Chatflow Streams Only at the End: Debug Agent, Fail Branch, and Answer Waits

A Dify Chatflow answer that appears in one burst at the end does not prove that the upstream model stopped streaming. It also does not prove that changing providers will fix the problem. Keep one short prompt, one model, and one Agent configuration constant, then compare four clocks: upstream first token, SSE events, agent_log, and user-facing message chunks.

The goal is to identify the layer that waits or buffers text. You do not need to remove production error handling to do that. Turning off Fail Branch is a control run in a test copy, not a default production fix.

Identify the layer from the event pattern

What you observe	Suspect first	Next check
An LLM-only control also produces no early text	Model, provider, gateway, or model settings	Verify a short plain-text request and the model’s streaming support
workflow_started, node_started, and other events are all delayed	Proxy, gateway, or SSE transport before Dify reaches the client	Repeat the same request directly against Dify and look for response buffering
agent_log arrives throughout the run, but message arrives in a final burst	The Agent-to-Answer text path, with Fail Branch as a control variable	Keep the production fallback and compare message timing in a cloned flow
message begins only after another variable completes	An earlier unresolved variable in Answer	Reorder variables or deliver stages through multiple Answer nodes

Dify’s Answer documentation says that an Answer node streams according to variable availability. It emits content up to the first unresolved variable, waits for that variable, and continues later. Variable order in Answer—not upstream execution order—therefore controls what the user can see first.

Create four controls that change one thing at a time

Duplicate the existing Chatflow for testing. Record its published version, model ID, temperature, Agent strategy, tools, maximum iterations, memory settings, and error-handling mode. Use the same short prompt in every run, for example:

Write 12 numbered one-sentence checks for debugging a streaming response.

The prompt should be long enough to create an observable generation span, but it should contain no real user data.

Repeat the same release step before controls A, B, C, and D: apply only that round’s change, publish the current state of the test copy, record a run label and the published workflow version, then call the Service API Base shown on that copy’s API Access page (the full path usually ends in /v1) with that copy’s App API Key. Do not reuse the original app’s key. When workflow_id is omitted, Dify runs the latest published version, so calling before publishing can make several rounds execute the same older build.

Control A: LLM → Answer only

In the test copy, send the same model’s output directly from an LLM node to Answer, without Agent or tools. Dify documents that LLM nodes stream by default. If message still arrives only at the end, do not blame Agent yet. Check the model provider, OpenAI-compatible gateway, reverse proxy, and Answer variable order first.

Control B: current Agent with Fail Branch preserved

Restore the existing Agent path and the production fallback. This is the real baseline. Do not remove the fallback merely to improve perceived speed: Fail Branch exists to run a separate recovery path, such as logging, notification, a different approach, or a backup service.

Control C: disable Fail Branch only in the test copy

Run this control only when the current Agent node exposes the same error-handling option. Return error handling to its default in the copy, while leaving the model, prompt, tools, and every other setting unchanged. This test answers whether the branch correlates with the delay in your current environment. It does not answer whether production can safely operate without a fallback.

A user report for Dify 1.17.1, langgenius/agent 0.0.33, and graphon 0.7.0 recorded 805 message events between 29.819 and 30.035 seconds with Fail Branch enabled—a span of 0.216 seconds. In the default-error-handling control, 866 messages ran from 3.397 to 40.991 seconds. The same report saw agent_log events throughout execution. This is evidence from one reported environment, not a result for every Dify release, and it does not establish a confirmed fixed version.

Control D: change only Answer variable order

First let Answer reference only the Agent’s text. Then add other variables back one by one. If Answer starts with a late variable and places Agent text after it, the text can wait even while Agent is generating. Put the streaming text variable earlier, or use the officially supported multiple Answer nodes to deliver content at different stages.

Record arrival time with one minimal API call

Call the published Chatflow API instead of judging only the browser animation. The Dify streaming guide defines response_mode: "streaming" as an SSE response. For Chatflow, concatenate message events in order to obtain the user-facing answer. An agent_log event is an informational Agent-step log; receiving it proves that the event channel is active, not that answer text has reached the user.

For every run, take the Service API Base and App API Key from the test copy’s API Access page. Enter the full base, normally including /v1; the script appends /chat-messages. When version pinning is available, put that round’s published workflow version ID in DIFY_WORKFLOW_ID. Leave it blank only when you deliberately want the latest published version. Either way, publish the round first and include the version in DIFY_RUN_LABEL.

Use {} for DIFY_TEST_INPUTS_JSON only when the copy has no required custom inputs. If Start/User Input defines required fields, send the same sanitized JSON values, with the application’s exact parameter names and types, in all four runs.

In Bash, read the App API Key without displaying it and enter the current run parameters:

unset DIFY_TARGET_NODE_ID DIFY_FALLBACK_NODE_ID DIFY_EXPECTED_FALLBACK
read -r DIFY_API_BASE && export DIFY_API_BASE
read -rs DIFY_API_KEY && export DIFY_API_KEY
read -r DIFY_TEST_QUERY && export DIFY_TEST_QUERY
read -r DIFY_RUN_LABEL && export DIFY_RUN_LABEL
read -r DIFY_WORKFLOW_ID && export DIFY_WORKFLOW_ID
read -r DIFY_TEST_INPUTS_JSON && export DIFY_TEST_INPUTS_JSON
python3 capture_dify_sse.py

Save the following as capture_dify_sse.py:

import json
import os
import sys
import time
from collections import defaultdict
from http.client import IncompleteRead
from pathlib import Path
from urllib import error, request


def redacted(value):
    return None if value is None else f"<redacted:{len(str(value))} chars>"


def load_inputs():
    raw = os.environ.get("DIFY_TEST_INPUTS_JSON", "{}").strip() or "{}"
    try:
        value = json.loads(raw)
    except json.JSONDecodeError as exc:
        raise SystemExit(f"DIFY_TEST_INPUTS_JSON must be valid JSON: {exc.msg}") from exc
    if not isinstance(value, dict):
        raise SystemExit("DIFY_TEST_INPUTS_JSON must decode to a JSON object")
    return value


api_base = os.environ["DIFY_API_BASE"].rstrip("/")
api_key = os.environ["DIFY_API_KEY"]
test_query = os.environ["DIFY_TEST_QUERY"]
raw_label = os.environ.get("DIFY_RUN_LABEL", "unnamed-run").strip() or "unnamed-run"
run_label = "".join(
    char if char.isalnum() or char in "-_" else "_" for char in raw_label
)[:80] or "unnamed-run"
workflow_id = os.environ.get("DIFY_WORKFLOW_ID", "").strip()
target_node_id = os.environ.get("DIFY_TARGET_NODE_ID", "").strip()
fallback_node_id = os.environ.get("DIFY_FALLBACK_NODE_ID", "").strip()
expected_fallback = os.environ.get("DIFY_EXPECTED_FALLBACK", "")

body = {
    "inputs": load_inputs(),
    "query": test_query,
    "response_mode": "streaming",
    "user": "streaming-probe",
}
if workflow_id:
    body["workflow_id"] = workflow_id

req = request.Request(
    f"{api_base}/chat-messages",
    data=json.dumps(body).encode(),
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    method="POST",
)

started = time.perf_counter()
stats = defaultdict(list)
answer_parts = []
message_end_seen = False
workflow_finished_seen = False
workflow_status = None
terminal_event = None
fatal_reason = None
target_node_failed = None if not target_node_id else False
fallback_node_succeeded = None if not fallback_node_id else False
top_error = {"status": None, "code": None, "message_redacted": None}

log_path = Path(f"{run_label}-events.jsonl")
summary_path = Path(f"{run_label}-summary.json")

with log_path.open("w", encoding="utf-8") as log_file:
    try:
        with request.urlopen(req, timeout=180) as response:
            event_name = None
            for line_no, raw_line in enumerate(response, 1):
                line = raw_line.decode("utf-8", errors="replace").strip()
                if line.startswith("event:"):
                    event_name = line.split(":", 1)[1].strip()
                    continue
                if not line.startswith("data:"):
                    continue

                elapsed = round(time.perf_counter() - started, 3)
                try:
                    event = json.loads(line.split(":", 1)[1].strip())
                except json.JSONDecodeError:
                    fatal_reason = f"json_parse_error_at_stream_line_{line_no}"
                    record = {
                        "elapsed_s": elapsed,
                        "event": "parse_error",
                        "detail": "<redacted:invalid JSON payload>",
                    }
                    log_file.write(json.dumps(record, ensure_ascii=False) + "\n")
                    log_file.flush()
                    stats["parse_error"].append(elapsed)
                    print(f"{elapsed:8.3f}s parse_error")
                    break

                kind = event.get("event") or event_name or "unknown"
                event_name = None
                data = event.get("data") if isinstance(event.get("data"), dict) else {}
                answer = event.get("answer") if isinstance(event.get("answer"), str) else ""
                answer_parts.append(answer)

                node_id = data.get("node_id")
                data_status = data.get("status")
                error_status = event.get("status") if kind == "error" else None
                error_code = event.get("code") if kind == "error" else None
                error_message = redacted(event.get("message")) if kind == "error" else None
                record = {
                    "elapsed_s": elapsed,
                    "event": kind,
                    "answer_chars": len(answer),
                    "task_id": event.get("task_id"),
                    "workflow_run_id": event.get("workflow_run_id"),
                    "node_id": node_id,
                    "node_title": data.get("title"),
                    "data_status": data_status,
                    "error_status": error_status,
                    "error_code": error_code,
                    "error_message_redacted": error_message,
                }
                log_file.write(json.dumps(record, ensure_ascii=False) + "\n")
                log_file.flush()
                stats[kind].append(elapsed)

                if kind == "message_end":
                    message_end_seen = True
                elif kind == "workflow_finished":
                    workflow_finished_seen = True
                    workflow_status = data_status
                    terminal_event = kind
                elif kind == "workflow_paused":
                    workflow_status = "paused"
                    terminal_event = kind
                elif kind == "error":
                    top_error = {
                        "status": error_status,
                        "code": error_code,
                        "message_redacted": error_message,
                    }
                    terminal_event = kind

                if kind == "node_finished" and node_id == target_node_id:
                    target_node_failed = data_status in {"failed", "exception"}
                if kind == "node_finished" and node_id == fallback_node_id:
                    fallback_node_succeeded = data_status == "succeeded"

                print(
                    f"{elapsed:8.3f}s {kind:<22} answer_chars={len(answer)} "
                    f"data_status={data_status} error_status={error_status} "
                    f"error_code={error_code} error_message={error_message}"
                )
    except KeyboardInterrupt:
        fatal_reason = "interrupted_by_user"
    except error.HTTPError as exc:
        fatal_reason = f"http_error_{exc.code}"
        top_error = {
            "status": exc.code,
            "code": "http_error",
            "message_redacted": redacted(exc.reason),
        }
    except error.URLError as exc:
        fatal_reason = "connection_error"
        top_error = {
            "status": None,
            "code": "connection_error",
            "message_redacted": redacted(exc.reason),
        }
    except (IncompleteRead, TimeoutError, OSError) as exc:
        fatal_reason = type(exc).__name__
        top_error = {
            "status": None,
            "code": type(exc).__name__,
            "message_redacted": redacted(exc),
        }

if fatal_reason:
    run_outcome, incomplete_reason = "incomplete", fatal_reason
elif workflow_status == "succeeded" and message_end_seen and workflow_finished_seen:
    run_outcome, incomplete_reason = "succeeded", None
elif terminal_event == "workflow_paused":
    run_outcome, incomplete_reason = "paused", None
elif workflow_status in {"failed", "partial-succeeded", "stopped"}:
    run_outcome, incomplete_reason = workflow_status, None
elif terminal_event == "error":
    run_outcome, incomplete_reason = "failed", None
elif workflow_finished_seen:
    run_outcome = "unknown_terminal_status"
    incomplete_reason = "workflow_finished_without_recognized_status"
else:
    run_outcome = "incomplete"
    incomplete_reason = "stream_ended_without_terminal_event"

expected_fallback_seen = (
    None if not expected_fallback else expected_fallback in "".join(answer_parts)
)
complete_for_timing_acceptance = run_outcome == "succeeded"
failure_validation_requested = any(
    (target_node_id, fallback_node_id, expected_fallback)
)
failure_validation_configured = all(
    (target_node_id, fallback_node_id, expected_fallback)
)
failure_validation_passed = (
    None
    if not failure_validation_requested
    else failure_validation_configured
    and run_outcome in {"succeeded", "partial-succeeded"}
    and all(
        value is not False
        for value in (
            target_node_failed,
            fallback_node_succeeded,
            expected_fallback_seen,
        )
    )
)
acceptance_checks_passed = (
    failure_validation_passed
    if failure_validation_requested
    else complete_for_timing_acceptance
)

summary = {
    "run_label": run_label,
    "requested_workflow_id": workflow_id or None,
    "version_selection": "pinned-published-version" if workflow_id else "latest-published-version",
    "run_outcome": run_outcome,
    "workflow_status": workflow_status,
    "terminal_event": terminal_event,
    "message_end_seen": message_end_seen,
    "workflow_finished_seen": workflow_finished_seen,
    "incomplete_reason": incomplete_reason,
    "top_level_error": top_error,
    "target_node_failed": target_node_failed,
    "fallback_node_succeeded": fallback_node_succeeded,
    "expected_fallback_seen": expected_fallback_seen,
    "complete_for_timing_acceptance": complete_for_timing_acceptance,
    "failure_validation_requested": failure_validation_requested,
    "failure_validation_configured": failure_validation_configured,
    "failure_validation_passed": failure_validation_passed,
    "acceptance_checks_passed": acceptance_checks_passed,
    "event_timings": {
        kind: {
            "count": len(times),
            "first_s": times[0],
            "last_s": times[-1],
            "span_s": round(times[-1] - times[0], 3),
        }
        for kind, times in sorted(stats.items())
    },
}
summary_path.write_text(
    json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
)
print(json.dumps(summary, ensure_ascii=False, indent=2))
if not acceptance_checks_passed:
    sys.exit(2)

The script never writes answer text to disk. It records timing, node ID/title, data.status, answer length, and the top-level status and code from an error event while replacing its message with a character-count marker. It distinguishes succeeded, failed, partial-succeeded, stopped, paused, a top-level error, and a stream that ends without a terminal event.

Output names use DIFY_RUN_LABEL. If the connection is interrupted or an SSE JSON payload cannot be parsed, captured timings are still saved, but run_outcome is incomplete and complete_for_timing_acceptance is false. Such spans remain diagnostic evidence only. A normal control passes timing acceptance only after both message_end and workflow_finished.data.status=succeeded are observed. HTTP/1.1 chunked SSE truncation that raises http.client.IncompleteRead is also caught: received event timings remain in the summary, acceptance_checks_passed=false, and the exception’s partial response body is never echoed.

Compare four timestamps, not one loading spinner

1. Upstream first token. Record the provider or self-hosted gateway metric when available. Without one, the LLM → Answer control is a practical way to isolate the upstream path.
2. First SSE data event. A Chatflow normally reports workflow_started early. That confirms acceptance of the run, not progressive answer delivery.
3. First agent_log. Early Agent logs show that step events can cross the SSE connection. If logs arrive while text waits, the entire connection is unlikely to be buffered as one object.
4. First and last message. These are the important user-facing times. Compare first_s, last_s, and span_s. Many message events compressed into a tiny span are measurable evidence of the final burst.

A successful Chatflow should emit message_end, then workflow_finished with data.status=succeeded. The official SSE contract lists succeeded, failed, partial-succeeded, and stopped for workflow_finished.data.status; record them separately. A human-input pause instead emits human_input_required followed by the separate workflow_paused event, from which the script records run_outcome=paused. Do not substitute a run-record status enum for the workflow_finished event contract. In an error event, status, error type, and message are top-level fields; the script shows status, code, and a redacted message. If the connection ends without a terminal event, or transport/JSON parsing is interrupted, the summary is incomplete. Even when some message events were captured, their span cannot prove an improvement unless complete_for_timing_acceptance is true.

Choose the change that matches the result

The LLM-only control also waits: fix the upstream layer first

Confirm model ID, endpoint, provider streaming behavior, and any intermediate gateway before returning to Agent. When BetterToken is your optional OpenAI-compatible model API baseline, its Dify setup guide recommends proving a short plain-text request first and enabling tool capabilities only after the selected model and endpoint support them. That can isolate the upstream model layer; it does not prove that Dify nodes will not buffer output.

Every event arrives late: inspect the transport path

Repeat the exact request with fewer CDN, reverse-proxy, or API-gateway layers. If events arrive progressively when connecting directly to Dify but are grouped after an intermediary is restored, inspect streaming forwarding and response buffering at that intermediary. Keep the model unchanged so the result remains attributable.

Only Fail Branch groups message events

Record the conclusion narrowly: the delay correlates with the current Dify version, Agent plugin, and workflow configuration. Preserve the production fallback by default. If live delivery is more important, design and verify another recovery path before migration; one control run is not a reason to delete alerts, backup service calls, or error logs.

Before a Dify or Agent-plugin upgrade, save one complete normal run and one controlled-failure verification. After the upgrade, publish the test copy and repeat with the same prompt, model, sanitized inputs, and version-recording procedure. A saved failure timeline alone is not completion: the target node must actually fail, the corresponding Fail Branch and fallback Answer must run, and the user must receive a sanitized fallback.

Answer reordering restores streaming

The upstream model may have been streaming all along while Answer waited on an earlier variable. Keep the new order or staged Answer nodes, then run the failure verification below. error_type and error_message can drive routing and internal logs, but the user-facing Answer should contain sanitized wording rather than raw error details.

Verify one controlled failure only in the test copy

Prefer an existing, repeatable failure condition that uses no real user data and makes the exact target node under review emit an abnormal node_finished result; a handled Fail Branch error can have data.status=exception, so failed must not be required. A tool’s internal error, an agent_log entry, or a model log does not by itself prove that the Agent node failed.

In the versioned source, graphon 0.7.0 converts a failure handled by FAIL_BRANCH into NodeRunExceptionEvent with status exception. Dify 1.17.1 emits it as node_finished.data.status=exception, while an unhandled node failure has status failed. The script’s target_node_failed accepts exception or failed as evidence of abnormal execution of the target itself—not failure of the whole workflow or proof of a successful fallback. Validation still requires the corresponding fallback node to succeed, a visible sanitized fallback, and one of the allowed final outcomes listed below.

When no suitable condition exists, add a temporary HTTP Request node on a diagnostic path in the test copy, point it to https://fail-branch-check.invalid/, and use a short timeout. .invalid is reserved for names that are guaranteed to be invalid. Set that node’s error handling to Fail Branch and connect the orange branch to the fallback Answer under review. This verifies the temporary HTTP node’s Fail Branch/Answer wiring. It does not replace an Agent-level failure test when the acceptance target is the Agent node itself.

1. Publish the test copy after the temporary change. Record the published version, the target node ID, the fallback Answer node ID, and a short non-sensitive phrase that appears only in the fallback.
2. Use the copy’s Base and Key, the same sanitized inputs, and that published version. Set all three validation values and run the same script:

read -r DIFY_TARGET_NODE_ID && export DIFY_TARGET_NODE_ID
read -r DIFY_FALLBACK_NODE_ID && export DIFY_FALLBACK_NODE_ID
read -r DIFY_EXPECTED_FALLBACK && export DIFY_EXPECTED_FALLBACK
python3 capture_dify_sse.py

3. The summary should show target_node_failed: true, fallback_node_succeeded: true, expected_fallback_seen: true, and failure_validation_passed: true. The workflow may finish as succeeded or partial-succeeded, but not failed, stopped, paused, or incomplete.
4. Remove the temporary failure node or restore the original controlled condition, restore the normal connections and configuration, and publish the copy again. Clear the three validation variables and rerun the normal request. That restored run must show complete_for_timing_acceptance: true, after which you can compare the first and last message times again.

Accept the fix only when streaming and fallback both pass

Do not use a faster first chunk as the only success criterion. Keep two summaries: one complete normal request showing message chunks spread across generation, and one controlled failure proving that the target node failed, the corresponding Fail Branch/fallback Answer executed, and the user received a sanitized fallback. A tool error or Agent log alone, a missing target-node failure, no user-visible fallback, or no normal rerun after restoring the test change all fail acceptance.