AnythingLLM Max Tokens: Fix Truncated Long Responses
A long reply stopping mid-sentence is not automatically a connection problem. Check the effective Generic OpenAI settings, obtain the upstream finish reason, and use controlled comparisons to identify the right fix.
Contents
When an AnythingLLM answer stops mid-sentence, first check whether Max Tokens capped that response. Increasing the context window or a proxy timeout is not an interchangeable fix. You need the output budget actually sent upstream, the response’s finish_reason, and a retest that changes only one variable while keeping your original configuration recoverable.
Use the ending evidence to choose a branch
Text stopping on screen is a symptom, not a diagnosis. In OpenAI Chat Completions, length means the request’s specified token limit was reached; stop can mean a natural ending or a supplied stop sequence. tool_calls means the model handed off to a tool, not that it delivered a final answer. For compatible services, also check the backend’s actual behavior.
| Evidence you have | Check first | Next action |
|---|---|---|
Upstream returns finish_reason=length, and you can verify the effective output budget | Output limit | Keep the input fixed and change only the supported output budget |
| The server explicitly reports excessive context; the same task works with less history or retrieved material | Context pressure | Reduce the input and check the real model window, rather than only increasing output allowance |
| A timeout, reset or cancellation is recorded, or the stream ends without a visible finish reason | Connection, cancellation or streaming compatibility | Match timestamps and locate the layer that ended the request; a missing field alone does not prove a network failure |
The response returns stop but omits a section or numbered item | Instructions, stop sequences and completeness | Inspect the missing material; a normal ending is not the same as a completed task |
The output budget limits how much a request may generate. The context window concerns how much context the model can handle. History, system instructions and retrieved passages can all be part of the input: AnythingLLM’s Generic OpenAI adapter assembles these components. A short question can therefore produce a large request, and entering a larger window in the interface cannot expand the backend model’s capabilities.
A user in issue #6703 reported investigating reverse-proxy and WebSocket timeouts before discovering an output-limit problem. That is an individual report supporting the order of checks, not evidence that every truncated response has the same cause.
Preserve your settings and identify the effective model
Record the configuration before changing it. Note the AnythingLLM version, Desktop or self-hosted deployment, global provider, provider and model selected in the workspace and chat, whether an agent is involved, and the current context and output settings. Redact keys in screenshots. Do not delete the original workspace, conversation or indexed documents for this experiment.
Open the LLM provider configuration in Settings and confirm that the test actually uses Generic OpenAI. Its official setup documentation requires compatibility in both API functionality and inference responses. If the request uses another provider, changing Generic OpenAI settings will not configure that different route.
The official interface source inspected on 2026-10-16 contains these fields:
| Field | What to verify |
|---|---|
Base URL, API Key, Selected Model | The previously working address, matching key and exact model identifier—not a guessed identifier based on a display name |
Model context window | The configured context capacity, not the response’s output allowance |
Max Tokens | The generation budget; inspect it separately from the context setting |
Save, reopen the settings and confirm the values persisted. Where you have authorized access to upstream request records, verify the actual model and output-limit field as well. A screenshot proves what the interface displayed, not what the server received. Test ordinary chat and agents separately: the Desktop v1.16.2 changelog records a fix for Generic OpenAI ignoring max tokens during agent streams.
Already using BetterToken? At this step, compare your current address, key and model configuration with its Chat Completions instructions; this is a configuration reference, not a guarantee that every AnythingLLM feature works or that output limits disappear.
Understand zero, parameter names and version differences
0 does not mean unlimited output. The Desktop v1.17.0 changelog says Generic OpenAI can use 0 to omit max_tokens from the request. The upstream default and limits then apply. For troubleshooting, prefer an explicit positive budget supported by the model so that the comparison is interpretable.
In the adapter source inspected on 2026-10-16, the output value comes from GENERIC_OPEN_AI_MAX_TOKENS, while the context value comes from GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT. The output parameter defaults to max_tokens; GENERIC_OPEN_AI_MODEL_MAX_TOKEN_KEY can select a different field name. These are properties of the inspected source, not a promise that every installed release exposes the same controls.
Do not rename parameters indiscriminately across compatible services. OpenAI’s current parameter reference specifies max_completion_tokens, whose budget includes visible output and reasoning tokens. A short visible answer can therefore still consume the budget. Check what your backend and model accept, and change the corresponding setting only where your installed version supports it.
Desktop users should use the interface, not edit .env based on a copied snippet: the configuration guide explicitly reserves those instructions for self-hosted or Docker deployments. Version v1.17.0 also changed how workspace settings are saved. Confirm persistence by reopening the settings rather than expecting every version to have the same save button.
Get the upstream finish reason, not just a closed chat bubble
Prefer the model service’s response to AnythingLLM. For non-streaming Chat Completions, inspect choices[0].finish_reason and any returned usage. For a stream, inspect the chunk carrying the finish reason and whether the backend’s complete termination sequence arrived. A null value in the first chunk is not the final status. See the official response definitions.
On the web or a self-hosted web interface, open the browser’s Network panel, preserve the log and reproduce the problem once. Record the chat request’s timestamp, status and response. Remember that this captures browser-to-AnythingLLM traffic, not necessarily the raw response from the model service.
The Generic OpenAI stream handler inspected on 2026-10-16 turns a non-empty upstream finish reason into AnythingLLM’s own close: true message instead of forwarding the reason unchanged. It also has a content-chunk counting estimate when provider usage is unavailable. An ended conversation and a token number on screen consequently do not establish either successful completion or an exact budget hit.
Obtain the original reason and usage from model-service logs or request details you are authorized to access. Otherwise, give your deployment maintainer the request timestamp and identifier. The Desktop v1.16.2 changelog describes opening logs and storage by clicking the version tag at the top of the window; finding the log directory does not mean that every version logs the raw finish field.
Usage may also be absent legitimately: OpenAI documents that interrupted or cancelled streams may never deliver the final usage chunk. Record it as unavailable, not zero, and do not treat an application estimate as billing data.
Run short and long controls without losing the old configuration
Fix the request route first, then vary the requested response length. Use a test workspace without attached knowledge documents. Select the same provider and model, and keep the system prompt, temperature, output budget and other settings fixed. Start each test in an empty conversation so the second does not inherit the first. Test ordinary chat before adding agents or tools.
Short-response test A:
List 3 recommendations for managing project documentation. Write 2–3 sentences for each. When finished, put
[END]on a separate line.
Long-response test B:
List 20 recommendations for managing project documentation. Write 2–3 sentences for each. When finished, put
[END]on a separate line.
Send the prompt and let generation finish without pressing Stop or refreshing. If it times out or disconnects, record when instead of repeatedly retrying. Copy the complete displayed responses into local files A.txt and B.txt. Record the request time, effective model and output budget, finish reason, whether usage is provider-reported, and whether the final numbered item and [END] are present. The numbers here define a test task, not model limits or guaranteed response lengths.
A complete A and an incomplete B show that the longer-output case is problematic; they do not yet prove a Max Tokens issue. If B returns length, keep B’s exact prompt and all other settings, change only the output budget to an explicit value the server allows, and run test C in an empty conversation. Save it as C.txt. All 20 items, the ending marker and no length termination support the conclusion that the budget change helped this case.
If B already completes, do not inflate the budget unnecessarily. Compare the same real task in the original conversation and an empty one while holding the output setting fixed. Investigate context when the failure follows history or retrieved material. Restore temporary settings you no longer need and retain the original records. Paid API tests can incur normal request charges.
Apply the fix that matches the evidence
Confirmed output limit: change the budget and verify the outgoing field
Increase only the budget used by the effective request route, within the model’s supported limits. If the interface value changes but the outgoing request does not, inspect provider selection, persistence, version and chat-versus-agent behavior. Increasing the context window will not repair an output setting that never reached the server.
Excessive context: reduce input while keeping the task and budget fixed
First retry the same long task in an empty conversation. Then independently reduce retrieved material or unnecessary history. When the server explicitly reports a context limit, configure the window to match the real model; typing a larger number cannot expand it. Improvement in an empty conversation without an explicit error still leaves context as a hypothesis to verify.
Connection or stream failure: locate the failing layer before changing timeouts
Compare browser, AnythingLLM and upstream records for the same interval. Look for cancellation, timeout or connection closure. If upstream completed but the browser received only part of the response, investigate the intermediate transport and rendering path. If upstream failed, address its error first. Change a layer’s timeout only when evidence points to that layer.
Self-hosted or Docker administrators can test streaming separately. The official configuration provides GENERIC_OPENAI_STREAMING_DISABLED="true", affecting only Generic OpenAI; restart the relevant service or container after changing it. Keep everything else fixed and rerun the same long prompt, then restore the previous value and restart. A successful non-streaming response narrows the investigation to streaming-related paths; it does not prove the proxy caused the problem. Do not use this .env procedure on Desktop.
Decide whether it is fixed—and what to send if it is not
A longer answer is not enough. Check that all requested sections or numbers exist, the last sentence is complete and the ending marker is present, then use upstream and connection records to check for another truncation. Neither stop nor [END] replaces inspection of the content. Finally, retest the real task that originally failed, not only the simplified control.
If it still fails, send the maintainer the version and deployment type, chat or agent mode, provider and model identifiers, redacted settings, a minimal prompt, the A/B/C differences, timestamps and any original finish reason obtained. Remove API keys, authorization headers and private document text. Mark missing fields as unavailable. That evidence makes the next investigation about a specific request instead of another round of speculative proxy changes.
