Multi-reference image drift: separate layout from style

You send a layout reference, a style reference, and a subject image to a generation tool. The request succeeds, but the result looks like generic stock art: the subject moves, the planned empty space disappears, and several visual styles collapse into one vague “AI look.” Do not respond by piling on more prompt text. First give every reference a specific job, then compare the output with the intended composition item by item.

In one public report, an agent concatenated multiple style references into a single flat parameter. The model averaged the inputs and returned generic textures without raising an API error. That report does not establish a universal model behavior, but it exposes the important distinction: a technically successful request is not the same as a successful visual task.

The practical rule: one image owns composition, another owns appearance

The most useful change is to stop treating every image as an equally weighted, unexplained reference. Separate at least these three roles:

Reference role	What it should control	What it should not control
Layout reference	Subject count, position, relative scale, camera, crop, negative space	Palette, material, brushwork, or lighting style
Subject reference	Identity, shape, and distinctive features of a person or object	Overall composition or unrelated background objects
Style reference	Palette, texture, line quality, grain, lighting, and rendering language	The reference’s objects, text, or composition

When the task needs only layout and style, use only those two images. More references do not automatically create more control; they can add competing signals that the system has to reconcile.

Why a flat reference list often produces a generic compromise

A flat list says, “all of these images matter,” but not why each image matters. The model or an intermediate agent must infer the relationship. It may take the color from one image, the texture from another, and an unwanted object from a third, then produce a visually plausible compromise.

This failure may produce no structural error at all. Authentication, upload, request syntax, and response parsing can all work correctly. An HTTP success status or a “generation completed” message therefore cannot serve as visual acceptance. Your workflow needs a composition check after generation.

Step 1: write a role contract for every reference

Before writing a long prompt, answer four questions for each image:

1. What must be preserved? For example: the subject sits in the lower-left corner, the top remains open for a headline, and the camera angle is low.
2. What must be ignored? For example: ignore the person’s identity, brand text, and background architecture in the style reference.
3. How strict is the requirement? Is composition a hard constraint and color a preference, or the reverse?
4. Who wins when references conflict? For example: the layout reference overrides object placement implied by the style image.

A usable role contract can look like this:

File	Role	Preserve	Ignore	Conflict rule
layout.png	Layout	Two subjects with a large-small relationship, right-side negative space, overhead view	Color and material	Its spatial relationships win over all other references
subject.png	Subject	Silhouette and distinctive details	Original background and camera	Replace the subject in the layout without moving it
style.png	Style	Warm-gray palette, paper grain, soft shadows	People and text in the image	Transfer appearance only; do not copy content

This is more useful than “reference 1, reference 2, reference 3” because it defines both desired signals and forbidden transfers.

Step 2: replace the flat list with a hierarchical request

Exact API fields vary by tool. The following is a conceptual request structure, not a real vendor contract. Use it to check whether your agent preserves role information all the way to the final request:

references:
  - id: layout
    source: layout.png
    role: composition
    preserve: [subject_count, position, scale, camera, negative_space]
  - id: subject
    source: subject.png
    role: identity
    preserve: [shape, distinctive_details]
  - id: style
    source: style.png
    role: appearance
    preserve: [palette, texture, line_quality, lighting]
    exclude: [objects, text, composition]
priority:
  - layout
  - subject
  - style
acceptance_reference: layout.png

The important question is not whether your API uses these field names. It is whether the same information survives the whole chain. Inspect the request that the agent actually sends: did an image array become one concatenated string, did role descriptions collapse into ordinary prose, or did retries and batch processing change the file order?

When the target API explicitly supports separate reference types, weights, or edit masks, map the role contract to its documented schema. When it does not, do not invent parameters. Switch to a staged workflow instead.

Step 3: use two stages when the tool cannot express reference roles

If the interface accepts only one undifferentiated set of images, lock the composition first and apply style second. This is usually easier to diagnose than pushing every signal through one request.

Stage A: establish composition and subject

Use the layout reference, plus the subject reference only when necessary. Describe subject count, placement, camera, crop, and negative space. Leave material and rendering language out of this stage. The goal is a structurally correct image, even if it still looks plain.

Stage B: apply appearance without rebuilding the scene

Use the Stage A output as the new base image, then edit or redraw it with the style reference. State that subject positions, boundaries, camera, crop, and negative space must remain fixed; only palette, texture, line quality, and lighting may change.

If the tool supports masks, expose only the regions that need editing. Without an edit mode, keep the Stage A image as the primary reference and make the style image secondary. Whether explicit weights are available depends on the current tool documentation.

Step 4: run four controlled variants to locate the lost signal

Do not keep changing one complicated request. Hold the prompt, dimensions, and other controllable settings constant, then generate four variants:

Test	Input	What to inspect
A	Layout reference only	Can the tool preserve subject count, positions, camera, crop, and negative space?
B	Style reference only	Which palette, texture, line, and lighting traits does it actually transfer?
C	Layout and style as a flat list	Does it create a compromise, average the styles, or import unwanted content?
D	Layout and style with explicit roles and priority	Is each target closer than in Test C?

This is not a test of whether a model is “good” or “bad.” It tells you whether the breakdown starts with single-image interpretation, multi-image composition, or the agent’s parameter handling. If A and B work, C drifts, and D improves, role ambiguity is a strong suspect. If D looks identical to C, inspect the final payload or use the staged method.

When the tool supports a seed, keep it constant across these variants to reduce random variation. When it does not, do not pretend the comparison is deterministic; generate more than one sample per variant and look for repeated structural failures.

Step 5: accept against the target composition, not “looks good”

The dangerous output is not always ugly. It may be polished enough to pass a casual review while missing the actual template. Check hard constraints before judging style.

Hard constraints: fail the candidate when any one is wrong

• The number of subjects is correct.
• Their positions and relative scale match the layout.
• Camera direction, crop, and viewpoint are correct.
• Reserved text space or negative space remains available.
• Objects, people, or text from the style reference have not leaked into the output.
• Distinctive subject features remain recognizable.

Soft constraints: use them to rank surviving candidates

• The palette is close to the target.
• Texture and grain feel intentional rather than muddy.
• Lines, edges, and shadows match the desired visual language.
• The result feels coherent instead of averaging several styles.

Place the layout reference and each candidate side by side, and record pass or fail for every hard constraint. Save more than the final image: keep the request version, reference order, and the exact payload sent by the agent. That evidence makes the next drift reproducible.

Diagnose common symptoms in the right order

Symptom	Check first	Preferred response
Style is strong, but composition changes	Was the style image treated as an equal primary reference?	Raise layout priority or split composition and styling into stages
Composition is right, but style is weak	Does the prompt contain only abstract mood words?	Name observable palette, material, line, and lighting traits
Several styles collapse into a generic look	Are conflicting style images included together?	Choose one dominant style reference; let others contribute one specific trait
The style image’s subject appears in the output	Did you say that only appearance should transfer?	Explicitly exclude its objects, text, and composition
Prompt edits have no visible effect	Did the agent send the new parameters?	Compare final payloads, not only the upstream form
API succeeds while outputs keep drifting	Is success status being used as visual acceptance?	Add hard composition gates and controlled comparisons

A minimum workflow you can reuse

1. Select only the references required to complete the task.
2. Give each image one primary role, with preserve, ignore, and conflict rules.
3. Inspect the final agent request to confirm arrays, order, and role information were not flattened.
4. Run layout-only and style-only baselines before the flat and role-separated variants.
5. Reject candidates on composition hard constraints before comparing style.
6. Save candidates, references, request versions, and acceptance results together.
7. When the interface cannot represent roles, use a composition-first, style-second workflow.

The goal is not a longer prompt. It is a workflow in which every visual signal has an owner. As long as you can answer “which image controls this, which rule wins a conflict, and what counts as a pass,” multiple references stop being a pile of images that the model must interpret on its own.