Knowledge-Centric Agents for Workflow Generation in ComfyUI

Conference: ECCV 2026
Paper: Official Paper Page · PDF
Area: LLM Agents
Keywords: ComfyUI, workflow generation, knowledge inversion, hierarchical supervised fine-tuning, self-refinement

TL;DR

The paper recovers strategies, reasoning explanations, and skeleton pseudocode from real ComfyUI workflows, then uses hierarchical SFT and self-refinement to reconstruct executable graphs from user instructions, improving average Pass / Resolve over ComfyAgent from 36.4% / 29.4% to 86.9% / 61.1% on its curated test set, although Resolve remains only 25.0% on the external ComfyBench subset.

Background & Motivation

ComfyUI connects diffusion models, prompt encoders, ControlNet, segmentation, face restoration, and super-resolution components into directed acyclic graphs. A request to preserve a photograph's layout, change its style, enhance facial detail, and restore its original colors does not identify a single model to call. A system must choose the order of operations, decide which controls interact, place local restoration appropriately, and distinguish images, latents, models, and conditioning signals at module interfaces. Workflow generation therefore combines visual-processing design with executable graph construction; valid JSON is only the final representational requirement.

Direct instruction-to-JSON generation compresses strategy selection, topology, parameters, and serialization into one prediction. Omitting an upscaling stage can produce a graph that runs but does not satisfy the request, whereas an incorrect output-slot index can prevent a sensible processing plan from running at all. Templates and meta-nodes simplify connectivity but restrict the space of compositions. Neither a larger model nor an added chain-of-thought prompt guarantees practical knowledge about when a particular processing branch is necessary.

The authors start with existing community workflows, which often lack accompanying user requests and design explanations. Their approach preserves the graph's structural information, infers higher-level descriptions that can serve as supervision, and subsequently generates new graphs in the opposite direction. Core idea: separate the implicit expertise in real workflows into strategy, skeleton, and parameter levels, explicitly train instruction-to-strategy and strategy-to-skeleton mappings, and delegate regularized reconstruction to nonlearned procedures.

Method

Overall Architecture

Training begins with collected ComfyUI workflows; online inference begins with a user instruction and the required input assets. The output is an executable ComfyUI JSON graph and its generated result. The system comprises Hierarchical Knowledge Inversion, Selective Knowledge Injection, and Hierarchical Inference and Self-Refinement: the first two prepare representations and learn mappings, while the last moves from abstract intent back to concrete configuration for a new task.

Inversion does not mean computing a mathematical inverse. It means recovering plausible design intent from a workflow. Likewise, reversible reasoning does not establish a bijection: different strategies can yield similar topologies, and one structure can support different objectives. The inferred explanations and instructions are synthetic supervision, not records of the original workflow author's actual reasoning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Raw["Real workflows"] --> Invert["Hierarchical Knowledge Inversion<br/>Graph to skeleton to strategy"]
    Invert --> Inject["Selective Knowledge Injection<br/>Two-stage SFT"]
    Inject --> Infer["Hierarchical Inference<br/>and Self-Refinement<br/>Strategy to skeleton to parameters"]
    Input["New instruction and input assets"] --> Infer
    Infer --> Output["Rule-based JSON reconstruction<br/>Execute workflow and return results"]

Key Designs

1. Hierarchical Knowledge Inversion: turn a working graph into supervision at several abstraction levels

A raw JSON workflow contains nodes, parameters, connections, and ComfyUI presentation details. A rule-based converter first produces full pseudocode that retains module names, topology, and parameter values while removing styling information and redundant syntax. This is a structured program representation suitable for a language model, not a freely generated textual summary. A language model then removes concrete values and parameter bindings to produce skeleton pseudocode. What disappears is low-level configuration, not the variable connections that express dependencies between nodes; otherwise, the skeleton would lose the graph structure it is intended to teach.

The next step asks a model to explain the skeleton's organization: why multiple controls are fused, why face detailing follows denoising, or why foreground and background processing use separate branches. The authors call this strategy reasoning extraction and subsequently compress the explanation into a high-level strategy. Because most source workflows lack user queries, the system combines retrieved exemplars with structural and semantic context to infer compatible instructions and input specifications. One graph can thus supply linked representations of a task, a strategy, a reasoning explanation, a skeleton, and full pseudocode rather than just an isolated JSON target.

This hierarchy distinguishes reusable processing logic from instance-specific settings. Which modules should connect can transfer across configurations, whereas sampling settings and model-file bindings determine how a particular instance runs. It also introduces supervision noise: a model may attach a plausible but inaccurate rationale to an existing topology. The paper does not validate these explanations against ground-truth human design rationales, so they should be treated as weak supervision rather than verified causal knowledge.

2. Selective Knowledge Injection: train the transitions that require compositional expertise

Instead of assigning every conversion to one training objective, the authors fine-tune Qwen3-14B in two stages. Instruction-to-strategy training teaches goal interpretation, module selection, and processing logic. Strategy-to-skeleton training teaches how to realize that plan as module calls and data dependencies. Training also includes an auxiliary strategy reasoning extraction objective, requiring explanations of strategic choices. The main text does not specify its weight or a complete loss equation; it should not be reinterpreted as an otherwise undocumented regularization algorithm.

Skeleton-to-full-pseudocode completion primarily restores parameters and bindings. The authors rely on pretrained model knowledge, lightweight prompting, or rules for this transition rather than additional supervised training. Full pseudocode is then reconstructed into JSON by rules. This does not imply that parameters are unimportant. It directs limited training data toward decisions that are harder to encode with explicit rules, allowing processing plans to be learned separately from the low-level configuration of a particular JSON file.

The authors position SFT as the capability-injection mechanism and reinforcement-style methods such as GRPO as possible tools for local improvement within an already feasible generation space. This is their rationale for the present design, not a general conclusion established by a controlled SFT-versus-RL experiment in this paper. No GRPO variant is trained in the reported experiments, so its appearance in future work should not be mistaken for an implemented component.

3. Hierarchical Inference and Self-Refinement: inspect the plan, inspect the structure, then restore configuration

At inference time, a trained strategy planner reads the instruction and input specifications and proposes a high-level processing strategy. The model is prompted to inspect and revise that strategy, addressing missing subgoals. A pseudocode generator then converts the strategy into a skeleton, followed by another self-refinement step to improve structural coherence and incomplete reasoning. These checks operate at the stages of deciding what to do and deciding how modules connect, instead of waiting for every error to become entangled in a final JSON document.

The resulting skeleton is passed to a pretrained model to fill parameters and bindings, after which rules convert the full pseudocode into a ComfyUI configuration. Variable dependencies can be reconstructed as node connections, with serialization handled deterministically. However, reconstruction rules do not guarantee the correct model files, installed plugins, or satisfactory visual output. The paper uses GPT-5 for pretrained agent components and aligns the baseline interface with the GPT-5 refine agent. The complete system therefore combines fine-tuned Qwen3-14B with components such as GPT-5; it is not an entirely standalone 14B model.

Self-refinement here is prompted review of strategies and pseudocode. The main text does not provide a fixed revision count, stopping criterion, runtime-error parser, or algorithm for replanning from the final image. Although the generated graph is executed, the description does not establish an implemented loop that automatically diagnoses execution failures and repairs the workflow afterward.

A Worked Example

Consider the photograph-stylization request in Figure 1: preserve layout and shapes, improve facial details, and harmonize the final colors with the source. A sufficient strategy cannot stop at image-to-image generation; it must explicitly include structural guidance, local restoration, and color matching. The illustrated plan first resizes the input and extracts a Canny edge map as a structural anchor. It also derives a prompt, uses CLIP to encode positive and negative conditioning, and loads the diffusion model.

At the skeleton level, ControlNet is combined with positive conditioning, a sampler generates using the relevant conditions, and the result is decoded into RGB. A face detailer then detects faces, segments masks, and performs localized inpainting on the generated image. Color matching against the source is applied at the end. Face restoration belongs after decoding because it operates on actual generated face regions; late color matching adjusts the result after stylization and restoration have taken place.

Parameter filling and rule-based reconstruction turn these dependencies into concrete node configurations. This example describes the paper's illustrated pipeline, not a reproduction executed by the note author. The cached paper does not supply every node ID, model file, and numerical setting for this example, so no supposedly executable JSON is invented here.

Loss & Training

Qwen3-14B is fine-tuned with LoRA on one NVIDIA H200, using rank 16 and scaling factor \(\alpha=32\). Maximum sequence length is 2048 tokens for instruction-to-strategy training and 4096 tokens for strategy-to-pseudocode training. The paper does not introduce a new explicit loss function. Its main text also omits the learning rate, training epochs, batch size, and auxiliary-objective weight; those settings are not reconstructed speculatively here.

Data preparation starts from more than 10,000 workflows and compares node-level structural differences. When the differing-node ratio is below 10%, GPT-4o evaluates semantic and procedural equivalence, favoring versions with more standardized and widely used node types. Identical graphs are merged by unifying input entries. Workflows with fewer than 6 or more than 90 nodes are removed, and task descriptions are normalized. The resulting collection contains 912 workflows, split into 882 training and 30 test instances. Consequently, the learned distribution is not the unrestricted distribution of workflows in the broader community.

Key Experimental Results

Main Results

The curated test set comprises Challenge 1, Challenge 2, and Real-World Cases with no comparable training examples, organized by decreasing similarity to the training data. A GPT evaluator assesses similarity using task objective, technical method, structural form, and application context. External evaluation uses 30 FlowBench tasks and 20 tasks sampled from the Creative and Complex categories of ComfyBench, not the complete benchmarks.

VND measures the diversity of valid node types in generated workflows, not task success. NodeComp measures the fraction of nodes whose types are registered, nonempty, and involved in at least one connection. LinkComp checks whether each node's declared inputs exist, output references are valid, and output-slot indices are in bounds. It is the proportion of nodes passing those checks, not the fraction of correct edges.

The following notation summarizes the prose definitions in the main text; it is not an original numbered equation from the paper. The main text does not fully specify aggregation across workflows:

\[ \mathrm{NodeComp}=100\frac{|\mathcal{V}_{\mathrm{valid,linked}}|}{|\mathcal{V}|},\qquad \mathrm{LinkComp}=100\frac{|\mathcal{V}_{\mathrm{IO\text{-}consistent}}|}{|\mathcal{V}|}. \]

TaskCons is a GPT-based score of alignment between the workflow's functional structure and its instruction. Pass requires successful ComfyUI execution without structural or runtime errors; Resolve additionally requires the output to satisfy the task, assessed through manual execution and human inspection. Tables 4 and 6 use the name Solve, retained below as Resolve / Solve where appropriate. The 86.9% TaskCons and 86.9% average Pass on the curated set happen to share a value but measure different things.

The following table selects structural metrics from original Table 1 and places them alongside the category-average execution metrics from original Table 2. All percentages are reported values. Average Pass / Resolve averages the three category percentages and should not be interpreted as a pooled fraction with denominator 30.

Method on curated test set	VND	NodeComp (%)	LinkComp (%)	TaskCons (%)	Average Pass (%)	Average Resolve (%)
GPT-5 + Few-Shot	20	36.7	36.7	34.7	26.4	16.7
GPT-5 + RAG	42	56.7	56.7	53.6	48.1	25.6
GPT-5 + ComfyAgent	40	56.7	56.7	55.1	36.4	29.4
Ours	152	96.3	98.3	86.9	86.9	61.1

Compared with ComfyAgent, average Pass improves by 50.5 percentage points and Resolve by 31.7 points. Cross-distribution evaluation, however, shows that structural improvement does not necessarily translate into a comparable increase in task completion. The next table reports results from original Tables 3 and 4:

External test set	Method	VND	NodeComp (%)	LinkComp (%)	TaskCons (%)	Pass (%)	Resolve / Solve (%)
ComfyBench, 20 tasks	GPT-5 + RAG	45	60.0	60.0	57.7	40.0	25.0
ComfyBench, 20 tasks	GPT-5 + ComfyAgent	43	65.0	65.0	59.3	35.0	20.0
ComfyBench, 20 tasks	Ours	138	79.8	82.7	85.0	50.0	25.0
FlowBench, 30 tasks	GPT-5 + RAG	42	73.33	73.33	67.43	46.7	26.7
FlowBench, 30 tasks	GPT-5 + ComfyAgent	41	53.33	53.33	50.80	40.0	26.7
FlowBench, 30 tasks	Ours	145	94.4	98.0	93.4	66.7	36.7

Ablation Study

Original Table 6 trains Qwen3-32B Q2P to map instructions directly to pseudocode as a comparison for hierarchical modeling. This analyzes the overall modeling choice, rather than removing individual modules under the same backbone and inference budget. It cannot isolate the contributions of knowledge inversion, auxiliary reasoning supervision, or self-refinement.

Modeling approach	VND	NodeComp (%)	LinkComp (%)	Pass (%)	Solve (%)
Qwen3-32B Q2P, instruction directly to pseudocode	148	92.3	95.8	56.7	33.3
Ours, Qwen3-14B hierarchical system	152	96.3	98.3	86.9	61.1

Structural differences are modest, but Pass / Solve differ by 30.2 / 27.8 percentage points. This supports the possibility that intermediate strategies help complete the intended processing task. It does not prove that a smaller parameter count is intrinsically better or that hierarchical supervision accounts for the entire gain.

Key Findings

• On the curated Real-World Cases category, Pass / Resolve reaches 90.0% / 50.0%, leaving a 40.0-point gap between execution and task completion. That gap reveals residual decision errors more directly than connection completeness alone.
• ComfyBench TaskCons is 85.0%, yet Resolve is only 25.0% and ties RAG. The paper's broad claim of outperforming every baseline on all metrics should therefore not be repeated without qualification.
• In original Table 5, the method ranks first under both GPT-5 and Gemini-2.5, with TaskCons scores of 86.9% / 85.6%. However, ComfyAgent changes from 55.1% to 40.8%, reversing its order relative to RAG. The two judges support the leading trend, not the elimination of judge bias.

Highlights & Insights

• Supervision is defined over relationships between a graph and its explanatory hierarchy, rather than only the final graph. In expert-workflow repositories without user queries, inversion offers a way to construct training pairs, provided that the credibility of synthetic intent and rationale is managed explicitly.
• Separating rule-based serialization from experience-dependent design addresses the task more directly than asking the model to handle every character-level detail. For data-processing pipelines or tool-call graphs, a transferable approach is to first define an intermediate representation that preserves types and dependencies, then train intent-to-structure mappings.
• More node types do not automatically mean better workflows. A valuable aspect of the evaluation is the distinction between structure, execution, and goal completion, which prevents a complex and valid graph from concealing an omitted user requirement.

Limitations & Future Work

• The authors observe that rare nodes are often replaced with common ones, potentially breaking functional dependencies and increasing structural errors. Better rare-node coverage is a direct improvement path; reinforcement-style optimization remains a proposal rather than an evaluated solution here.
• Data cleaning favors standardized, common nodes and excludes very small or large graphs. Long-tail plugins and highly complex workflows therefore fall outside the strongest evidence. Evaluation should be stratified by node coverage and graph size instead of relying only on a larger sample count.
• The curated test contains only 30 tasks, and the external subsets contain 20 and 30 tasks. The paper does not report multi-seed error bars, significance tests, or a complete human-scoring protocol. GPT-based similarity grouping and synthetic task construction warrant further examination of data independence, but do not establish leakage by themselves.
• Revision counts, call costs, latency, parameter-filling failure rates, and node-environment versions are incompletely reported. The paper names the GPT-5 interface as gpt-5-turbo, 2025 release; the local cache supplies no reproducible configuration, so this string is not treated as a verified callable model identifier.
• A more discriminating follow-up would hold the backbone and call budget fixed while separately removing the strategy layer, auxiliary reasoning supervision, and both refinement stages. Runtime failures and visual-goal failures should then be measured in an identical node environment to determine where the improvement originates.

Related Work & Insights

• Compared with ComfyAgent / ComfyMind: the paper characterizes these approaches as constraining composition with templates or meta-nodes, with ComfyMind additionally supporting tree-based rollback. The proposed method learns strategy-to-node-topology mappings to broaden expressiveness, but the main text does not provide a direct quantitative comparison with ComfyMind.
• Compared with ComfyGPT / ComfyUI-R1: these are also workflow-learning systems involving fine-tuning or reinforcement learning. The principal distinction here is explicit supervision at the strategy, reasoning-explanation, and skeleton levels; they should not be reduced to generic models without workflow knowledge.
• Compared with RAG and direct Q2P: RAG reuses examples, Q2P directly maps requests to structure, and this method inserts an inspectable strategy representation. The external results suggest that retrieval and hierarchical learning need not be mutually exclusive, especially when node-document retrieval can cover long-tail functionality missing from training. This is an extension idea, not a reported experiment.

Rating

• Novelty: 4/5. Inverting real workflows into multilevel supervision and selectively learning transitions is a clear contribution, although hierarchical planning and intermediate representations are established ideas.
• Experimental Thoroughness: 3/5. External benchmarks, execution checks, two judges, and the Q2P comparison are useful, but test sets are small and module-level ablations with controlled inference budgets are missing.
• Writing Quality: 3/5. The central argument and framework diagram are clear, but some claims of universal superiority exceed the tables, and Resolve / Solve terminology and reproducibility details need clarification.
• Value: 4/5. The approach offers a practical modeling strategy for specialist visual workflows, while actual task-completion rates show that reliability and environment compatibility still require substantial engineering.