Skip to content

Learn2Fold: Structured Origami Generation with World Model Planning

Conference: ECCV 2026
Paper: ECCV Official Page
Area: LLM Agent
Keywords: origami generation, program induction, world model, model predictive control, constraint-aware generation

TL;DR

The paper casts origami folding-sequence generation as conditional program induction over a crease-pattern (CP) graph: an LLM proposes structured folding actions from a text goal, a learned graph-structured world model predicts residual states and per-edge violation probabilities by short-horizon rollout before execution, and a deterministic Level-0 constraint kernel performs hard verification inside an MPC search loop. On a 25-category origami benchmark it lifts step-level F1 from 0.266 (best baseline) to 0.739 and macro category success rate from 0.675 to 0.891.

Background & Motivation

Generative AI has become remarkably good at synthesizing images, videos, and 3D assets, but most of these successes live at the static or perceptual level: the output only has to look right, while physical executability and execution constraints are ignored or only weakly enforced. Pushing generative models from "visually plausible" to "physically executable process" remains open, and the difficulty is sharpest for tasks that demand long-horizon reasoning under strict geometric and topological constraints. Cloth folding looks adjacent but benefits enormously from the material: cloth is compliant and error-tolerant, so local inaccuracies get absorbed by smoothing and deformation and learning-based methods can recover from imprecise actions. Origami sits in a fundamentally different regime. Folding is governed by strict geometric axioms and topological constraints, and a single misplaced crease is not a local artifact — it can violate the surface topology and render every subsequent fold mathematically infeasible. Discrete topological changes and continuous geometric motions must therefore be coordinated precisely over long horizons, with almost no room for error.

The two existing lines of work each supply only half of what is needed. Optimization-based methods such as TreeMaker and Origamizer formulate generation as a constrained optimization problem (circle packing / tuck folding) and can mathematically guarantee that a target mesh folds from a single sheet, producing simulation-ready, physically grounded crease patterns; but they require a precise 3D mesh as input, so they cannot consume sparse inputs like a single image or a text prompt, and they are highly sensitive to topological errors, where a slight violation in the CP graph makes the whole optimization infeasible. On the other side, generative foundation models including LLMs and VLMs, trained on large-scale multimodal origami videos, images, and tutorials, can produce descriptive tutorials or high-level folding guidance, but because they optimize approximate visual plausibility rather than exact physical feasibility they frequently hallucinate geometries that look coherent yet violate folding constraints, and cannot deliver a physically executable process.

The core tension is thus whether the physical rigor and simulation-ready representation of computational origami can be retained while exploiting the strong semantic priors of language models — concretely, whether executable origami processes can be reconstructed from sparse text descriptions without giving up hard constraint-level guarantees. The core idea of this paper is to decouple semantic proposal from physical verification: the LLM only proposes candidate folding actions in CP-graph state space (the symbolic layer, never touching physics), physical feasibility is delegated to a learned graph-structured world model acting as a differentiable surrogate simulator that rolls out short-horizon futures and prunes branches leading to invalid states before execution, and a symbolic simulator performs the exact final constraint check — an LLM-proposes / world-model-lookahead / hard-verifier-backstop propose-verify loop.

Method

Overall Architecture

The input is a high-level semantic goal \(g\) (natural-language text) plus a crease pattern: a planar graph \(G=(V,E)\) whose vertices are points on the sheet and whose edges are creases. The output is an executable folding program — a sequence of structured folding actions, each specifying the graph elements to manipulate (which edge, which face) and continuous geometric parameters (dihedral angles, etc.). Four stages chain together. First, raw CP data is canonicalized into a permutation-invariant state representation so structurally identical patterns map to the same index space. Then a language model autoregressively proposes candidate actions in a unified token space. A graph-structured world model then rolls out each candidate over a short horizon, predicting how the fold changes the state and which edges it breaks. Finally, inside an MPC loop, a deterministic Level-0 constraint kernel filters out invalid candidates and the surviving ones are ranked by a fused score of proposal likelihood, goal progress, and constraint satisfaction; if nothing survives the filter (or the best score falls below a threshold), the predicted violation mask is converted into a negative constraint and candidates are re-sampled under the updated constraint set. Inference is therefore a search loop with hard filtering rather than a single decode.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["text goal + crease pattern"] --> B["Canonicalized CP-graph state representation"]
    B --> C["Structured action generation with a language model"]
    C --> D["Graph-structured world model"]
    D --> E
    subgraph E["Graph-guided MPC planning with loopback re-sampling"]
        direction TB
        E1["sample K candidate actions"]
        E2["Level-0 hard verification filter"]
        E3["world-model lookahead scoring"]
        E1 --> E2 --> E3
        E3 -->|no valid candidate or low score<br/>add negative constraint and re-sample| E1
    end
    E3 --> F["executable folding sequence"]

Key Designs

1. Canonicalized CP-graph state representation: structurally identical origami lands in the same index space

An origami instance is written as \(O_t = (G, s_t)\) — half static topology, half dynamic state. The static CP is a planar graph whose vertices are 2D points on the sheet (normalized to \([0,1]^2\)), and each edge carries an initial crease type \(z_j^0 \in \{\text{M}, \text{V}, \text{U}\}\) for mountain, valley, and unknown. The problem is that raw CP data uses arbitrary vertex indexing, so the same structure under a different numbering becomes a different set of indices and the model fits instance-specific IDs instead of folding patterns. A deterministic canonicalization map \(\Phi: G \rightarrow G^*\) fixes this: vertices are reindexed by lexicographic sorting of their coordinates, edges are then reindexed by their sorted endpoint indices, and to remove orientation bias the training data is augmented with dihedral symmetries (rotations and reflections) applied to the vertex coordinates before canonicalization. Structurally identical patterns are thereby guaranteed to land in the same index space.

The dynamic state is a state vector \(s_t = (\alpha_t, \rho_t, z_t, \psi_t, b_t, t)\): \(\alpha_t \in [-\pi, \pi]^{|E|}\) gives the signed dihedral angle of every edge (how far it is folded, with the sign separating mountain from valley), \(\rho_t \in [0,1]^{|E|}\) the folding progress ratio, \(z_t\) the current crease types, \(\psi_t\) the global frame angle, \(b_t\) an MV-flip flag, and \(t\) the step counter. Keeping "how far it is folded" separate from "which way it folds" is what later lets the world model make residual predictions on individual edges.

2. Structured action generation with a language model: packing hybrid discrete-continuous actions into one token space

The folding action space is inherently hybrid: it requires selecting discrete graph elements (which edge to fold, which face to move) and continuous geometric parameters (how many degrees). Asking an LLM to emit geometric numbers directly is unstable. The authors unify both modalities into one vocabulary \(\Sigma = \Sigma_{\text{ops}} \cup \Sigma_{\text{graph}} \cup \Sigma_{\text{geo}}\): continuous geometric parameters are quantized into discrete bins in \(\Sigma_{\text{geo}}\), indices on the canonicalized graph are mapped to semantic tokens in \(\Sigma_{\text{graph}}\), and operation primitives go to \(\Sigma_{\text{ops}}\). The control problem becomes autoregressive sequence modeling, so the model can capture joint dependencies between topological intent and geometric specification within a single sequence — it first says what to do, then to what extent.

The policy \(\pi_\theta(a_t \mid C_t)\) is conditioned on a context \(C_t = (g; G^*, s_t)\), i.e. the semantic goal, the canonicalized graph structure, and the current dynamic state. Because the graph is canonicalized, the policy learns structure-invariant folding motifs such as the rabbit-ear fold rather than memorizing instance-specific vertex indices. Training is maximum likelihood on expert trajectories, i.e. the sum of per-token negative log-likelihoods of each action sequence under teacher forcing: \(\mathcal{L}_{\text{policy}}(\theta) = \mathbb{E}_{(G, a^*) \sim \mathcal{D}}\left[-\sum_t \sum_k \log \pi_\theta(a_{t,k} \mid C_t, a_{t,<k})\right]\) (equation (1) is corrupted in the extracted cache; reconstructed from the readable parts, ⚠️ refer to the original paper). This supervised pre-training instills the grammar of valid folding operations; the implementation is a lightweight decoder-only transformer emitting structured actions under a fixed JSON schema, adapted with LoRA.

3. Graph-structured world model: sparse residual dynamics on the graph instead of expensive mesh simulation

The policy's actions only say "this looks like the right fold", they do not guarantee feasibility, yet calling a mesh-level simulator for lookahead at every step is prohibitively expensive. The authors therefore learn a differentiable surrogate simulator \(\mathcal{M}_\phi\) that operates directly on the graph state \(s_t\) — the most fundamental difference from pixel-based world models, which have no explicit geometric constraints and whose learned dynamics cannot be tied to "which creases just broke". It predicts three quantities: a sparse residual state update \(\Delta\hat{s}_t\), a per-edge locality mask \(\hat{m}_t \in [0,1]^{|E|}\), and a per-edge constraint-violation likelihood \(\hat{c}_{t+1} \in [0,1]^{|E|}\); the state update is confined to the edges the mask selects:

\[\Delta\hat{s}_t,\ \hat{m}_t,\ \hat{c}_{t+1} = \mathcal{M}_\phi(s_t, a_t), \qquad \hat{s}_{t+1} = s_t + \Delta\hat{s}_t \odot \mathrm{expand}(\hat{m}_t)\]

where \(\mathrm{expand}(\cdot)\) broadcasts the per-edge mask to all state channels. Two things matter here. First, sparsity: a single fold physically affects only a local set of edges, so explicitly predicting which edges are affected matches the structure of the data better than densely rewriting the whole state, and is easier to learn. Second, framing feasibility as an attributable per-edge probability rather than a binary label — \(\hat{c}\) is the soft counterpart of the hard violation mask \(m\) — is exactly what later lets the planner know why something fails and where the problem sits. The world model is trained on large-scale transitions produced by the deterministic Level-0 constraint engine (about 76,000, synthesized from expert demonstrations plus constraint-guided perturbations). Crucially the perturbations are made near the feasibility boundary, so the model sees both feasible and infeasible outcomes; trained only on successful samples it would simply learn that everything is fine.

4. Graph-guided MPC planning with loopback re-sampling: hard verification as backstop, soft scoring to pick

At test time the system performs constrained lookahead search on the CP graph. At every step it first draws K candidate actions from the policy by nucleus sampling (\(N = 8\) proposals per step in the implementation). These candidates are not taken on trust: each is passed through Level-0, a deterministic constraint kernel returning whether the fold is valid \(v_t \in \{0,1\}\), the reason for invalidity \(r_t\), and an affected-edge mask \(m_t \in \{0,1\}^{|E|}\); invalid candidates are discarded outright. The surviving ones go through world-model rollout and are ranked by a fused objective:

\[a_t^\star = \arg\max_{a_t^{(k)} \in \mathcal{A}_t^{\text{valid}}} \left[ \frac{1}{|a_t^{(k)}|}\log \pi_\theta(a_t^{(k)} \mid \mathcal{C}_t) - \lambda_{\text{goal}}\, U_{\text{goal}}\!\left(\hat{s}_{t+1}^{(k)}\right) + \lambda_{\text{cst}}\, \log\!\Big(\frac{1}{\epsilon + \|\hat{c}_{t+1}^{(k)}\|_\infty}\Big) \right]\]

The three terms are the proposal log-likelihood normalized by action token count (without normalization long actions are systematically penalized), a goal-progress term measuring how far the predicted next state is from the target, and a constraint-satisfaction term taking the log of the reciprocal of the maximum per-edge violation probability, with \(\epsilon > 0\) only preventing numerical blow-up; \(\lambda_{\text{goal}}\) and \(\lambda_{\text{cst}}\) balance the two ends (equation (6) is corrupted in the extracted cache and the original paper gives no definition of \(U_{\text{goal}}\); reconstructed from the readable parts, ⚠️ refer to the original paper).

What distinguishes this loop from "retry on failure" is the failure handling. When the valid candidate set is empty, or the best score falls below a threshold \(\tau\), the system does not roll back the whole step. It takes the top-M edges with the highest predicted \(\hat{c}\), constructs them as a negative constraint, injects that into the context, and re-samples candidates under the updated constraint set. Failure thus becomes causal attribution instead of a binary signal: the model learns not only that the step fails but which edges cause it, so the next round of sampling actively avoids that region. This is precisely the advantage over the reactive rollback of BrickGPT — rollback feedback only arrives after execution and carries just success or failure, and once sequences grow long, exhaustive backtracking becomes computationally prohibitive; here the negative constraint is reusable local information, which is why recovery stays efficient even on out-of-distribution crease patterns.

A Worked Example

Take one step of folding a crane. The current state is the canonicalized CP graph plus state vector, and the semantic goal \(g\) is "form the crane's head". The LM proposes \(N = 8\) candidate actions for this step (different target edges crossed with different fold angles), and all of them go into Level-0: several are judged invalid — say they would cause self-intersection between adjacent faces or break flat-foldability — and are discarded together with their reasons, leaving three valid candidates. Each of the three is rolled out through the world model to obtain its residual state \(\hat{s}_{t+1}\) and per-edge violation probabilities \(\hat{c}_{t+1}\). One candidate, although it has the highest proposal likelihood, is predicted to push two edges to violation probability near 1, so its constraint term crushes its score; another with slightly lower likelihood ends up closer to the goal with low violation probability and is selected, executed, and written into the folding program. If all eight candidates had been rejected by Level-0, or the best score had fallen below \(\tau\), the planner would take the top-M edges by violation probability as a negative constraint and return to sampling to propose a fresh batch — repeating this sample → verify → score → (if needed) add-negative-constraint-and-resample cycle at the same position until an executable action is selected, then moving on to the next step.

Loss & Training

Training splits into two parts. The world model is trained by supervised learning on large-scale folding transitions synthesized by the Level-0 simulator: about 76,000 transitions from expert demonstrations plus constraint-guided perturbations, 50 epochs, roughly 30 hours on a single NVIDIA RTX Pro 6000 GPU (the main text does not state the world model's supervised loss, presumably regression/classification over the residual state, mask, and violation probability, ⚠️ refer to the original paper). The language model is trained by maximum likelihood for conditional program induction on roughly \(10^4\) expert folding steps augmented with simulator-verified perturbations (the cache extracts "104", which from context is presumably \(10^4\), ⚠️ refer to the original paper); it converges within 6 hours with LoRA adapters on the same hardware and emits output constrained by a fixed JSON schema. Inference runs as an MPC loop: the LM proposes \(N = 8\) candidates per step, the simulator filters invalid ones, and the world model scores the rest by short-horizon rollout before the final action is selected. All experiments use fixed random seeds.

Key Experimental Results

Metrics first. Step level uses Precision, Recall, and F1, measuring the correctness and coverage of predicted structured folding actions under a unified action schema. Trajectory level uses two: Category Success Rate (Cat-SR) is defined as the fraction of folding sequences that successfully complete the target origami within a category, macro-averaged across categories to mitigate class imbalance (the decision relies on the simulator's constraint verification; the paper does not detail the judging protocol, ⚠️ refer to the original paper); Edge-IoU measures whether a predicted action affects the correct set of creases, as the intersection-over-union between the predicted affected-edge set and the simulator-derived ground truth — note this ground truth also comes from the simulator and is thus same-source as the verifier.

Data and baselines: with no standard benchmark for origami process generation, the authors curate a held-out set of 25 categories stratified into three difficulty tiers — Simple (10 categories: paper airplanes, hearts, cups), Intermediate (10: boats, flowers), and Complex (5: insects, cranes, dragons, involving high-frequency folding and strict circle-packing constraints). The test set contains 3,840 text prompts, from which 1,150 cases are selected for validation, with two independent runs per prompt per method, yielding 7,680 results per method. Baselines are BrickGPT (fine-tuned on the proposed OrigamiCode dataset, filtering unstable steps via physics-aware rollback), GPT-5.1 and GPT-5.2 (prompted with in-context examples to emit structured folding programs), and Gemini (prompted).

Main Results

Method Precisionµ ↑ Recallµ ↑ F1µ ↑ Edge-IoU ↑ Cat-SR_macro ↑
Gemini (prompted) 0.2874 0.4213 0.3420 0.1126 0.4942
GPT-5.1 (prompted) 0.2625 0.2996 0.2663 0.0937 0.6753
GPT-5.2 (prompted) 0.1243 0.3575 0.1648 0.1322 0.1600
BrickGPT (finetuned) 0.3969 0.2250 0.2461 0.0505 0.5455
Learn2Fold (Ours) 0.7661 0.7113 0.7394 0.5820 0.8912

Ablation Study

In-distribution (IID) CP holdout:

Config Step Valid ↑ Traj SR ↑ Goal Dist ↓
LM 70.8% ± 45.5% 22.2% ± 45.5% 0.796 ± 0.194
LM+Level0Sim 49.3% ± 7.2% 23.7% ± 0.0% 0.515 ± 0.130
LM+WM 54.2% ± 49.8% 25.0% ± 43.3% 0.759 ± 0.214
LM+WM+Level0Sim (Ours) 64.2% ± 41.8% 33.3% ± 47.1% 0.855 ± 0.196

Out-of-distribution (OOD) CP holdout:

Config Step Valid ↑ Traj SR ↑ Goal Dist ↓
LM 47.6% ± 29.2% 20.7% ± 55.5% 0.633 ± 0.192
LM+Level0Sim 36.5% ± 18.0% 23.3% ± 38.3% 0.256 ± 0.600
LM+WM 32.3% ± 28.7% 17.8% ± 51.7% 0.560 ± 0.248
LM+WM+Level0Sim (Ours) 41.2% ± 32.3% 27.7% ± 50.1% 0.487 ± 0.353

Key Findings

  • Locally valid is not the same as foldable to the end. With the LM alone, step-level validity is as high as 70.8% on IID while trajectory success is only 22.2%; long-horizon error accumulation is the real bottleneck. This also explains the gap between F1 and Cat-SR for LLM baselines in the main table (GPT-5.2 reaches recall 0.358 but precision only 0.124).
  • The world model buys global progress, not local safety. Going from LM to LM+WM raises IID trajectory success from 22.2% to 25.0% and lowers goal distance from 0.796 to 0.759 (OOD: 0.633 → 0.560), while step-level validity actually drops — short-horizon lookahead prefers actions that are locally risky but better over the long run.
  • The three components are complementary and all are needed. Adding Level-0 hard verification gives the full system the highest trajectory success in both settings (IID 33.3%, OOD 27.7%) and restores the step-level validity that the world model had pulled down. The authors argue on this basis that LLM proposal, world-model lookahead, and symbolic verification play distinct roles.
  • Some numbers conflict and should be quoted carefully. On IID the full system has the highest Goal Dist at 0.855 even though the metric is marked as lower-is-better, contradicting the prose claim of "further reducing final goal distance"; on OOD its 0.487 is also above LM+Level0Sim's 0.256. In addition every ablation entry carries an enormous standard deviation (±45.5%, ±55.5% level), indicating a small validation sample and limited robustness of any single-point conclusion (⚠️ refer to the original paper).
  • Each baseline has its own skew. BrickGPT's rollback gives it higher precision than the LLM baselines (0.397) but recall of only 0.225, showing that its trial-and-error strategy produces coarse, incomplete actions that fail to cover the full required sequence; the LLM baselines are semantically plausible but geometrically mismatched, typically collapsing within a few steps in the qualitative study, while BrickGPT stays stable for the first three to four steps and then loses long-term consistency as sequences grow.

Highlights & Insights

  • Turning failure into an attributable constraint rather than a retry signal. The world model emits per-edge violation probabilities and the planner converts them into a negative constraint for re-sampling, so failure information changes from binary into "these edges are the problem" and can be reused. This is the most transferable idea in the paper: any generate-and-verify task can ask whether its verifier can say where things went wrong, and whether that "where" can be written back into the generation condition.
  • State-level world models beat pixel-level ones in hard-constraint domains. Sparse residual updates directly on the graph state come with the inductive bias that one action affects only a local region, and the predicted mask doubles as an interpretable intermediate quantity — none of which survives a move to pixel or latent dynamics.
  • Canonicalization is a general way to decouple "instance ID" from "structural pattern". Lexicographic reindexing plus dihedral symmetry augmentation costs almost nothing, yet makes the policy learn motifs instead of IDs; the same recipe transfers to assembly, circuit, or molecule synthesis — any graph-structured program generation.
  • Manufacture negative samples near the boundary. A world model's value lies in separating feasible from infeasible, so training only on successes yields a constantly-true discriminator; deliberately applying near-boundary perturbations so both outcomes appear in the data is a general lesson for learning feasibility discriminators.

Limitations & Future Work

  • The evaluation is self-built and small. With no standard benchmark, 25 categories, 1,150 validation cases, and two samples per prompt, ablation variance reaches ±50%, so the statistical strength of the main claims is limited; the Cat-SR judging protocol (who decides, and at which step success counts) is not spelled out.
  • "Physically valid" only extends to geometry/topology. All validation is at the simulation and symbolic level; the generated sequences are never executed on real paper (by a robot or a human) as a control. Satisfying flat-foldability and avoiding self-intersection is not the same as a real sheet actually folding.
  • Evaluator and verifier are same-source, risking circularity. Edge-IoU's ground truth comes from the simulator, while planning-time Level-0 verification uses the same constraint kernel — the same ruler defines both "correct" and "judged correct".
  • Lookahead horizon is never reported. The world model is trained on short-horizon rollouts, but rollout length and how error accumulates with it are not reported, and that is precisely what decides whether Complex-tier sequences of dozens of steps are handled.
  • Strong dependence on Level-0's correctness and coverage. Hard verification is the backstop of the whole pipeline; if the constraint kernel misses a class of violations, neither the world model nor the semantic proposal can compensate.
  • Possible directions: replace Level-0 with an independent physical simulator or real-hardware feedback to break the same-source problem; report rollout length versus accumulated error; re-evaluate on a larger benchmark with standard splits; feed the failure reason \(r_t\) back to the language model explicitly (currently only the violation mask is used).
  • vs BrickGPT: Both are "autoregressive proposal + physics-aware". The difference is the failure-handling mechanism: BrickGPT relies on reactive rollback, executing first and retreating afterwards, with only success/failure signals and backtracking costs that explode with sequence length; this paper uses a learned world model for pre-execution short-horizon lookahead and turns failure into reusable local constraints via violation-mask attribution. The cost is an extra world model that must be trained and can itself predict wrongly.
  • vs TreeMaker / Origamizer: The optimization route uses circle packing and tuck folding to mathematically guarantee that a mesh folds from one sheet and produces simulation-ready CPs, but it demands an exact 3D mesh input, is deterministic, and is extremely sensitive to topological errors. This paper treats generation as sequential decision making from sparse text and allows recovery from intermediate errors, but gives up the mathematical guarantee — "valid" degrades to "passes Level-0".
  • vs world models such as Dreamer / Genie / AVID: Those model dynamics in pixel or latent space for control or generation in visual domains; they do not directly enforce hard discrete geometric constraints and do not produce executable structured programs. This paper moves the world model onto CP-graph states, predicting per-edge geometric quantities and violation probabilities in service of symbolic program generation.
  • vs LLMs/VLMs emitting origami tutorials directly: Such methods give descriptive, semantically reasonable guidance (GPT-5.1's Cat-SR of 0.675 is not low) but are poor at the action level (F1 0.266) because they lack explicit geometric state tracking and feasibility checking. This paper's value proposition is exactly "semantics to semantics, geometry to geometry".

Rating

  • Novelty: ⭐⭐⭐⭐ Formulating origami generation as a neuro-symbolic planning loop of LLM proposal + graph-structured world-model lookahead + symbolic hard verification is a combination rarely seen before, with a state-level world model serving constraint-aware symbolic program generation.
  • Experimental Thoroughness: ⭐⭐⭐ Main results plus IID/OOD ablations are complete, but the benchmark is self-built, the validation sample is small with huge variance, and there is no analysis of world-model rollout length or any real-paper comparison.
  • Writing Quality: ⭐⭐⭐⭐ The architecture and motivation are clearly told and the propose-verify thread runs through the paper; however several key definitions are missing (\(U_{\text{goal}}\), the world-model loss, the Cat-SR protocol) and the Goal Dist direction in the ablation tables conflicts with the prose.
  • Value: ⭐⭐⭐⭐ It offers a reusable paradigm for constraint-aware long-horizon planning (attributable failure + state-level lookahead + hard-verifier backstop) that transfers to assembly and manipulation, and origami is a clean testbed for it.