Skip to content

TRACE: Triplet Recovery via Arrowhead-Centric Extraction for Flowchart Understanding

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/nju-websoft/TRACE
Area: LLM Reasoning
Keywords: Flowchart Understanding, Arrowhead-Centric Extraction, Semantic Triplets, Vision-Language Models, Neuro-Symbolic Reasoning

TL;DR

TRACE transforms flowchart parsing from holistic end-to-end generation into localized, arrowhead-anchored triplet extraction, synthesizing aligned training data from textual sources and outperforming large-scale models like GPT-5 across nine benchmarks while providing verifiable per-edge traceability.

Background & Motivation

Flowcharts serve as a ubiquitous visual lingua franca for modeling business workflows, approval sequences, software control flows, and user interactions across modern enterprise operations. Accurately recovering the underlying semantics of flowchart images into structured, directed semantic triplets of the form \((s, c, t)\)—where \(s\) and \(t\) denote the text labels of the source and target nodes, and \(c\) denotes an optional condition label along the connecting edge (e.g., "Yes", "No", or inequality thresholds)—is foundational to high-stakes downstream applications such as visual question answering (QA) over technical documentation and retrieval-augmented generation (RAG) grounded in procedural workflows. However, achieving robust and complete triplet extraction remains an arduous computer vision and natural language processing challenge due to diverse rendering styles ranging from hand-drawn sketches to hierarchical BPMN pools and lanes, dense graph topologies, and intricate connector geometries.

Existing diagram parsing literature predominantly adheres to two contrasting paradigms, each suffering from severe structural failure modes. The classical detection-reconstruction paradigm operates via a sequential multi-stage pipeline: it detects isolated graphical primitives (nodes, text containers, connector paths, and arrowheads) via standard object detectors and subsequently attempts to reconstruct the directed topology through heuristic geometric clustering and spatial contact-point matching. This bottom-up pipeline is brittle and fails to generalize: hand-drawn connectors, unusual custom node shapes, or overlapping edges violate hard-coded geometric assumptions, and early detection errors catastrophically cascade into global graph corruption. In response, modern end-to-end vision-language model (VLM) approaches feed the entire flowchart image into a multimodal backbone to directly emit all triplets in a single decoding pass. While this eliminates heuristic assembly, it introduces fundamental representation weaknesses: visual attention dilutes across spatially dispersed and visually fine connectors, resulting in missed edges, pervasive edge direction reversals, and an opaque black-box output where individual erroneous predictions cannot be localized or corrected.

Cognitive science investigations into human visual perception reveal that people do not parse dense, complex diagrams monolithically. Instead, human readers perform localized visual searches guided by salient directional anchors, segmenting high-density visual content into manageable units to prevent extraneous cognitive load. In flowcharts, the sparse, readily detectable arrowhead functions as the definitive directional cue for each directed dependency edge. Core idea: TRACE (Triplet Recovery via Arrowhead-Centric Extraction) bridges the gap between heuristic assembly and end-to-end generation by reformulating flowchart parsing into per-arrowhead localized extractions, using an automated text-to-SVG data synthesis pipeline to train an arrowhead detector and a focused VLM.

Method

Overall Architecture

TRACE decomposes diagram parsing into a three-stage pipeline that transitions from offline automated supervision synthesis to online localized extraction: 1. Stage 1: Offline Data Synthesis & Model Training: Automatically derives paired supervision from text-format flowchart sources (such as Mermaid, PlantUML, and BPMN XML) without manual human annotations, yielding whole-image arrowhead bounding boxes \(\mathcal{D}_{\text{det}}\) for detector fine-tuning and single-arrowhead highlighted images paired with structured triplets \(\mathcal{D}_{\text{vlm}}\) for VLM adaptation; 2. Stage 2: Online Arrowhead Detection: Given an arbitrary query flowchart image, a fine-tuned detector localizes every arrowhead and superimposes a high-contrast visual bounding box marker onto the original full-context canvas, producing \(n\) distinct per-arrowhead annotated images; 3. Stage 3: Online Triplet Extraction: A fine-tuned VLM inspects each annotated image independently under a structured prompt, tracing backwards along the marked connector to retrieve the source node text and forward to identify the target node and condition, subsequently aggregating all \(n\) local triplets into the complete graph.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Flowchart Source or Image"] --> B["Stage 1: Offline Data Synthesis & Model Training<br/>SVG geometric bounding box reconstruction"]
    B --> C["Stage 2: Online Arrowhead Detection<br/>Localize arrowheads and overlay visual markers"]
    C --> D["Stage 3: Online Triplet Extraction<br/>Trace source and target from highlighted anchor"]
    D --> E["Aggregated Directed Semantic Triplets"]

Key Designs

1. Offline Data Synthesis & Model Training: Automated supervision derivation without manual geometry annotation

Flowchart datasets notoriously lack ground-truth pixel-level bounding boxes for arrowheads, making the supervision of local detectors prohibitively expensive if done manually. TRACE bypasses this barrier by leveraging structured text sources (Mermaid, PlantUML, BPMN XML, INKML). For formats containing explicit layout coordinates (e.g., BPMN XML), bounding boxes and triplets are directly parsed. For topology-only languages (e.g., Mermaid), TRACE renders the graph into Scalable Vector Graphics (SVG), extracting endpoint text mappings from the <path> element's class attributes (LS- and LE- prefixes). Because SVGs lack explicit arrowhead bounding boxes, TRACE introduces a geometric reconstruction algorithm: letting \(Q = (x_{n-1}, y_{n-1})\) and \(T = (x_n, y_n)\) represent the terminal coordinates of the connector path, the unit tangent direction is \(\vec{u} = (T - Q) / \|T - Q\|\). Reversing along \(-\vec{u}\) by length \(\ell\) determines the arrowhead base center \(B = T - \ell \vec{u}\), which is expanded perpendicularly by \(w/2\) to locate base corners \(L\) and \(R\). The axis-aligned bounding box enclosing triangle \(\Delta TLR\) is padded by \(\rho\) (empirically \(\ell = 10\text{ px}, w = 10\text{ px}, \rho = 5\text{ px}\)). This pipeline delivers aligned pairs for both detector training \(\mathcal{D}_{\text{det}} = \{(I_j, \{b_k\}_{k=1}^{n_j})\}\) and VLM extraction \(\mathcal{D}_{\text{vlm}} = \{(I_j^{(k)}, \mathcal{P}, (s_k, c_k, t_k))\}\) at zero manual annotation cost.

2. Online Arrowhead Detection: Visual prompt anchoring to counter attention dilution

Monolithic end-to-end models struggle on dense flowcharts because visual attention diffuses over the wide expanse of blank space and crisscrossing lines, weakening the weak visual signal of thin connectors. In Stage 2, TRACE deploys a text-promptable SAM 3 detector (or closed-vocabulary YOLOv11x) fine-tuned with the prompt "arrowhead" to predict all candidate arrowhead regions \(\{b_k\}_{k=1}^n\). Instead of cropping out local image patches—which would sever multi-line node labels, truncate long connectors, and eliminate global spatial context—TRACE overlays a prominent blue rectangular bounding box (RGB (0, 0, 255), line width 3 pixels) directly onto the full-resolution image at each detected coordinate \(b_k\). This synthetic visual prompt serves as an explicit visual beacon, forcing the vision encoder to concentrate its representational capacity on the single designated connector during subsequent inference.

3. Online Triplet Extraction: Directionally grounded backward-forward tracing with diagnostic traceability

In Stage 3, each per-arrowhead annotated image \(I^{(k)}\) is queried by the fine-tuned VLM using a role-based structured prompt. The model follows an explicit reasoning protocol: (1) fixate on the blue bounding box; (2) determine edge direction from the arrowhead's physical orientation; (3) trace backward along the incoming line to read the source node label \(s_k\) and its container pool/lane; (4) inspect the edge stroke for textual decision conditions \(c_k\); and (5) trace forward past the arrowhead tip to identify the target node label \(t_k\). For converging branches that physically share an arrowhead, the prompt instructs the model to emit parallel paths separated by "||". The resulting localized extractions are parsed and aggregated into the final directed graph \(\mathcal{T}\). Crucially, every triplet remains explicitly indexed to a physical arrowhead location \(H_k\), providing unprecedented diagnostic transparency: if a downstream auditing tool or human verifier flags a specific relation as suspicious, the system can instantly highlight the exact arrow bounding box on the original diagram and re-query that single edge in isolation, eliminating the need to blind-resample the entire image.

Loss & Training

The framework trains the two online components with dedicated objectives: 1. Arrowhead Detector: SAM 3 is adapted using bipartite Hungarian matching, optimizing a linear combination of binary cross-entropy for presence classification, \(L_1\) loss, and Generalized IoU (GIoU) loss for bounding box regression (cost weights: classification 2.0, \(L_1\) 5.0, GIoU 2.0) across 8 GPUs using AdamW with learning rate \(8 \times 10^{-5}\) for 30 epochs; 2. Vision-Language Model: Open-source VLM backbones (Qwen3-VL-4B/8B, MiniCPM-V-4.5-8B, Gemma3-4B-IT, LLaVA-v1.6-7B) are adapted on \(\mathcal{D}_{\text{vlm}}\) via LoRA (\(r=32, \alpha=64\), dropout 0.05) applied to all linear layers in both the visual encoder and language model. Training uses DeepSpeed ZeRO-3 in bfloat16 for 10 epochs with a cosine learning rate scheduler, searching optimal batch sizes \(B \in \{8, 16, 32, 64, 128\}\) via validation loss.

Benchmark & Evaluation Protocol

Main Comparative Evaluation

Evaluated across nine diverse benchmarks: FlowVQA, CBD, FC_A, FC_B, FlowLearn, BPMN-VLM, and synthetic multi-density FlowGen (easy, medium, hard). Evaluation metrics report exact set-level triplet \(F_1\) and relaxed \(rF_1\) (edit similarity threshold \(\ge 0.85\) after canonical normalization).

Model Backbone / Method FlowVQA \(F_1\) FC_B \(F_1\) FlowLearn \(F_1\) BPMN-VLM \(F_1\) FlowGen-h \(F_1\)
Large-Scale VLMs (3-shot, CoT)
GPT-5 85.85 42.78 78.94 59.82 34.04
GLM-4.6V 86.93 49.76 60.28 56.71 37.78
Detection-Reconstruction Baselines
BlockNet – 0.88 – – –
Flowmind2digital – 2.60 – – –
Qwen3-VL-4B
End-to-end (ZS) 71.72 23.44 45.25 40.22 24.10
TRACE (ZS) 51.91 28.84 39.89 47.53 29.62
End-to-end (FT) 92.28 80.42 88.50 64.26 68.91
TRACE (FT) 93.77 84.32 93.61 87.07 74.82
Qwen3-VL-8B
End-to-end (FT) 92.57 80.59 91.46 71.18 70.66
TRACE (FT) 94.38 85.35 94.52 87.08 76.21

Ablation Study: Detector Variance & Error Propagation

Investigating sensitivity to the detector choice by comparing ground-truth oracle boxes TRACE (GT), SAM 3 detections TRACE (SAM3-FT), and closed-vocabulary YOLOv11x detections TRACE (YOLO-FT) using fine-tuned Qwen3-VL-4B.

Detector Configuration FlowVQA \(F_1\) / \(rF_1\) FC_B \(F_1\) / \(rF_1\) FlowLearn \(F_1\) / \(rF_1\) BPMN-VLM \(F_1\) / \(rF_1\) FlowGen-h \(F_1\) / \(rF_1\)
TRACE (GT) [Oracle Boxes] 93.77 / 95.26 83.34 / 88.60 94.06 / 98.69 87.19 / 89.56 79.70 / 79.81
TRACE (SAM3-FT) 93.77 / 94.47 84.32 / 88.94 93.61 / 98.25 87.07 / 89.39 74.82 / 74.95
TRACE (YOLO-FT) 93.43 / 94.06 85.07 / 89.81 92.99 / 97.56 87.09 / 89.52 74.87 / 75.01
End-to-end (FT) [No Decomposition] 92.28 / 92.72 80.42 / 87.07 88.50 / 96.50 64.26 / 66.62 68.91 / 69.10

Key Findings

  • Paradigm Shift Dwarfs Detection Imperfections: Ablation results verify that switching between SAM 3 and YOLOv11 changes \(F_1\) by under 1 pp, and both track oracle bounding boxes within 1–2 pp. In stark contrast, moving from whole-image End-to-end (FT) to TRACE yields an immense surge of +22.81 pp on BPMN-VLM (87.07 vs. 64.26) and +5.91 pp on dense FlowGen-h. The gain stems decisively from the arrowhead-anchored task formulation rather than detector precision;
  • Compact Models Outperform Giant Frontier Models: With Qwen3-VL-4B, fine-tuned TRACE substantially outperforms 3-shot GPT-5 across the board—scoring 93.77 vs. 85.85 on FlowVQA, 87.07 vs. 59.82 on BPMN-VLM (+27.25 pp), and 74.82 vs. 34.04 on FlowGen-h (+40.78 pp)—demonstrating that architectural decomposition overcomes the fundamental limits of monolithic visual attention;
  • Downstream QA Superiority: When paired with tool-calling graph reasoning engines, TRACE triplets enable a 4B reasoner to achieve 98.17% accuracy on FlowLearn, outperforming TextFlow supplied with ground-truth Mermaid code (88.33%, +9.84 pp). Deterministic symbolic query tools (e.g., get_outgoing, find_path) avoid the parsing ambiguities inherent in free-form textual flowchart descriptions.

Highlights & Insights

  • Neuro-Symbolic Cognitive Grounding: Rather than demanding that vision encoders miraculously master dense, long-distance topological tracing in one glance, TRACE respects cognitive load constraints by anchoring attention on localized directional cues (arrowheads), achieving high-precision structural extraction without brittle heuristic code;
  • Data-Free Geometric Synthesis: By exploiting the mathematical representation of vector SVG paths, TRACE autonomously calculates exact arrowhead boundaries and topological labels, providing an infinite engine for training vision detectors and multimodal reasoners without human annotation.

Limitations & Future Work

  • Undirected Edges and Implicit Connectors: TRACE relies strictly on arrowheads as directional anchors. Diagrams with undirected association lines or implicit flow conventions (e.g., top-to-bottom without arrow tips) cannot be processed without substituting the arrowhead detector with an endpoint detector;
  • Inference Latency Trade-Off: Prompting the VLM once per detected arrowhead results in an average inference latency 2.14× that of single-pass end-to-end baselines (though multi-arrowhead batching with \(K=3\) successfully compresses this ratio to 1.33× at a negligible 0.49 pp drop in macro \(F_1\)).
  • vs. BlockNet / Flowmind2digital: Classical detection-reconstruction methods collapse on complex or hand-drawn benchmarks (scoring \(<3\%\) on FC_B) due to rigid shape dictionaries and fragile contact heuristics. TRACE bypasses shape classification entirely, letting the VLM read arbitrary node geometries while relying only on generic arrowhead anchors;
  • vs. TextFlow: TextFlow translates images into intermediate text code (Mermaid) to assist downstream QA, but dense diagrams still trigger halluncinations during text generation. TRACE produces verifiable, canonical triplets directly amenable to programmatic graph query execution.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant reframing of diagram comprehension into arrowhead-anchored local parsing, bridging the gap between heuristic assembly and end-to-end generation.
  • Empirical Rigor: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 9 distinct benchmarks, 5 open-source VLM backbones, cross-style transfer analysis, and real-world downstream QA.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear pedagogical flow, solid mathematical grounding of SVG geometry, and transparent error analysis.
  • Impact & Practicality: ⭐⭐⭐⭐⭐ High direct utility for enterprise document parsing, robotic workflow automation, and auditable visual intelligence systems.