NeurIPS2026 (task-list filing; the cached paper does not establish acceptance at this conference) Code Intelligence execution feedback line-by-line code generation classifier-free guidance execution traces inference-time search

Execution Guided Line-by-Line Code Generation¶

Conference: NeurIPS2026 (task-list filing; the cached paper does not establish acceptance at this conference)
arXiv: 2506.10948
Code: https://github.com/boazlavon/eg_cfg
Area: Code Intelligence
Keywords: execution feedback, line-by-line code generation, classifier-free guidance, execution traces, inference-time search

The reading version is arXiv v2, dated 2025-10-23. All results and methodological boundaries below refer to that version, not to confirmation of NeurIPS2026 acceptance.

TL;DR¶

At inference time, EG-CFG previews and executes several short continuations of a code prefix, injects runtime traces into the prompt, and generates code token by token with dual-distribution classifier-free guidance while refreshing feedback at line boundaries; with DeepSeek-V3-0324, the paper reports 96.6% / 99.4% / 69.9% accuracy on MBPP / HumanEval / DS-1000, respectively, but requires executable provided tests and additional search computation.

Background & Motivation¶

Code language models can produce plausible-looking programs without knowing a variable's runtime value, whether a loop processes every element, or what a library call actually returns. Methods such as Self-Debugging, MapCoder, and LPW generally generate a complete program before executing tests and repairing it. Such feedback can locate errors, but if the first few lines already choose an unsuitable data structure or state update, subsequent generation follows that choice until the entire block must be revised.

Earlier execution is not as simple as running unfinished code text. A short continuation may end after a function declaration, conditional branch, or loop header without the required body. Even after making it parseable Python, it may return nothing, raise an exception, or implement only part of the task. The paper therefore does not treat every intermediate candidate as a correct answer. Instead, it provides variable values, types, return values, and exceptions on concrete inputs as observations from which the model can assess the consequences of current choices.

The method moves execution checks to code-line boundaries and separates obtaining observations from committing the next line: tentative continuations produce runtime evidence, dual-distribution decoding incorporates it, and the actual program still grows token by token. Core idea: use real execution traces from short-horizon candidates to continually reshape token preferences for the next code line, rather than adding a repair instruction only after a complete answer fails.

Method¶

Overall Architecture¶

The input consists of a task description, tests available during inference, and the target Python function name; the output is a candidate program. EG-CFG does not train a new model: it first retains the model's reasoning prefix before the solution code block, then enters a loop of short-horizon candidate preview, execution-trace feedback, and dual-distribution decoding. Independently configured instances can explore the same task in parallel.

The diagram shows inference data flow within each instance. There is no training supervision because the paper introduces neither fine-tuning nor reward learning. The final test check is a controller stopping criterion, not generation supervision supplied by the hidden evaluation set.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    I["Task and provided tests"] --> A["Short-horizon<br/>candidate preview"]
    A --> B["Execution-trace<br/>feedback"]
    B --> C["Dual-distribution<br/>line-by-line guidance"]
    C -->|Refresh after newline| A
    C -->|Reuse feedback within line| C
    C -->|Code end or length limit| D["Independent parallel<br/>solution selection"]
    D -->|First solution passing available tests| O["Return program and stop other instances"]

Key Designs¶

1. Short-horizon candidate preview: inspect several possible continuations before committing an entire answer

The model first generates an initial response using the selected instruction template, locates the start of the final code block, and retains the preceding reasoning text. It also records a signal-insertion position so that feedback can enter a fixed location in the prompt instead of becoming part of the Python statements being written. Once code generation begins, each round produces several tentative continuations from the current committed prefix and stops after a specified number of newlines. These candidates are neither already selected next lines nor complete answers copied into the final program: they probe the short-term consequences of local choices.

The experiments fix the candidate count at \(s=3\) and use continuation lengths \(d\in\{2,3,6,8\}\). Short previews provide earlier feedback; longer ones are more likely to form a meaningful loop body or return path. Different configurations explore different lengths, without evidence that one length is universally optimal. The text calls candidate generation beam search, but Equation (6) and Appendix C describe temperature-based token sampling without specifying conventional beam retention and ranking rules. This note retains the authors' terminology while limiting the confirmed mechanism to generating multiple short continuation candidates, rather than supplying an invented standard beam-search algorithm.

Short continuations are frequently incomplete. The authors first attempt AST parsing, then add pass and retry if parsing fails, and otherwise progressively remove the last line and retry; extracted results are subsequently deduplicated. This addresses syntactic truncation only: an empty statement does not supply missing algorithmic logic, and a successful AST parse does not establish that variables are defined, return values are correct, or tests will pass. Deduplication can also leave fewer candidates than the original count, so three previews do not imply three distinct valid paths in every round.

2. Execution-trace feedback: preserve what happened instead of providing only a success or failure label

Tentative continuations use the existing code as context, and the extracted program fragments are executed on every provided test. The observed object should be understood as the partial program formed by the existing code prefix plus a candidate continuation, not arbitrary middle lines of a function body run in isolation. Both the main text and Appendix C abbreviate extraction and execution using the candidate symbol, without listing prefix concatenation as a separate step. The cache therefore does not establish where the implementation performs that concatenation, and it does not justify claiming that arbitrary fragments can run without context.

A custom debugger records four event types: call, line, return, and exception. Each event includes a source line number, mappings from variable names to values and types, and, where applicable, a return value and exception information. The model can thus observe not just an incorrect result, but when an accumulator changed, whether a branch was entered, and what exception an operation raised. Runtime failure in an unfinished candidate can itself provide feedback rather than requiring all failing traces to be discarded.

The dynamic signal combines a fixed instruction with candidate code, test input, and the corresponding trace, inserts this material at the fixed prompt position, and retains the current model response and code prefix after it. Traces consequently remain tied to the candidate and input that produced them instead of becoming context-free exception strings. Appendix B illustrates this structure with two candidates and one test; that prompt illustration is not evidence that the experimental candidate count was changed to two.

This stage does not guide generation with hidden tests or train a program-correctness classifier. Raw runtime information is a soft condition for the language model to interpret, not an instruction declaring a best candidate certified by an external scorer. Final solution selection nevertheless uses available-test pass/fail results: avoiding explicit correctness labels for token guidance does not mean that the system never checks whether tests pass.

3. Dual-distribution line-by-line guidance: use trace-induced probability changes to steer the committed continuation

For the same committed code prefix, the model computes one distribution from a prompt without the added traces and another from a prompt containing them. The authors call the former unconditional, but it still contains the task, examples or instructions, reasoning prefix, and generated code. It is unconditional only with respect to execution feedback, not unprompted generation. The difference between these two queries supplies classifier-free guidance (CFG).

The central combination in Equation (13) is:

\[ \log M_{\text{CFG}}(w_i\mid p_{\text{sol}},p_{\text{dyn}}) =\log M(w_i\mid p_{\text{sol}}) +\gamma\left[\log M(w_i\mid p_{\text{dyn}})-\log M(w_i\mid p_{\text{sol}})\right]. \]

Here, \(p_{\text{sol}}\) is the current prompt without added traces, \(p_{\text{dyn}}\) is the trace-augmented prompt, and \(\gamma\) controls amplification. The equation omits the normalization term needed to turn combined scores into normalized probabilities. That term is token-independent at the current step and does not affect token ranking or argmax. The unnormalized log scores should not be described as strictly normalized probabilities, and CFG should not be reduced to merely appending text to a prompt.

At \(\gamma=0\), decoding returns to the branch without execution feedback. At \(\gamma=1\), it is the trace-conditioned distribution and does not remove feedback. At \(\gamma>1\), it further amplifies the differences by which traces make a token relatively more or less likely. Thus, the “w/o CFG” ablation actually fixes decoding to ordinary conditional generation: it tests contrastive distribution guidance, not complete removal of runtime information.

The text describes committed continuation using “sample,” but Equation (14) and Appendix Algorithm 2 explicitly choose the next token with \(\arg\max\). This note follows the equation for the committed token decision while retaining the wording discrepancy as an implementation boundary; candidate sampling and committed code generation should not be conflated. Within a line, the code prefix continues to grow and both distributions change accordingly. What remains fixed is the trace material for that line, not every token probability.

Feedback is established before the first code token. Subsequently, candidates are regenerated, executed, and traces refreshed only when the previous token contains a newline. Execution observations therefore update per line, while their influence reaches every token within that line. This avoids changing observations repeatedly inside an unfinished line, but does not execute the program after every token. On a code-end marker or length limit, the instance returns its candidate program for the controller to check.

4. Independent parallel solution selection: explore configuration diversity rather than exchanging messages

Several independent instances can receive the same task, each choosing its own temperature, preview length, guidance strength, and prompt template. They do not exchange messages or merge traces, and there is no planner instructing a programmer role. The selection node in the diagram represents the controller collecting completed results; each instance has already been independently running the preceding three stages, rather than parallelism beginning only at the end.

The experiments list temperatures \(t\in\{0.7,0.75,0.85,0.95,1.2,1.5\}\), guidance strengths \(\gamma\in\{0,0.5,1,3\}\), and two prompt templates: the standard DeepSeek-Coder 3-shot template and an alternative encouraging more atomic, step-by-step implementations. Combined with the four preview lengths, the full Cartesian product contains 192 possible configurations. The paper does not adequately specify the number actually enabled per task, concurrency limits, or scheduling budget. Therefore, 192 is not a confirmed number of continuously active parallel instances or a cost estimate.

The controller returns the first program passing every available test and terminates the remaining instances; if none finds such a program, it returns failure. This success is limited to tests available during inference and does not imply prior knowledge of hidden-test correctness. Multi-configuration search can cover different local choices, but weak tests may cause early acceptance of an example-specific program. Final hidden evaluation measures that gap rather than serving as an online solution-selection oracle.

A Worked Example¶

Consider an independently constructed illustrative task asking for the sum of all positive numbers in a list, with available input [-2, 3, 4] and expected output 7. This is neither a code example copied from the paper nor a new experimental result. The committed prefix has initialized an accumulator and started traversing the list; the system uses three short continuations to observe different state-update paths.

One candidate also adds negative numbers, with its trace showing the accumulator changing from 0 to -2 and eventually returning 5. Another filters nonpositive values and eventually returns 7. A third may return prematurely inside the loop, showing that only part of the input was processed. If a candidate ends at an unfinished branch header, AST extraction can add pass to make the partial program parseable, but this does not establish that the summation task has been implemented.

The system injects each candidate, the test, and its runtime trace together without first declaring the second candidate a winner. Dual-distribution line-by-line guidance then adjusts preferences for the next committed tokens, which need not reproduce any candidate verbatim. Traces are reused within the line, and previews are rebuilt from the updated committed prefix after a newline. Finally, the controller knows only whether the program passes the available example; correctness on other inputs still requires independent tests.

Loss & Training¶

This is an inference-time algorithm, with no new loss function, back-propagation, fine-tuning data, or reinforcement learning reward. The smaller DeepSeek-Coder-1.3B runs on RTX 2080 Ti / RTX 3090 GPUs, while DeepSeek-V3-0324 uses a Fireworks AI cloud endpoint. The interface must expose token log probabilities for dual-distribution combination. The paper does not adequately detail vocabulary coverage and combination when log-prob output is limited, so reproducibility through an arbitrary black-box chat interface should not be assumed.

Key Experimental Results¶

Main Results¶

Accuracy is the percentage of tasks passing all tests required by evaluation, not pass@1 from a single generation without search. MBPP contains 500 tasks, HumanEval 164, and DS-1000 1000; CodeContests uses the ExecEval framework. Final evaluation on HumanEval, HumanEval-ET, MBPP-ET, CodeContests, and DS-1000 uses hidden tests inaccessible during inference. MBPP's provided tests must be distinguished from these hidden evaluations.

The table selects only DeepSeek-V3-0324 results from Tables 1–4. Results from other models, external papers, and custom CodeContests test sets are not mixed into a strict same-setting comparison. Cross-method gains also do not establish gains under equal token or GPU budgets.

Dataset	Baseline LLM accuracy (%)	EG-CFG accuracy (%)	Same-model reference methods and accuracy (%)
MBPP	82.8	96.6	MapCoder 87.2; MGDebugger 86.8; LPW 84.0
MBPP-ET	64.8	73.0	MapCoder 69.6; MGDebugger 64.8; LPW 65.2
HumanEval	82.92	99.4	MapCoder 96.95; MGDebugger 87.20; LPW 95.12
HumanEval-ET	79.20	89.02	MapCoder 81.70; MGDebugger 81.09; LPW 84.74
CodeContests	41.81	60.6	MapCoder 50.30; CodeSim 52.72
DS-1000	38.9	69.9	Table 4 lists no other same-model method

Sources: Table 1 covers the MBPP variants, Table 2 the HumanEval variants, Table 3 CodeContests, and Table 4 DS-1000. On HumanEval, the paper matches its listed LDB result of 99.4%, rather than holding an exclusive lead. Other SOTA statements likewise refer only to comparisons included in this version, not to a continuously updated leaderboard.

Ablation Study¶

The following table comes from Table 6 and fixes the model to DeepSeek-Coder-1.3B, not the larger model above. The minimal trace retains only the final event of the full trace; it does not disable execution.

Config	MBPP accuracy (%)	MBPP-ET accuracy (%)	Note
EG-CFG	83.2	59.8	Full method
no beam search	58.2	43.6	Replace multi-candidate preview with a single continuation
w/o CFG	75.2	48.2	Fix guidance strength to 1; execution-feedback conditioning remains
minimal trace	76.4	51.2	Provide only the final runtime event
Baseline LLM	49.4	42.6	Standard-prompt baseline

Removing multi-candidate preview lowers MBPP from 83.2% to 58.2% and MBPP-ET from 59.8% to 43.6%, the largest decline among the three ablations. Ordinary trace-conditioned decoding remains above baseline but below the full method, and detailed intermediate traces outperform retaining only the final event. This supports a joint contribution from preview diversity, trace detail, and contrastive distribution guidance, without establishing independent additive effects from individual removals.

Key Findings¶

Per-task MBPP runtimes in Table 5 show that parallel design does not mean consistently faster execution. All figures below are author-reported means and standard deviations in seconds, not measurements from a local reproduction.

Model	EG-CFG (s)	MGDebugger (s)	MapCoder (s)	LPW (s)
DeepSeek-Coder-1.3B	123.23 ± 344.91	495.16 ± 411.07	121.9 ± 213.89	197.71 ± 128.07
DeepSeek-V3-0324	271.37 ± 271.45	842.24 ± 705.19	283.84 ± 197.54	87.51 ± 210.84

With the smaller model, EG-CFG's mean is close to but slightly higher than MapCoder's; with the larger model, it is substantially slower than LPW. Large standard deviations preclude interpreting mean runtimes as a stable speed advantage on every task.
On large-model MBPP, the authors increase MGDebugger and MapCoder retries from 5 to 200: the former rises from 86.8% to 93.6%, and the latter from 87.2% to 88.8%, both remaining below EG-CFG's 96.6%. This is a stress comparison with 40 times as many retries, not proof of strictly matched actual token consumption, GPU time, or total cost.
Extended-test accuracy remains substantially below original-test accuracy. Execution feedback helps produce more reliable solutions but does not close the gap between finite tests and generalization.
The authors report debugging effort when adapting baselines to DeepSeek, especially unusually low LPW scores on the smaller model. QualityFlow had no public code and was not reproduced with the same model. These implementation and provenance boundaries should accompany comparisons.

Highlights & Insights¶

Observation precedes commitment. Tentative candidates can expose the runtime consequences of local choices before code is committed. Unlike repairing a complete answer, the placement of feedback becomes part of the inference-time algorithm.
Syntactic validity and semantic correctness are handled separately. AST extraction allows truncated programs to produce traces, while traces show runtime problems to the model. This distinction prevents an executable fragment from being mistaken for a correct solution.
Feedback content and feedback weight are different. Injected traces provide information, whereas CFG changes their influence relative to the existing code prior. Ordinary conditional decoding still improves over baseline, so the full effect should not be attributed to CFG alone.
Parallelism comes from independent exploration, not collaborative dialogue. This reduces sequential dependencies between instances but makes resource scheduling and stopping criteria important engineering concerns. Accuracy alone cannot characterize search cost.

Limitations & Future Work¶

Executable tests with meaningful coverage are required. Missing tests, too few inputs, or difficult environment setup limit the value of trace feedback, and passing examples does not prove hidden-input correctness. Coverage-driven test selection is a possible direction, not an improvement validated here.
Parallelism does not eliminate extra computation. Candidate generation, repeated execution, and dual-distribution queries all add work. Reporting actual instances, tokens, executions, concurrency limits, and costs per task would support efficiency analysis; 192 possible configurations cannot substitute for a measured budget.
Short previews do not replace global planning. The authors acknowledge that the bottom-up method does not exploit task decomposition; multi-file state and long-horizon dependencies are also outside the demonstrated experiments. Combining it with top-down planning is a reasonable direction, not a validated result.
Execution requires isolation, not merely AST parsing. Generated programs should run in a controlled sandbox with timeouts, resource limits, and restrictions on external access. Accuracy experiments and syntactic extraction provide no code-security guarantee, and the paper does not establish a complete isolation mechanism.
Implementation descriptions retain gaps. Beam-search versus sampling notation, sample versus argmax for committed generation, prefix concatenation before candidate execution, and actual parallel scheduling require implementation verification. Cached Appendices A/B mainly supply figure captions, from which the unseen full prompts cannot be reconstructed.

vs Self-Debugging / Reflexion: These methods chiefly repair complete answers using post-execution error feedback; EG-CFG previews local consequences while forming the final code. Its cost is more frequent execution and dual-distribution computation, not elimination of test dependence.
vs MGDebugger / LPW: MGDebugger uses hierarchical debugging and model-simulated traces, while LPW organizes planning and debugging into phases. This paper primarily changes next-line decoding with real traces from short candidates. Combining planning with local execution guidance is worth exploring, but the paper does not already include a high-level planner.
vs MapCoder / AgentCoder: These approaches emphasize interaction between roles or stages; the instances here differ only in configuration and operate independently. Multi-instance search should be distinguished from collaborative multi-agent systems when attributing contributions.
vs conventional text CFG: Conventional conditioning signals are relatively static; here, traces refresh with the program prefix at line boundaries. Dynamic behavioral evidence from an executable environment may transfer to database queries or simulation, but the paper provides no experiments in those domains.

Rating¶

Novelty: 4/5 — Dynamic short-horizon execution traces and dual-distribution decoding are combined with a distinctive feedback schedule.
Experimental Thoroughness: 4/5 — Six evaluations, two model scales, and mechanism ablations provide broad evidence, but actual compute budgets and some implementation details are insufficient.
Writing Quality: 3/5 — The main loop is clear, but beam search, sampling, argmax, and program concatenation are not described rigorously enough.
Value: 4/5 — Useful for code generation with tests and isolated execution, but not yet a general solution for real-world software engineering.