Enhancing Complex Method Implementation via Fine-Grained AST Execution Feedback
Add an abstract syntax tree level runtime tracing and debugging feedback loop to map execution failures directly to mathematical equations in the experiment memory.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
During the Engineer Agent's 'Experiment Tree Search' (Section 3.4 and Section 5.2), specifically at Stage 3 (Proposed Method execution), the execution success rate remains very low (21.57%) even after experiment strategy evolution.
“The remaining low success rate also highlights clear headroom for improvement, suggesting that richer interaction histories and more fine-grained execution feedback may further strengthen EvoScientist’s performance on complex proposed-method implementations.”
Limitation
The current experimentation memory struggles to adequately guide the engineer agent through complex, novel algorithm implementations, indicating that the summarized execution feedback is too coarse-grained to resolve intricate logical code bugs.
Proposed Solution
Implement an AST-level (Abstract Syntax Tree) execution tracing and debugging feedback loop. Instead of just summarizing high-level logs, utilize a specialized LLM to perform line-by-line runtime analysis and map specific stack trace failures directly to the mathematical equations in the proposed method, establishing a fine-grained reasoning trajectory in the experimentation memory.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Improving the reliability of executing entirely novel proposed methods is the most critical bottleneck in fully autonomous scientific discovery. Raising this success rate would drastically increase the throughput of viable AI-generated research.
How to Get Started
How to test it: Hypothesis: Replacing coarse log summaries in experimentation memory with AST-level runtime tracing mapped to proposal equations will substantially boost Stage 3 (proposed method) execution success rate. Baseline: EvoScientist with standard Experiment Strategy Evolution (ESE) across all 30 queries (4 workers, N_E^3=12 attempts), measuring Stage 3 and overall execution success rates; stop if Stage 3 success rate improves by less than 5 percentage points over the 21.57% baseline or overall execution drops.
- Analyze failed code execution traces from current engineer agent experiments to categorize common error types.
- Develop an AST parsing script to extract variable dependencies and method signatures from proposed code implementations.
- Prompt a specialized language model to map runtime stack traces back to specific lines of the mathematical specification.
- Test the debugging loop on a benchmark of intentionally flawed scientific code repositories.
Code linked to this paper on Hugging Face: github.com/EvoScientist/EvoScientist
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates and CoTran explore using execution feedback for code optimization or translation. However, they do not specifically use AST-level line-by-line runtime analysis to map stack trace failures directly to proposed mathematical equations within a reasoning memory.
- Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates · 2024 · 27 citations
- CoTran: An LLM-Based Code Translator Using Reinforcement Learning with Feedback from Compiler and Symbolic Execution · 2024 · 90 citations
We searched OpenAlex for “Add an abstract syntax tree level runtime tracing and debugging feedback loop to map execution failures directly to mathematical equations in the experiment memory.” and “Enhancing Complex Method Implementation via Fine-Grained AST Execution Feedback” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific DiscoveryYougang Lyu, Xi Zhang, Xinhao Yi +9 more · 2026
This paper introduces EvoScientist, a multi-agent framework that automates the end-to-end scientific discovery process by leveraging self-evolving AI agents. Its primary contribution is the integration of persistent memory modules that allow the system to continuously learn from past idea validations and code execution outcomes, significantly improving the novelty, feasibility, and success rates of its generated research.
Related Ideas
- Grounding AI Ideation: Multi-Objective RLCF for Feasible Scientific DiscoveryAI Can Learn Scientific Taste⛔Very hard
- Verifiable Tree-Search with Partial State FeedbackDeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints🟠Medium
- Recursive Associative-Memory Graph Traversal for Multi-Step Scientific ReasoningWeight-Space Programming Doesn't Need Transformers🟡Fairly easy
- End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic WorkflowsSage S2.1 Flash Technical Report🟠Medium
- Graph-Based Branching Protocols for Wet Lab AutomationProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows⛔Very hard