Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Cross-Repository Micro-Skill Generalization via Project Manifest Embeddings

Implement a hierarchical skill bank combining global ecosystem skills with repository-specific overrides by embedding project setup manifests from public GitHub repositories.

Time

1 to 2 weeks for manifest embedding retrieval, hierarchical skill-bank implementation, and multi-seed benchmark evaluation.

Compute level

Single GPU or less

Resources

API credits for frontier coding models across 277 SWE-Bench tasks (6 repositories) and 200 security tasks (CredData/ReposVul) with 3 seeds, Docker container infrastructure for isolated task execution, and a single GPU or CPU server to run a dense code embedding model over repository manifests.

What the paper used

Hardware and dollar spend are not stated in the paper; experiments run containerized commercial CLI agents across 5 agent-model settings (Claude Code 2.1.170 with Sonnet 5 and Opus 4.8, Copilot CLI 1.0.84-7 with Sonnet 5, Codex 0.153.1 with GPT-5.6 Terra and GPT-5.6 Sol) on 60 SWE-Bench profiling tasks and 579 main evaluation tasks (277 SWE-Bench, 102 Web-Bench, 100 CredData, 100 ReposVul) with 3 seeds per setting.

Prerequisites

Vector retrieval systems, AST/manifest parsing, and prompt engineering.

Scope

Modify existing code to add a global vector index and manifest extraction module.

The Gap

In Section 4.3 and Appendix C, micro-skill injection was completely disabled for the security auditing benchmarks (CredData and ReposVul) because tasks did not share a recurring repository. Similarly, in SWE-Bench (Section 3.1), micro-skill state Si is strictly isolated per repository, preventing any transfer of operational conventions across codebases.

“TokenEconomizer begins each repository without prior traces, so its benefit may take several tasks to emerge.”
From the paper, Section 4.2, page 8. Matched against the PDF text.

Limitation

Micro-skills cannot transfer across projects. High-utility operational facts (such as framework-standard test commands, virtual environment activations, or linter behaviors) must be relearned from scratch in every new repository, preventing any cost savings during the first 4-9 tasks of a stream.

Proposed Solution

Implement a hierarchical skill bank combining global ecosystem-level skills with repo-level overrides. Represent project environments by embedding setup manifests (e.g., pyproject.toml, package.json, setup.cfg, and Dockerfiles) using a dense code embedding model. Before task 1 of an unseen codebase, retrieve top-k candidate micro-skills from structurally similar repositories and evaluate their scope detectors in cold-start dry runs or initial environment probing steps.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Enabling transfer learning across codebases eliminates the warm-up tax observed in Figure 5b, where TokenEconomizer is initially more expensive than default execution for the first 5 tasks, thereby broadening applicability to single-task or multi-repo development pipelines.

How to Get Started

How to test it: Hypothesis: retrieving candidate micro-skills via embedded project manifests (pyproject.toml, package.json, setup.cfg) and validating their scope in cold-start runs reduces early-stream token costs without degrading accuracy compared to cold-start TokenEconomizer and default execution. Measure cumulative cache-aware cost over tasks 1-5 and full-stream mean cost alongside Pass@1 on SWE-Bench (and F1 on CredData/ReposVul); stop if early-stream or overall cost savings fail to beat standard TokenEconomizer or if Pass@1 drops by more than 1.5 percentage points.

  1. Scrape setup manifests including pyproject.toml, package.json, and Dockerfiles from a diverse set of public GitHub repositories.
  2. Embed all collected manifests using a standard dense code embedding model such as CodeBERT or OpenAI text-embedding-3-small.
  3. Construct a hierarchical skill indexing structure mapping ecosystem-level skills to repository clusters via vector similarity.
  4. Evaluate cold-start micro-skill retrieval performance and scope detector accuracy on unseen benchmarks like SWE-Bench Lite.

Code linked to this paper on Hugging Face: github.com/microsoft/TokenEconomizer

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

None of the listed papers cover cross-repository micro-skill generalization using project manifest embeddings and hierarchical skill banks. Papers like Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries and RESOURCE2SKILL discuss agent skill extraction and lifecycles, but they do not address retrieving skills from structurally similar codebases via environment manifests.

We searched OpenAlex and arXiv for “Implement a hierarchical skill bank combining global ecosystem skills with repository-specific overrides by embedding project setup manifests from public GitHub repositories.” and “Cross-Repository Micro-Skill Generalization via Project Manifest Embeddings” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents

Aaron J. Li, Weijia Xu, Chandan Singh +8 more · 2026

This paper addresses the excessive token costs incurred by frontier coding agents when executing sequential software engineering tasks within shared environments. The authors introduce TokenEconomizer, an external, black-box controller that monitors execution traces to adaptively inject environment-specific micro-skills, prune unused tool definitions from agent contexts, and interrupt unproductive failure loops. Evaluated across Claude Code, Codex, and GitHub Copilot CLI on benchmarks including SWE-Bench, Web-Bench, and security auditing datasets, TokenEconomizer achieves an 18.4% to 30.9% token cost reduction while preserving or improving task success rates.

Related Ideas

Is this a good idea?

Share onXLinkedIn