AI / Technology
SkillOpt: controlled text editing for reusable agent skills
This paper studies how to improve a frozen agent by editing a separate skill document instead of changing model weights. The core idea is plain: keep the agent fixed, let another model propose bounded text edits to the skill, and accept an edit only when it improves held-out validation score.
Why agent skills are hard to optimize
An agent skill here is a reusable instruction document that tells a frozen model how to behave on a task. The problem is not just writing a better prompt once, but improving that prompt from feedback without making it unstable or overfit to a few failures.
The source notes say common alternatives are hand-crafted skills, one-shot LLM-generated skills, or loosely self-revised skills. Those approaches do not behave like a real optimizer for the skill itself, so they may fail to improve reliably from evidence.
Hand-built or one-shot skills
A human can write a skill directly, or a model can draft one in a single shot. That can work as a starting point, but the source says these methods do not reliably improve when they are pushed through feedback loops.
- Human-skill is a manual baseline.
- LLM-skill is a one-shot generated baseline.
Skill as external state, optimizer as editor
SkillOpt keeps the target agent frozen and treats the skill as a separate text file. A different optimizer model reads rollout evidence and proposes small edits to that file.
The edits are not free-form. They are bounded patch-style changes, and the paper adds a textual learning-rate budget to limit how much the skill can move in one update.
Validate, keep only better versions
The main control rule is simple: an edit is accepted only if it strictly improves a held-out validation score. That makes the skill update behave more like a cautious optimization loop than like open-ended rewriting.
The source also mentions a rejected-edit buffer and an epoch-wise slow or meta update. Those pieces preserve useful negative feedback and longer-horizon lessons instead of letting the skill forget them after one noisy minibatch.
- Rollouts create evidence.
- Evidence becomes bounded edits.
- Validation decides whether the skill version survives.
How far the evidence really carries the claim
SkillOpt treats an agent skill as external text state and updates it with a separate optimizer while keeping the target agent frozen.
Edits are bounded and only accepted if they strictly improve a held-out validation score.
Across 6 benchmarks, 7 target models, and 3 execution harnesses, SkillOpt is reported as best or tied-best on all 52 evaluated cells.
The provided source is partial and includes notes, not the full paper or full experimental appendix.
Product interpretation
A product team could use a separate offline optimizer to improve reusable agent instructions without retraining the model. That may be useful when the same procedural behavior needs to work across chat and tool-execution products.
The validation gate and bounded edits are the main practical controls to notice. They may make instruction changes easier to audit and roll back, but that is an interpretation, not an experimental claim.
Research appendixTerms, sources, and open questions
Paper evidence
SkillOpt treats an agent skill as external text state and updates it with a separate optimizer while keeping the target agent frozen.
Section-by-section notes, method; Paper section 3; Appendix A
Edits are bounded and only accepted if they strictly improve a held-out validation score.
Section-by-section notes, method; Paper section 3; Paper section 4; Appendix A
Across 6 benchmarks, 7 target models, and 3 execution harnesses, SkillOpt is reported as best or tied-best on all 52 evaluated cells.
arXiv metadata; arXiv HTML full text; Paper section 4
On GPT-5.5 direct chat, average no-skill accuracy improves from 58.8 to 82.3, a +23.5 gain.
Paper section 4
On GPT-5.5 Codex, SkillOpt is reported best on all five benchmarks, with +24.8 over no skill and +14.0 over EvoSkill.
Paper section 4
On GPT-5.5 Claude Code, SkillOpt is reported best on all five benchmarks, with +19.1 over no skill and +3.2 over EvoSkill.
Paper section 4
Ablations report that bounded learning, validation gating, rejected-edit buffering, and slow/meta updates matter more than unrestricted rewriting.
Paper section 5
Case studies report that the final skill artifact is compact, under 2,000 tokens, and often changes only 1 to 4 times before export.
Paper section 6
Limits and caveats
- The provided source is partial and includes notes, not the full paper or full experimental appendix.
- Claims like first, best, and broad transfer are author claims unless independently checked against the full paper.
- The exact benchmark definitions, harness implementations, and optimizer prompts are not fully specified in the source notes.
- Training cost can be high even when the deployed skill is compact.
- Some transfer results are reported only at a high level, so the limits of reuse are not fully pinned down.
How it compares
What remains uncertain
Exact benchmark names, all per-cell scores, and all protocol details are not fully specified in the provided notes.
The source context is abbreviated and does not reproduce the full tables or appendix text.
Check the full PDF tables and appendix for benchmark-by-benchmark values, harness definitions, and exact hyperparameters.
The full cost of skill optimization is unclear relative to the reported deployment footprint.
The notes mention compact exported skills but also mention high training-token costs on some tasks.
Inspect the paper's compute accounting, token usage tables, and any runtime or cost analysis.
Transferability limits across substantially different models, harnesses, and tasks are not fully bounded in the notes.
Transfer is reported, but the source does not provide exhaustive failure cases or negative transfer examples.
Review the transfer tables and any appendix discussion of where transfer did not work.
The independent contribution of each control, such as cosine decay, rejected-edit buffering, and slow/meta updates, is not isolated here.
The notes summarize ablations but do not provide a full factorial breakdown in this source context.
Look for ablation tables or appendix experiments that vary one control at a time.
Implementation details of the optimizer prompts and adapter interfaces are incomplete in the supplied source.
The notes say the details are partially truncated or unspecified.
Inspect the methods appendix, prompts, and code if available.
Glossary
- Frozen agent
- A target model whose weights are not changed during skill training.
- Skill document
- A reusable text artifact that tells the agent how to behave on a task.
- Validation gate
- A rule that accepts an edit only if it improves held-out score.
- Bounded edit
- A limited add, delete, or replace change instead of free-form rewriting.
- Textual learning rate
- A budget that limits how much the skill can change per update.
- Rejected-edit buffer
- A memory of failed edits so the optimizer can preserve useful negative feedback.
- best_skill.md
- The compact exported skill file described in the source notes.
Sources
Source grounding
Abstract / metadata: SkillOpt is presented as a controllable text-space optimizer for agent skills and is described as the first systematic one in the abstract. arXiv metadata
Method: The skill is treated as external state, edited by a separate optimizer using bounded add/delete/replace operations. arXiv HTML full text
Method: An edit is accepted only if it strictly improves a held-out validation score. arXiv HTML full text
Results: The paper reports best-or-tied results on 52 evaluated cells across 6 benchmarks, 7 target models, and 3 execution harnesses. Paper section 4
Results: GPT-5.5 direct-chat average accuracy improves from 58.8 to 82.3, a +23.5 gain. Paper section 4
Results: GPT-5.5 Codex and Claude Code both report best-on-all-five-benchmarks results with +24.8 and +19.1 gains over no skill. Paper section 4
Ablation / transfer: Bounded edit budgets, rejected-edit buffering, and slow/meta updates are highlighted as important design choices. Paper section 5
Case studies: The exported skill artifact is compact, often under 2,000 tokens, and the number of committed edits is small. Paper section 6
Three key questions about this paper
What problem does SkillOpt: controlled text editing for reusable agent skills address?
An agent skill here is a reusable instruction document that tells a frozen model how to behave on a task. The problem is not just writing a better prompt once, but improving that prompt from feedback without making it unstable or overfit to a few failures.
What evidence supports the main claim in SkillOpt: controlled text editing for reusable agent skills?
SkillOpt treats an agent skill as external text state and updates it with a separate optimizer while keeping the target agent frozen. Section-by-section notes, method; Paper section 3; Appendix A
What limitation should readers know about SkillOpt: controlled text editing for reusable agent skills?
The provided source is partial and includes notes, not the full paper or full experimental appendix.