LLM-Driven Neural Architecture Search: Automate the Boring Parts of Model Design

ArXiv: 2603.12091 — Published 2026-03-13

---

The Problem

Neural Architecture Search (NAS) — finding the optimal network design for a task — has historically required either massive compute budgets or teams of ML engineers who know what they're doing.

Neither scales.

What the Paper Does

This paper proposes a closed-loop LLM pipeline for NAS. The LLM proposes architectures, evaluates them, receives feedback, stores what it learns in memory, and iterates.

Critical detail: it has feedback memory. Most LLM pipelines are stateless — each call starts from scratch. Here, the system accumulates knowledge across iterations, learning what designs work and why.

Results:

• Competitive architectures found with significantly fewer compute cycles than traditional NAS
• The feedback memory loop accelerates convergence — the LLM stops proposing architectures it already knows fail
• Generalizes across tasks without task-specific tuning

Why This Matters for Builders

The real insight here isn't NAS — it's the feedback memory pattern.

Most agentic AI systems today are amnesiac. They complete a task, the context window closes, and they start fresh next time. This paper demonstrates a simple architectural fix: give the agent a memory of outcomes, not just a memory of instructions.

Apply this pattern anywhere you're running iterative optimization loops with an LLM — prompt engineering, hyperparameter tuning, content A/B testing.

Builder Takeaway

If your LLM agent is doing iterative work (trying, failing, retrying), it should be writing down what failed and why. A simple key-value store of {attempt: outcome} can cut your iteration cycles dramatically.

---

Source: Xiaojie Gu, Dmitry Ignatov, Radu Timofte — ArXiv cs.AI, March 2026