Frontier LLMs can directly inspect and edit raw neural network weights in Python without gradients or training loops:
Asked to improve Qwen3.5-0.8B at math with no gradient descent or training, GPT-6 Astra uses a Python kernel over the model's 0.8 billion parameters and a budget of 50 forward passes to trace why 85 of the base model's first 96 practice answers run into repetitive thinking loops at the 32,768-token limit. Astra isolates the looping behavior to a single direction written by the MLPs of layers 15 and 19 and subtracts that direction from 64 output columns in each—natively editing 0.016% of the weights (131,072 parameters). That edit cuts token-limit looping on MATH-500 from 93.0% to 33.1%, raising MATH-500 accuracy from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and out-of-distribution instruction following (IFEval) from 21.0% to 62.3%.
2026-11-12: Splicing code layers into training via phantom gradients
Ordinary weight gradients average across a batch, hiding token-level failures. Computing ∂L/∂h_ℓ with respect to every token's hidden state at every layer—the phantom gradient—maps where representation demand persists across checkpoints. By centering phantom vectors per target token, clustering failing tokens with spherical k-means, and splicing Python code layers into the residual stream wherever they beat a linear baseline, a coding agent inside training permanently lowers validation loss on a 12-layer, 286M-parameter nanochat model, raises CLRS-Text algorithmic reasoning accuracy on unseen intermediate lengths from 55% to 84%, and lets DeepSeek-V4.1-Flash build a linear probe across blocks 24–39 that rejects 80% of its own wrong answers (at a 6.7% false-rejection rate) to lift AIME and HMMT 2026 accuracy from 64% to 71%.