Differentiable Relaxation of Written Associative Memories for Few-Shot Fine-Tuning
A differentiable relaxation of discrete hash tables and concept weights that allows gradient descent to fine-tune human-readable written weights on low-data regimes.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 7.5 and Section G, Claude Opus 5.5 writes network weights for AG News text classification achieving 84% accuracy with zero training data, but the discrete nature of the highest-performing associative-memory fact tables prevents subsequent fine-tuning via backpropagation.
“Used to initialize gradient descent on 100 labeled AG News articles, a written network reaches up to 84.7% accuracy, whereas training from random initialization on the same labels reaches at most 63.6%.”
Limitation
The top-performing written models rely on non-differentiable hash addressing and hard top-k selections, preventing gradient descent from refining hand-written concept matrices when labeled data becomes available.
Proposed Solution
Replace the hard modulo hash index mapping and hard top-k fact filtering with a continuous soft-hash relaxation using kernel density estimators or differentiable learned hash projections (such as soft product-key memory formulations). Allow gradient descent with low learning rates to adjust the non-zero concept weights and mixing parameter µ on downstream task splits while preserving row sparsity.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Allowing gradient descent to fine-tune WSP models would turn hand-authored symbolic knowledge into a flexible initialization framework for extreme few-shot learning.
How to Get Started
How to test it: Test whether fine-tuning relaxed table weights on 100 labeled AG News and ARC examples outperforms both the frozen written table and random initialization trained with AdamW, measured by test classification accuracy. Halt if gradient fine-tuning destroys the sparsity structure and causes catastrophic forgetting below the zero-shot written baseline.
- Formulate a continuous surrogate for the pair-table hash lookup h(a, b) = (ap + b) mod H using a softmax-weighted slot allocation.
- Implement end-to-end backpropagation through the concept weight vector w and table A on 100 labeled AG News examples.
- Compare post-fine-tuning accuracy and parameter drift against the 84.7% reported in Table 25.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works such as Instant neural graphics primitives with a multiresolution hash encoding and VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks discuss hash encodings and parameter-efficient fine-tuning respectively, but none of the listed works cover the differentiable relaxation of written associative memories via soft-hash projections for few-shot fine-tuning.
- Instant neural graphics primitives with a multiresolution hash encoding · 2022 · 6848 citations
- VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks · 2024 · 84 citations
We searched OpenAlex and arXiv for “differentiable memory networks hash lookup fine-tuning” and “Differentiable Relaxation of Written Associative Memories for Few-Shot Fine-Tuning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Weight-Space Programming Doesn't Need TransformersHyunjong Ok, Jaeho Lee · 2026
This paper investigates weight-space programming (WSP), where large language models directly write the tensor parameters of a target model without gradient-based training. When given architectural freedom on multiple-choice science benchmarks, frontier LLMs abandon Transformers in favor of sparse associative-memory tables that achieve higher accuracy with up to 1,800 times fewer parameters. The authors also design an associative-memory model that uses direct hash addressing to evaluate queries up to 17 times faster on GPUs than equivalent-sized Transformers.
Related Ideas
- Content-Adaptive Semantic Band GatingThe Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding🟡Fairly easy
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Retrieval-Augmented Decoding for Low-Parameter Hallucination ReductionPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🟡Fairly easy
- Accelerating 3D-Guided RL Fine-Tuning with Trajectory DistillationGeometry-Guided Reinforcement Learning for Multi-view Consistent 3D Scene Editing🔴Hard
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy