Can an AI Model Run Without Keeping Every Weight in RAM?

Three different sizes behind one model

The stored model contains its full set of parameters. In this design, NAND flash is the persistent storage that holds those weights. DRAM, or dynamic random-access memory, is the faster working memory into which selected weights are brought for execution. Flash offers storage capacity, but its access and transfer characteristics differ from DRAM's. [1] [3]

Active parameters are the parameters used in the selected computation. They are not automatically the complete set of weights occupying RAM: a system can retain useful weights between steps, and some components remain active throughout. Nor does the active count reveal the number of bytes used for each numerical value. [1] [3]

Full-runtime memory includes more than weights. A language model can also retain attention information about earlier tokens in a key-value cache, alongside intermediate calculation data and software buffers. A token is a unit of text processed by the model, not necessarily a whole word. These allocations are separate from the stored parameter count. [4]

Why more storage is not enough

Keeping a large model on flash solves a capacity problem: the full weights need not all occupy DRAM at once. It does not solve the transfer problem. If generation continually asks for different weights, the system must repeatedly read them from storage before it can use them. The older LLM in a flash paper explains why naive loading on every forward pass can be expensive. [3]

Both the amount transferred and the pattern of reads matter. Many small scattered reads can make poorer use of flash than larger contiguous transfers. Meanwhile, keeping previously loaded weights available consumes RAM but can avoid another read. Practical execution therefore needs a policy for what to load and what to retain, not just a larger storage device. [3]

Select a useful subset from the instruction

Apple's research page for Instruction-Following Pruning, marked June 2025, describes a learned predictor that reads the user instruction and chooses which model parameters to activate for that task. Its sparse mask is a selection pattern: it identifies parts to use rather than permanently deleting the same parts for every request. The predictor and language model are trained together. [2]

The research selects rows and columns in feed-forward matrices, which are arrays of weights used in a model's calculation layers. Fixing a selection for the task makes those weights reusable during generation. The shared idea with mixture-of-experts models is conditional computation: activate a subset instead of using every parameter. The flash-relevant distinction is how frequently that selection changes. [2]

This predictor is not a lookup table that knows every answer's exact needs. It learns a selection strategy, and the resulting model must still be evaluated for the intended tasks. The research's quality results do not establish that every chosen subset preserves the full model's abilities. [2]

Follow a request through Apple's flash-backed design

Apple's June 2026 description says AFM 3 Core Advanced stores a 20-billion-parameter model in NAND flash and activates 1–4 billion parameters at a time, depending on the request. These are Apple's architecture figures, not a measured RAM requirement. The company describes the model as optimized for its most capable Apple silicon systems, not as compatible with every phone. [1]

During initial prompt processing, a lightweight dense block makes a routing decision. Routing means choosing the model components that will participate. Here, experts are learned groups of weights associated mainly with feed-forward blocks; the word does not promise that each group has a neatly labelled subject specialty. [1]

The system combines selected routed experts with shared weights in DRAM. Many shared experts are always active. Input-dependent routed experts are brought into DRAM when needed, allowing the selected weights to be used for generation without fetching the entire stored model. [1]

The selection is periodically reconsidered while generation continues. So per-prompt routing does not mean one unchanging subset must serve the entire response. Apple's stated reason for reducing selection churn is that NAND-to-DRAM bandwidth is too slow for token-by-token weight swapping in this design. It is not a rule that all sparse models have identical loading requirements. [1]

Earlier flash research shows why reuse matters

LLM in a flash, first submitted in December 2023, describes a different execution method. In the January 2024 version considered here, the system predicts which feed-forward weights are needed at token steps and uses windowing to keep weights associated with recent tokens in DRAM. It loads newly needed weights rather than discarding and rereading everything at each step. [3]

Its row-column bundling method places paired feed-forward weight data together so it can be read in larger contiguous chunks. Windowing reduces repeated transfers; bundling changes how necessary transfers are organized. A larger retained window can reduce new reads but consumes more working memory. [3]

These techniques illustrate the same systems lesson—weight reuse and read layout matter—but they are not interchangeable descriptions of Apple's later architecture. The older token-step predictor, the 2025 task-conditioned pruning research, and the 2026 periodically reselected expert design have different selection details. The older paper does not establish that the current model uses its exact windowing or bundling methods. [1] [2] [3]

What selective loading does not remove

The full model still occupies storage, selected weights still require DRAM, and routing plus transfers still take work. A low active-parameter count cannot by itself establish response speed, energy use or answer quality. It also cannot be turned into a proportional whole-runtime RAM saving: the retained weight set, numerical representation and non-weight allocations all matter. [1] [2] [3] [4]

A meaningful memory claim should say whether it measures weights alone or the entire runtime, and on which hardware. A speed claim should distinguish the initial wait for output from the pace of generation. Both should be paired with task-quality results and an explanation of whether weights were already loaded or had to be read. These questions reveal the transfer cost hidden behind a simple parameter label. [1] [3] [4]

An on-device model design also does not establish that every feature or request is processed locally. Apple's foundation-model family includes server-based models as well. Selective loading addresses weight residency; it is not a complete description of application data flow or device deployment. [1]

Common questions about weights and RAM

Can I put any model on flash and get this benefit?

Not simply by changing where the file is saved. The described approaches require a model and runtime that can select, load and reuse the appropriate weights. A conventional execution path may still need its full weight set resident, or spend substantial time on transfers. Available storage alone does not establish practical flash-backed inference. [1] [3]

Is a smaller active subset just a smaller model?

It is a smaller selected computation, not necessarily a smaller stored model. Different instructions can select different weights from the larger retained set. This differs from keeping only one permanently reduced model for every request. The broader weight set does not, however, guarantee that a selected subset matches every capability of full-model execution. [1] [2]