I did not expect to open a frontier model paper in 2026 and recognize the equations. Not the architecture diagrams or the scaling curves, the actual equations:
A hidden state that evolves, an output read out from it. I last met that structure as an engineering undergraduate, writing a B.Sc. thesis on stochastic forecasting of solar photovoltaic output with Kalman filters, exponential smoothing, and the expectation-maximization algorithm. Fifteen years later the same linear state space model, wearing new clothes and renamed a selective state space model, is the most credible challenger to the transformer's monopoly on sequence modeling. That is not a coincidence, and I want to explain why, partly because the explanation is useful and partly because statisticians so rarely get to say told you so to computer scientists.
The usual caveat, because this topic attracts more confident opinions than it deserves: I am not an AI researcher and will not pretend to follow every proof. My claim to the subject is narrower; I speak the language these models were borrowed from. Solar output is a textbook stochastic process, a deterministic diurnal backbone with cloud noise on top, and my thesis was an exercise in tracking its hidden state with a Kalman filter and learning the parameters with EM. The research group I did that work in kept pulling on the thread after I left, and it became a paper in Technometrics (Carpio, Juan, and López, 2014): multivariate exponential smoothing with a dynamic factor structure, estimated with EM, applied to hourly electricity prices in the Omel, Powernext, and Nord Pool markets. Keep that paper in mind. One detail from it comes back when we get to Mamba.
Sixty years of the same equation
The state space model has been quietly running the world since 1960, when Rudolf Kalman published the filter that carries his name. Assume the system you care about has a hidden state, a compact set of numbers summarizing everything relevant about its past. The state evolves by a transition rule, you observe it only through noisy measurements, and the filter gives you the statistically optimal way to update your belief every time a measurement arrives. A version of it navigated Apollo; variants sit in every GPS receiver and autopilot today. This is the most battle-tested idea in applied mathematics that most of the AI industry had simply never met.
Two details from that world matter here. First, exponential smoothing, the humble method every statistics student learns in week one, is secretly a state space model: simple exponential smoothing is the steady-state Kalman filter of a model with one hidden level, a correspondence Hyndman and colleagues worked out in full. Its single parameter α decides how fast you forget the past, and it is the entire personality of the model. The Technometrics paper makes the point with unusual clarity: its factor decomposition reduces a high-dimensional process to independent smoothing processes, each defined by one parameter running from zero, white noise, the past forgotten instantly, to one, a random walk, the past never forgotten at all. One number between zero and one, deciding what the model remembers. Hold that thought.
Second, when you do not know the system's parameters, you learn them from data with expectation-maximization, which Shumway and Stoffer brought to time series in 1982. That was my toolkit, and the intuition underneath it all is the one to keep: you never store the history. You maintain a compressed belief about the world and update it one observation at a time.
What transformers pay for perfect memory
Transformers made the opposite bet. Attention compresses nothing; every token can look back at every previous token, which is why transformers are so good at exact retrieval. The bill is structural: compute that grows with the square of sequence length, and a key-value cache, the internal representation of every token seen so far, parked in expensive GPU memory for as long as the context lives. The transformer's state is the entire uncompressed context. Lossless memory, in other words, and lossless memory does not scale gracefully, which is why long context is priced like a luxury good and a small army of engineers now works full-time on compressing, evicting, and paging KV caches. The industry treats these costs as facts of nature. They are not. They are the consequences of refusing to compress, a strategy whose most enthusiastic supporters sell GPUs.
From Kalman gain to selection
The rediscovery came in stages. HiPPO in 2020 worked out how to initialize the state matrix A so the state provably retains long memory of the input, and S4 in 2021 turned the classical model into a trainable sequence layer: learn A, B, and C by gradient descent, and exploit the fact that a linear time-invariant system computes as one big convolution. S4 crushed the Long Range Arena benchmark, but as a language model it lagged, for a reason any statistician would guess on sight: time-invariant dynamics treat every token identically. It is exponential smoothing with a fixed α, forgetting a filler word and a crucial name at exactly the same rate.
Mamba, in late 2023, made the dynamics depend on the input. The step size Δ and the matrices B and C are computed from the current token, so the model decides, per token, how much to write into the state and how much of the old state to keep. The authors call this selectivity. I call it the smoothing parameter from that Technometrics paper, the number that was estimated once and then frozen, becoming a function the network computes fresh for every token. The deeper pattern is the Kalman gain, the mechanism that decides at every step how much to trust the new measurement against the existing belief; the filter derives that tradeoff from assumed noise covariances, while Mamba learns it end to end, no Gaussian assumptions asked. Selection is the Kalman gain's wilder, learned descendant. Even the training story rhymes: my thesis learned A, B, and C by EM, these models learn them by gradient descent, jointly with everything else.
Selectivity broke the convolution trick, so Mamba's authors built a hardware-aware parallel scan to keep training fast. The payoff is the part that matters economically: inference becomes a fixed-size state updated once per token. Constant memory, constant per-token cost, no cache growing with the context, several times the throughput of a comparable transformer on long sequences. Mamba-2 then closed the conceptual loop in 2024 with a title that does my arguing for me, "Transformers are SSMs": both are parameterizations of the same underlying object, and the real question is how much state you keep and how you decide what goes into it.
The honest tradeoff
A fixed-size state is a compressed state, and compression has casualties. The best-documented one is exact recall: ask a pure SSM to copy a long string verbatim, or to fetch one phone number it saw two thousand tokens ago, and it loses to a transformer, which kept everything while the SSM had to guess at write time what would matter later. That is not an implementation detail. It is the price of the architecture's central bet, and pretending otherwise is how this topic gets oversold.
The field's answer has been refreshingly unideological: hybrids. Sprinkle a few attention layers into a mostly-SSM stack, and the attention supplies precise retrieval while the SSM layers do the bulk of the work at near-linear cost. NVIDIA's empirical study found that roughly one attention layer per ten state space layers recovers most of the gap. The recipe is in production: AI21's Jamba first, then NVIDIA's Nemotron-H and IBM's Granite 4.0, with linear-attention cousins inside Qwen3-Next and MiniMax-01. The serious labs have stopped asking whether to mix recurrence and attention. They argue about the ratio.
Streaming, which is where I take this personally
There is a second lens on this architecture that I have not seen used much, from the part of my career spent around Kafka and Flink. A stateful stream processor holds bounded, checkpointable state, updates it once per arriving event, and never re-reads the historical log. That is, point for point, SSM inference. A transformer behaves like a batch job re-run over the entire accumulated log every time one new event arrives, a design no streaming engineer would sign off on sober.
I spent years watching what becomes possible when batch turns into streaming: not the same work done faster, but different applications, because the system lives inside the event flow instead of summarizing it afterwards. Expect the same shape of change here. Constant-memory, constant-latency inference is what robotics control loops, on-device assistants, and always-on monitoring agents actually need. HormuzWatch ran a Flink pipeline because vessel tracking needs a model of the world updated per event; an architecture whose inference step is a per-event state update is the native fit for that entire class of problem, and that class is much bigger than chat.
Why I think this is a high-potential path
The economics: when the millionth token costs the same as the hundredth, long context stops being a premium feature and always-on inference stops being absurd, and bending the cost curve has historically mattered more than bending the quality curve. The theory: state space models inherit sixty years of control theory and signal processing, mature tools for stability, memory, and frequency response, more usable mathematics than any frontier architecture has arrived with in a long time. And the part I would actually bet on: state as explicit, bounded memory. A compact state that summarizes everything a model has absorbed is something you can persist, checkpoint, and carry across sessions, which looks far more like a path to continual learning than an ever-larger context window does.
To be clear, attention earned its dominance, and the likely future is a settled hybrid default rather than a coup. But the direction is hard to miss. The industry that spent a decade scaling lossless memory is now busy learning how to forget intelligently, and forgetting intelligently is the oldest problem in statistics. We even wrote it up.
The long way around
There is a private satisfaction in all of this that I will admit to. The undergraduate fitting Kalman filters to cloudy-day solar data, coaxing EM into convergence and wondering whether any of it would matter outside a thesis defense, was apparently preparing for this decade the whole time. Good mathematical ideas in statistics rarely die. They wait, sometimes sixty years, for the hardware and the scale that let them matter, and then they come back wearing a new name and considerably better marketing. I would not bet against the rest of the time series literature getting its turn.
Further reading
The foundations: Kalman (1960) on linear filtering and prediction; NASA's history of the filter in Apollo; Dempster, Laird, and Rubin (1977) on EM and Shumway and Stoffer (1982) on applying it to state space models; Hyndman et al. on exponential smoothing as state space, with a gentler version in Forecasting: Principles and Practice; and Carpio, Juan, and López (2014) in Technometrics, from the research group where I wrote my thesis.
The modern line: HiPPO, S4, Mamba, and Mamba-2 for the architecture; Jelassi et al. on the copying gap; Waleffe et al. on hybrid ratios; Jamba, Nemotron-H, Granite 4.0, Qwen3-Next, and MiniMax-01 for production.