The Energy Cost of Certainty
Shannon's waterfall leaves decoding off the energy bill. Charging for it turns the waterfall into a waterslide. Later work uses the entropy of floating-point numbers to bound the thermodynamic cost of fitting a line to data.
A curve that priced decoding at zero
Shannon's channel-coding theorem permits arbitrarily reliable communication above a threshold transmit power. Its familiar waterfall plot leaves the energy spent encoding and decoding off the bill.
Codes that operate close to the waterfall — sparse-graph codes decoded by message passing — typically demand more decoding work than codes that stay farther away. That work consumes energy, even though it does not appear in the classical channel-capacity calculation.
The omission matters increasingly at short distances. Deep-space communication helped shape coding theory; there, transmission dominates the energy bill. Across a room, a circuit board, or a chip, decoding a bit can cost more than sending it. The work grew out of Pulkit Grover's interest in decoding: starting in 2008, we put the decoder and transmitter on the same power bill and watched what happened to the waterfall.

Left: Shannon's waterfall for a rate- code, beside uncoded transmission (dashed red). Right: charging for decoding in our idealized message-passing model turns the lower bound on total power (blue) into a gradual waterslide. An achievable upper bound runs parallel to it. The transmit power that minimizes the total (dashed red) lies above the Shannon threshold.
Figures 1 and 8 of the long version of The price of certainty: iterative decoding from a total power perspective (2008).
Certainty has to travel through wires
Our decoder passes messages on a graph of bounded degree: each computational node talks only to its neighbors, one round at a time. The encoder is free, computation within a node is unrestricted, and the decoder may be as parallel as it likes. But every node-round costs a fixed energy, and a decision cannot use evidence that has not yet reached it.
Operating closer to capacity requires each decision to gather more evidence. A bounded-degree graph can enlarge that neighborhood only a limited amount per round. More rounds mean more decoding energy. This is why a little extra transmit power can lower the total bill: it can save enough work inside the receiver.
How neighborhood growth gives the decoding bound
We adapted sphere-packing arguments from our work on fixed-delay streaming. For power, we needed bounds that also held for small neighborhoods and average rather than maximal error. The task was to turn the resulting complexity bound into energy and power.
The growing complexity of near-capacity sparse-graph codes was already understood in several forms. Sason and Urbanke bounded parity-check density; Khandekar and McEliece conjectured iteration lower bounds over the erasure channel; Sason and Wiechman established behavior for important code families. We charged for each node-iteration so that decoding power could be added to transmit power and the two minimized jointly.
The waterslide
For the same communication rate and error target, compare two choices: minimize transmit power alone, or minimize the sum of transmit and decoding power. The calculation below charges a fixed energy per node-iteration on a bounded-degree graph. Increasing transmit power can reduce the required rounds enough to lower the sum.
Charging for decoding moves the minimum-total-power operating point away from the Shannon transmit threshold.
Uses the BSC rare-error asymptotic expression and power model in Towards a Communication-Theoretic Understanding of System-Level Power Consumption, Eqs. (5), (10), and (12) (2011). It illustrates the asymptotic tradeoff, not the finite-neighborhood converse curves.
At decoded throughput and code rate in bits per channel use, the power bill is schematically
The second term multiplies the energy of a node-step by the number of coded symbols processed per second and the required rounds . In this asymptotic calculation, positive energy per node-step moves the minimum-total-power operating point above the Shannon transmit threshold. Closing the gap to capacity sends the required neighborhood, and the decoding bill, to infinity. Backing away costs transmit power but saves decoding power. Tracing the minimized total as the error target falls gives the waterslide.
Within this idealized model, we also exhibited constructions with the same scaling order as the lower bound. They exploit the unrestricted computation allowed inside each node: they establish tight scaling, not a practical decoder. How far the optimum moves depends on the ratio of decoding cost to transmit cost — in the figure. When transmission dominates, the correction is small. At short range it can change the preferred operating point substantially.
When every transmitted watt becomes somebody else's noise
The point-to-point waterslide leaves out interference: power radiated by one link becomes noise at another. Our 2011 JSAC paper with Kristen Woyach therefore asked how many transmitter-receiver pairs a square meter could support, with fixed rate and reliability targets and both transmit and decoding power counted. We placed the pairs on a triangular grid.
Coding can tolerate a lower signal-to-interference ratio, letting co-channel transmitters move closer. Turning every transmitter up cannot do this: in the interference limit, desired signal and interference rise together. For uncoded signaling, rarer errors require greater separation, and link density tends to zero. Capacity-approaching codes can retain a nonzero density by moving the reliability burden into the decoder.
At finite total power, however, the receiver's work competes with transmission. In the paper's numerical comparison, uncoded transmission can outperform the measured decoder's extrapolated performance at small budgets, while coding supports higher densities. The free-decoder curve is an upper benchmark; it leaves open the cost of implementing such a code.
In the interference limit, stricter reliability pushes uncoded links apart while coded links can retain a nonzero density. At finite power, the decoder’s own bill can limit that advantage.
Based on the triangular-grid analysis in Towards a Communication-Theoretic Understanding of System-Level Power Consumption (2011), using a finite lattice sum. The finite-power curves compare uncoded signaling, the paper's extrapolated decoder fit, and a free-decoder benchmark; they omit the paper's additional implementation-constrained density bound.
Green coding is a co-design problem
Code rate, graph geometry, wire lengths, circuit technology, and the error target affect the same power bill. The interconnect bounds show why additional reliability margin can require longer wires and more circuit activity. Its cost depends on the implementation model, as does the advice about transmit power:
What goes on the power bill?
Three models, three design lessons. Compare them at a fixed communication rate as the error target becomes more demanding.
| Model | What is charged? | What changes in the design? |
|---|---|---|
| Transmission onlyShannon waterfall | Radiated power. Encoding, decoding, and communication inside the circuits are free. | Approach the Shannon transmit threshold. At a fixed rate, arbitrarily small error is possible with bounded transmit power. |
| Transmission + node iterationsThe interactive above | Radiated power plus a fixed energy per decoder node per round. The graph has bounded degree; wire lengths and encoding are free. | Transmit above the Shannon threshold to save decoding work. As bit error vanishes, the asymptotically optimizing transmit power stays bounded, while the decoding bill diverges. |
| Transmission + physical wires2012 encoding and decoding bounds | Radiated power plus the cost of switching wires at both encoder and decoder. Wire area × clock cycles sets the energy per block; throughput converts it to power. Energy inside nodes is free. | As block error vanishes, minimizing the joint lower bound calls for increasing transmit power. Keeping it bounded instead gives a faster-growing lower bound on total power. |
Transmission only
Shannon waterfall
- What is charged?
- Radiated power. Encoding, decoding, and communication inside the circuits are free.
- What changes in the design?
- Approach the Shannon transmit threshold. At a fixed rate, arbitrarily small error is possible with bounded transmit power.
Transmission + node iterations
The interactive above
- What is charged?
- Radiated power plus a fixed energy per decoder node per round. The graph has bounded degree; wire lengths and encoding are free.
- What changes in the design?
- Transmit above the Shannon threshold to save decoding work. As bit error vanishes, the asymptotically optimizing transmit power stays bounded, while the decoding bill diverges.
Transmission + physical wires
2012 encoding and decoding bounds
- What is charged?
- Radiated power plus the cost of switching wires at both encoder and decoder. Wire area × clock cycles sets the energy per block; throughput converts it to power. Energy inside nodes is free.
- What changes in the design?
- As block error vanishes, minimizing the joint lower bound calls for increasing transmit power. Keeping it bounded instead gives a faster-growing lower bound on total power.
The node model studies average bit error; the 2012 wire model studies block error. The scaling comparisons belong to their respective models, not to a common numerical curve.
Node model: Towards a Communication-Theoretic Understanding of System-Level Power Consumption, Section III-C and Eq. (14) (2011). Wire model: the 2012 encoding-and-decoding power bounds, Section III and Corollary 2 (especially Eq. (6)).
The 2012 model processes whole blocks in batches on a two-dimensional VLSI circuit; it excludes streaming and pipelined implementations. Minimizing its joint lower bound identifies a tradeoff, but does not supply a circuit that attains the bound.
How the implementation models developed, 2008–2012
What is a number, physically?
A related accounting question arose nearly two decades later in work with Sultan Daniels, Samuel D'Ambrosia, and Michael DeWeese: how much information does a learning computation discard? Landauer's principle connects irreversible information loss to an energy floor. Applying it requires counting information in the numbers a machine actually stores.
For floating-point numbers, scale has an unexpected effect. A zero-mean Gaussian occupies approximately bits of entropy at bits of precision, almost regardless of its variance, provided little probability mass reaches the exponent limits. Widening the distribution also widens the quantization bins, so it adds little entropy. Underflow and overflow eventually break this pattern. The floating-point entropy paper (ITW 2026) derives this approximation and bounds its error.
Large changes in scale can leave floating-point entropy almost unchanged, until underflow or overflow merges distinctions.
An interactive rendering of the simplified normalized floating-point model and entropy approximation in The Entropy of Floating-Point Numbers (2026).
The entropy formula and what scale invariance means
The thermodynamic cost of fitting a line
Our regression paper, published in the IEEE Journal on Selected Areas in Information Theory, applies that entropy calculation to fitting a line through the origin, exactly or by stochastic gradient descent. For an irreversible cyclic device that keeps only the fitted slope, it bounds the energy associated with the difference between input and output entropy.
In the Gaussian setting analyzed in the paper, the bound grows mainly with the number of data points and their precision. Within the approximation’s range, higher-signal-to-noise data lowers the bound because less information is discarded on the way to the estimate.
Across repeated datasets, the quantized inputs carry bits of entropy and the saved slope carries . Their difference sets the paper's Landauer cost for a cyclic device retaining only the slope.
Original explanatory diagram based on Eqs. (6)–(8) and the cyclic-device protocol in The Thermodynamic Costs of Simple Linear Regression (2026).
What this energy accounting assumes
The paper ends with an illustrative economic model: accuracy attracts inference demand, and energy is priced using this thermodynamic lower bound. Revenue saturates as prediction error approaches its floor, while the bound keeps growing with dataset size. The resulting optimum shifts to fewer samples as the price of energy rises.

Profit versus training-set size for exact regression (left) and SGD (right), at four energy prices. Stars mark the optima.
Figure 5(a–b) of The Thermodynamic Costs of Simple Linear Regression (2026).
The illustrative economic objective
The bound is an all-at-once thermodynamic floor for an idealized cyclic device that keeps only the slope. It omits intermediate registers and time-resolved control, so it is not an estimate of what a training chip consumes; the economic model is illustrative, not advice about dataset sizes on current hardware.
What this line changed
The decoder work put transmission and implementation on a shared power bill. That changed the optimization: minimizing total power need not mean operating as close to capacity as possible. The network analysis added another consideration — interference — and the wiring bounds showed how conclusions depend on circuit assumptions.
The later thermodynamic work asks a related question at a different level: how much information is lost between loading data and retaining a fitted parameter? Floating-point entropy makes it possible to calculate how sample count and precision enter that loss, and hence the energy floor for the stated protocol. In both parts of the story, specifying the physical accounting changes what can be said about a computation's cost.
The papers
The thirteen papers below run from the 2008 waterslide through the implementation bounds to the 2026 floating-point entropy and regression papers, including the broadcasting, interconnect, throughput, and interference-aware work.
Information theory meets circuit design: Why capacity-approaching codes require more chip area and power
Information-theoretic tradeoffs of throughput and chip power consumption for decoding error-correcting codes
Read Next
Interpolation Can Be Harmless
Fitting noisy data exactly can be harmless, but the same beneficial regularizing effect that extra features have also lets them bleed away the signal. Treating overparameterization as undersampling makes the relevant effects, contamination and survival, calculable.
Read StorySpectrum Jails: Engineering Trust After the Fact
Regulators cannot certify every line of a cognitive radio's code. Making identity, evidence, and sanctions technical moves trust to what can be enforced after the fact, and opens questions about commitment and learning.
Read Story