Information TheoryWirelessMachine Learning

The Energy Cost of Certainty

Shannon's waterfall leaves decoding off the energy bill. Charging for it turns the waterfall into a waterslide. Later work uses the entropy of floating-point numbers to bound the thermodynamic cost of fitting a line to data.

A curve that priced decoding at zero

What was already understood

Shannon's channel-coding theorem permits arbitrarily reliable communication above a threshold transmit power. Its familiar waterfall plot leaves the energy spent encoding and decoding off the bill.

Codes that operate close to the waterfall — sparse-graph codes decoded by message passing — typically demand more decoding work than codes that stay farther away. That work consumes energy, even though it does not appear in the classical channel-capacity calculation.

The omission matters increasingly at short distances. Deep-space communication helped shape coding theory; there, transmission dominates the energy bill. Across a room, a circuit board, or a chip, decoding a bit can cost more than sending it. The work grew out of Pulkit Grover's interest in decoding: starting in 2008, we put the decoder and transmitter on the same power bill and watched what happened to the waterfall.

Two plots. Left: Shannon waterfall curves where error probability drops sharply at a threshold power, compared with slowly decaying uncoded transmission. Right: waterslide curves where the lower bound on total power slides gradually and the optimal transmit power sits to the right of the Shannon waterfall.

Left: Shannon's waterfall for a rate-1/31/3 code, beside uncoded transmission (dashed red). Right: charging for decoding in our idealized message-passing model turns the lower bound on total power (blue) into a gradual waterslide. An achievable upper bound runs parallel to it. The transmit power that minimizes the total (dashed red) lies above the Shannon threshold.

Certainty has to travel through wires

Our decoder passes messages on a graph of bounded degree: each computational node talks only to its neighbors, one round at a time. The encoder is free, computation within a node is unrestricted, and the decoder may be as parallel as it likes. But every node-round costs a fixed energy, and a decision cannot use evidence that has not yet reached it.

Operating closer to capacity requires each decision to gather more evidence. A bounded-degree graph can enlarge that neighborhood only a limited amount per round. More rounds mean more decoding energy. This is why a little extra transmit power can lower the total bill: it can save enough work inside the receiver.

How neighborhood growth gives the decoding bound

We adapted sphere-packing arguments from our work on fixed-delay streaming. For power, we needed bounds that also held for small neighborhoods and average rather than maximal error. The task was to turn the resulting complexity bound into energy and power.

What this work added

The growing complexity of near-capacity sparse-graph codes was already understood in several forms. Sason and Urbanke bounded parity-check density; Khandekar and McEliece conjectured iteration lower bounds over the erasure channel; Sason and Wiechman established Ω(1/gap)\Omega(1/\mathrm{gap}) behavior for important code families. We charged for each node-iteration so that decoding power could be added to transmit power and the two minimized jointly.

The waterslide

For the same communication rate and error target, compare two choices: minimize transmit power alone, or minimize the sum of transmit and decoding power. The calculation below charges a fixed energy per node-iteration on a bounded-degree graph. Increasing transmit power can reduce the required rounds enough to lower the sum.

Charging for decoding moves the minimum-total-power operating point away from the Shannon transmit threshold.

Uses the BSC rare-error asymptotic expression and power model in Towards a Communication-Theoretic Understanding of System-Level Power Consumption, Eqs. (5), (10), and (12) (2011). It illustrates the asymptotic tradeoff, not the finite-neighborhood converse curves.

At decoded throughput RdataR_{\mathrm{data}} and code rate RchR_{\mathrm{ch}} in bits per channel use, the power bill is schematically

Ptotal  ≳  PT+Enode RdataRch l.P_{\text{total}} \;\gtrsim\; P_T + E_{\text{node}}\,\frac{R_{\mathrm{data}}}{R_{\mathrm{ch}}}\,l.

The second term multiplies the energy of a node-step by the number of coded symbols processed per second and the required rounds ll. In this asymptotic calculation, positive energy per node-step moves the minimum-total-power operating point above the Shannon transmit threshold. Closing the gap to capacity sends the required neighborhood, and the decoding bill, to infinity. Backing away costs transmit power but saves decoding power. Tracing the minimized total as the error target falls gives the waterslide.

Within this idealized model, we also exhibited constructions with the same scaling order as the lower bound. They exploit the unrestricted computation allowed inside each node: they establish tight scaling, not a practical decoder. How far the optimum moves depends on the ratio of decoding cost to transmit cost — γ\gamma in the figure. When transmission dominates, the correction is small. At short range it can change the preferred operating point substantially.

Green coding is a co-design problem

Code rate, graph geometry, wire lengths, circuit technology, and the error target affect the same power bill. The interconnect bounds show why additional reliability margin can require longer wires and more circuit activity. Its cost depends on the implementation model, as does the advice about transmit power:

What goes on the power bill?

Three models, three design lessons. Compare them at a fixed communication rate as the error target becomes more demanding.

Transmission only

Shannon waterfall

What is charged?
Radiated power. Encoding, decoding, and communication inside the circuits are free.
What changes in the design?
Approach the Shannon transmit threshold. At a fixed rate, arbitrarily small error is possible with bounded transmit power.

Transmission + node iterations

The interactive above

What is charged?
Radiated power plus a fixed energy per decoder node per round. The graph has bounded degree; wire lengths and encoding are free.
What changes in the design?
Transmit above the Shannon threshold to save decoding work. As bit error vanishes, the asymptotically optimizing transmit power stays bounded, while the decoding bill diverges.

Transmission + physical wires

2012 encoding and decoding bounds

What is charged?
Radiated power plus the cost of switching wires at both encoder and decoder. Wire area × clock cycles sets the energy per block; throughput converts it to power. Energy inside nodes is free.
What changes in the design?
As block error vanishes, minimizing the joint lower bound calls for increasing transmit power. Keeping it bounded instead gives a faster-growing lower bound on total power.

The node model studies average bit error; the 2012 wire model studies block error. The scaling comparisons belong to their respective models, not to a common numerical curve.

Node model: Towards a Communication-Theoretic Understanding of System-Level Power Consumption, Section III-C and Eq. (14) (2011). Wire model: the 2012 encoding-and-decoding power bounds, Section III and Corollary 2 (especially Eq. (6)).

Scope and caveats

The 2012 model processes whole blocks in batches on a two-dimensional VLSI circuit; it excludes streaming and pipelined implementations. Minimizing its joint lower bound identifies a tradeoff, but does not supply a circuit that attains the bound.

How the implementation models developed, 2008–2012

What is a number, physically?

A related accounting question arose nearly two decades later in work with Sultan Daniels, Samuel D'Ambrosia, and Michael DeWeese: how much information does a learning computation discard? Landauer's principle connects irreversible information loss to an energy floor. Applying it requires counting information in the numbers a machine actually stores.

For floating-point numbers, scale has an unexpected effect. A zero-mean Gaussian occupies approximately p+2.46p+2.46 bits of entropy at pp bits of precision, almost regardless of its variance, provided little probability mass reaches the exponent limits. Widening the distribution also widens the quantization bins, so it adds little entropy. Underflow and overflow eventually break this pattern. The floating-point entropy paper (ITW 2026) derives this approximation and bounds its error.

Large changes in scale can leave floating-point entropy almost unchanged, until underflow or overflow merges distinctions.

An interactive rendering of the simplified normalized floating-point model and entropy approximation in The Entropy of Floating-Point Numbers (2026).

The entropy formula and what scale invariance means

The thermodynamic cost of fitting a line

Our regression paper, published in the IEEE Journal on Selected Areas in Information Theory, applies that entropy calculation to fitting a line through the origin, exactly or by stochastic gradient descent. For an irreversible cyclic device that keeps only the fitted slope, it bounds the energy associated with the difference between input and output entropy.

In the Gaussian setting analyzed in the paper, the bound grows mainly with the number of data points and their precision. Within the approximation’s range, higher-signal-to-noise data lowers the bound because less information is discarded on the way to the estimate.

Four stages of the paper’s thermodynamic accounting: load n independent quantized sample pairs, retain one quantized fitted slope, take the input entropy minus the retained output entropy, and multiply that difference by k B T ln 2 to obtain the all-at-once Landauer energy floor. Only the slope is kept; costs of intermediary registers are omitted.

Across repeated datasets, the quantized inputs carry H(DQ)=nH(XQ,YQ)H(D_Q)=nH(X_Q,Y_Q) bits of entropy and the saved slope carries H(W^Q)H(\widehat W_Q). Their difference sets the paper's Landauer cost for a cyclic device retaining only the slope.

Original explanatory diagram based on Eqs. (6)–(8) and the cyclic-device protocol in The Thermodynamic Costs of Simple Linear Regression (2026).

What this energy accounting assumes

The paper ends with an illustrative economic model: accuracy attracts inference demand, and energy is priced using this thermodynamic lower bound. Revenue saturates as prediction error approaches its floor, while the bound keeps growing with dataset size. The resulting optimum shifts to fewer samples as the price of energy rises.

Two plots of profit against training-set size for exact regression and stochastic gradient descent, each with four curves for different energy prices; stars mark the maxima, which move to smaller dataset sizes as the energy price increases.

Profit versus training-set size for exact regression (left) and SGD (right), at four energy prices. Stars mark the optima.

The illustrative economic objective
Scope and caveats

The bound is an all-at-once thermodynamic floor for an idealized cyclic device that keeps only the slope. It omits intermediate registers and time-resolved control, so it is not an estimate of what a training chip consumes; the economic model is illustrative, not advice about dataset sizes on current hardware.

What this line changed

The decoder work put transmission and implementation on a shared power bill. That changed the optimization: minimizing total power need not mean operating as close to capacity as possible. The network analysis added another consideration — interference — and the wiring bounds showed how conclusions depend on circuit assumptions.

The later thermodynamic work asks a related question at a different level: how much information is lost between loading data and retaining a fitted parameter? Floating-point entropy makes it possible to calculate how sample count and precision enter that loss, and hence the energy floor for the stated protocol. In both parts of the story, specifying the physical accounting changes what can be said about a computation's cost.

The papers

The thirteen papers below run from the 2008 waterslide through the implementation bounds to the 2026 floating-point entropy and regression papers, including the broadcasting, interconnect, throughput, and interference-aware work.

Related Papers
Sultan Daniels, Samuel H. D'Ambrosia, Michael R. DeWeese, Anant Sahai
2026IEEE Information Theory Workshop (ITW), accepted
Samuel H. D'Ambrosia, Sultan M. Daniels, Michael R. DeWeese, Anant Sahai
2026IEEE Journal on Selected Areas in Information Theory