An expansion that was supposed to be slowing

Every mass in the universe attracts every other one, so an expanding universe must be slowing down. That statement has no loopholes in Newtonian gravity and only one in general relativity, and the one had been dismissed by its author as a blunder. By the early 1990s the deceleration was the obvious thing to measure: the plan was to find distant supernovae, plot them against redshift, and read the deceleration off the curvature of the relation.

Two independent teams did it, using different software, different supernova samples and different assumptions about dust, and in 1998 they published the same answer. The distant supernovae were about a quarter of a magnitude fainter than a universe with no deceleration at all would predict — which is not a smaller deceleration than expected, and not zero deceleration. It is the other sign.

The measurement is a residual, and that matters #

The essential fact about this result, and the one most often lost when it is retold, is that nobody looked at a graph and saw a curve bending the wrong way. The signal is a fifth of a magnitude on an axis that spans four, riding on a relation whose overall slope carries no information at all.

Because it is a residual, the result stands or falls on the relative calibration of nearby and distant supernovae, not on any absolute quantity. The Hubble constant does not enter: it sets the zero point of the vertical axis and is fitted away. The absolute luminosity of a type Ia supernova does not enter either, for the same reason — the standard candle here is not a candle of known brightness but a candle of reproducible brightness, which is a much weaker requirement and the only one the measurement needs. What has to be true is that a supernova at $z = 0.5$ is the same kind of object as one at $z = 0.05$, and that the corrections applied to both are applied consistently.

The quantity on the vertical axis is a luminosity distance, which in an expanding universe is the comoving distance multiplied by $(1+z)$ — one factor for each photon’s lost energy and one for the reduced rate of arrival. Both of those are geometry rather than model, so they are common to all three curves and cancel out of the residual too. What does not cancel is the comoving distance itself, which is the integral of $c\,dz/H(z)$ and is therefore the only place the expansion history can enter. The entire experiment is a measurement of that integral.

That is why the result was believed as fast as it was. Two teams with different systematics is worth a great deal when the quantity is a difference, because most of the ways to get it wrong are ways of shifting everything together.

What a type Ia supernova has to be #

A standard candle is a hypothesis about intrinsic brightness, and the hypothesis here has a physical basis that is unusually specific.

A white dwarf is held up by degeneracy pressure, and there is a mass above which degeneracy pressure cannot hold anything up. A white dwarf accreting from a companion approaches that mass, and near it carbon fusion ignites in the core under conditions where the star cannot expand to regulate itself — degenerate matter does not respond to a temperature rise with a pressure rise — so the burning runs away and unbinds the star entirely. The mass at ignition is nearly the same every time, so the amount of nickel-56 produced is nearly the same, so the peak luminosity is nearly the same.

“Nearly” is not good enough. Raw type Ia peak magnitudes scatter by about 0.4 magnitudes, which is 20 per cent in distance and swamps the signal by a factor of two. The result was only possible because of a correction found in 1993: brighter supernovae decline more slowly, and the relation between peak brightness and light-curve width is tight enough that using it reduces the scatter to about 0.15. There is a second correction, for colour: redder supernovae are fainter, partly from dust in the host galaxy and partly intrinsically, and the two cannot be separated cleanly. The colour correction was the biggest worry in 1998 — a population of dust that reddened distant supernovae more than nearby ones would produce the same faintness. It was addressed by measuring the reddening rather than assuming it, and by noting that grey dust, which dims without reddening, would have to be finely tuned to mimic the observed redshift dependence.

A third correction, found much later, has no accepted physical explanation at all: after the width and colour corrections, supernovae in massive host galaxies are still about 0.06 magnitudes brighter than those in small ones. Nobody knows why, and the leading guesses — a difference in progenitor age, in metallicity, in dust grain size — all imply that the correction should itself change with redshift, because the population of host galaxies does. It is now fitted as a step function and its size is one of the larger entries in the systematic budget. A standardisation that works empirically and is not understood is a standing liability, and this one is the reason nobody claims the supernova constraint is limited by statistics.

What has to be in the universe to do this #

An expansion that accelerates requires something with negative pressure, and the amount of negativity required is set by the Friedmann acceleration equation:

$$\frac{\ddot a}{a} = -\frac{4\pi G}{3}\left(\rho + \frac{3p}{c^2}\right).$$

Ordinary matter has $p \approx 0$ and radiation has $p = \rho c^2/3$; both make $\ddot a$ negative. To get acceleration something needs $p < -\rho c^2/3$, and a cosmological constant has $p = -\rho c^2$ exactly, which is as negative as anything in the standard framework goes.

That last point is the strongest internal check the supernova data have. If the faintness were caused by dust, or by supernovae evolving with redshift, it would grow monotonically with distance. Acceleration does not: the universe was decelerating before $z \approx 0.6$, so the residual should turn over and come back. Supernovae beyond $z = 1$ do show the turnover, and finding it was the point of the highest-redshift searches.

What is measured, and what the word hides #

Nothing here measures a cosmological constant. What is measured is a set of apparent magnitudes and redshifts. What is fitted is a two-parameter family of expansion histories. What comes out is a preferred region in that plane, and the region excludes $\Omega_\Lambda = 0$ at high confidence.

“Dark energy” is a name for whatever occupies that region, and the name should be read as an admission rather than a description. The observations constrain one number about it — the equation-of-state parameter $w = p/\rho c^2$, measured as $-1.03 \pm 0.03$ — and are consistent with a cosmological constant and with a great many other things.

Its density, if it is constant, is about $6\times10^{-27}$ kilograms per cubic metre — roughly the mass of four hydrogen atoms in a cubic metre. Written as an energy scale it is $2.3$ millielectronvolts, and the disagreement between that and any natural scale in quantum field theory is the largest discrepancy between prediction and observation in physics. That is a problem for field theory rather than for cosmology, but it is the reason the word “energy” in the name is doing so much unearned work.

What the pictures cannot show #

The residual figure subtracts a model, so its vertical axis has no meaning without that model. The empty universe is chosen as the reference because it is the natural dividing line between deceleration and acceleration, not because anybody believes it. A different reference gives a different-looking plot with identical content.

No figure here distinguishes a cosmological constant from a slowly rolling field. Everything drawn assumes $w = -1$ exactly. A $w$ of $-0.9$ produces curves that differ by a few hundredths of a magnitude over the plotted range, which is inside the scatter of the points, and separating them is the entire purpose of the current generation of surveys.

And the supernovae are not drawn from where they were found. The samples that produced this result are heterogeneous — different telescopes, different filters, different selection — and combining them requires cross-calibrating photometric systems to about a per cent. That cross-calibration, rather than any astrophysics, is the dominant systematic in modern supernova cosmology, and no plot of the kind drawn here can show it.

How it happened #

The two teams were not collaborators. The Supernova Cosmology Project, led from Berkeley, had been searching since 1988 and had developed the batch-discovery technique that made the whole thing possible: image a field, image it again three weeks later, subtract, and schedule follow-up on whatever appeared — which turns supernova discovery from luck into a scheduled activity. The High-z Supernova Search Team formed in 1994 out of the group that had established the light-curve-width correction.

Both had expected to measure $\Omega_{\rm m}$ from the deceleration. Both spent months looking for the error. The published papers are notable for how much of them is devoted to alternatives — dust, evolution, gravitational lensing, selection — and for the fact that neither team led with the interpretation.

What made it stick was that the answer was already needed elsewhere. The ages of the oldest stars had been embarrassing a matter-dominated universe for a decade — globular clusters at twelve and a half billion years inside a universe the model said was nine and a half. Weighing galaxy clusters gave a matter density around 0.3 rather than 1, by three independent routes. And the microwave background was about to show that the total density was 1. A missing 0.7 with negative pressure resolved all three at once, and within four years the cluster and microwave-background results had arrived independently. The supernovae are the famous measurement, and they are the least precise of the three that now constrain it.

It is worth being explicit about how the three combine, because it is the clearest case in the subject of constraints that are useless alone and decisive together. The supernovae measure the difference $\Omega_\Lambda - \Omega_{\rm m}$ well and the sum badly, because a luminosity distance depends mostly on the acceleration. The microwave background measures the sum $\Omega_{\rm m} + \Omega_\Lambda$ very well and the difference badly, because the angular scale of the acoustic peaks depends mostly on the total. Plotted in the same plane the two allowed regions are long thin ellipses crossing at a large angle, and their intersection is small in both directions. Neither one on its own excludes a matter-dominated universe at any great confidence; together they exclude it overwhelmingly.

What has happened to the measurement since #

The 1998 result rested on a few dozen distant supernovae. Compiled samples now hold well over a thousand, and the interesting thing about the intervening twenty-five years is what changed and what did not.

What did not change is the answer. The accelerating solution has survived every enlargement of the sample, and the constraint on the equation-of-state parameter has tightened around $-1$ without moving away from it.

What changed is which uncertainty dominates. The early result was limited by how many supernovae had been found; it has been limited by systematics for most of the time since. The largest of those was the heterogeneity described above — samples assembled from many telescopes, cross-calibrated against each other with a per cent-level uncertainty that no amount of additional data reduces.

The response was to change how the observing is done. A rolling search images the same fields repeatedly with one telescope and one set of filters, discovering the supernovae and following them up in the same data stream. Everything is then measured on one photometric system, so the calibration error that used to sit between the nearby and distant samples is confined to the join between the survey and whatever nearby sample it is anchored on.

That reorganisation is the reason the field is now arguing about the host-galaxy step and the low-redshift anchor rather than about dust and evolution. The systematic that dominates a measurement is not a fixed property of the method, and identifying it is most of what a mature experiment does.

The two things that would still break it #

Two possibilities remain live, and both are about the sample rather than about the cosmology.

The first is the nearby end. The residual is a comparison between distant supernovae and nearby ones, and the nearby ones sit in a region whose peculiar motions are not negligible: galaxies within a few hundred million light years are falling towards local overdensities at hundreds of kilometres a second, which is a substantial fraction of their recession speed. Correcting for that requires a model of the local flow, built from a survey of the mass distribution, and the correction is not small compared with the effect being measured. A systematic error in the local flow model shifts the nearby anchor and therefore the whole residual.

The second is evolution of the progenitors. A type Ia supernova is a white dwarf that has been pushed over a mass limit, and there is more than one way to do that — accretion from an ordinary companion, or the merger of two white dwarfs. The two channels have different delay times between star formation and explosion, so their relative contribution changes with redshift. If the two produce even slightly different peak luminosities after standardisation, the mix changing with redshift produces exactly the kind of drift the measurement cannot distinguish from cosmology.

Neither is a reason to doubt the result, because the cluster and microwave-background constraints do not share either weakness and agree. They are reasons why the supernovae are no longer the most precise leg of the argument, and why the effort has moved to measurements that do not involve a standard candle at all.

The generalisation #

The methodological lesson is not about cosmology and is worth stating on its own: when the signal is a small departure from a well-understood baseline, the right observable is the departure, and the experiment should be designed to make the baseline cancel.

That is why the two teams could tolerate not knowing the absolute luminosity of a supernova, not knowing $H_0$, and not knowing the absolute calibration of their photometry: every one of those affects nearby and distant supernovae alike and drops out of a difference. The same structure appears in a transit depth, which is a ratio of radii and needs no absolute flux, in the equation of time, which is the difference between two clocks neither of which needs to be right, and in the residual that revealed Mercury’s precession — an anomaly of 43 arcseconds per century found by subtracting a 5,600-arcsecond prediction.

The corresponding hazard is the one that makes such measurements hard: anything that differs between the two ends of the comparison does not cancel, and there is no internal way to find it. Dust, evolution and calibration are dangerous here for exactly the reason that everything else is safe.

It is worth naming what those alternatives are, since they are the reason the supernovae have stopped being the centre of the subject. Baryon acoustic oscillations use a length scale imprinted on the matter distribution before recombination as a ruler rather than a candle, and a ruler needs no standardisation and no local anchor. Weak lensing measures the growth of structure rather than the expansion history, which responds to dark energy through a different combination of the same parameters. Neither is more precise than the supernovae today; both have systematic budgets that are unrelated, which is the property that matters when the question is whether a twenty-five-year-old result is real.

One more set of equations of state brackets the measurement more tightly than the essay’s own figure.

Where the ladder goes next #

The supernovae give one constraint in a plane, and a single constraint in a plane does not fix two numbers. What closes it is the microwave background, which constrains the total density almost orthogonally — and that takes the next three essays.

Later rungs on this anchor: the equation of state $w$ and the searches for a time dependence in it; the coincidence problem, and whether it is a problem; quintessence and the field-theoretic alternatives to a constant; the effect of dark energy on the growth of structure, which is a completely separate observable from the expansion history; and the cosmological constant problem proper — the sixty-order-of-magnitude gap between the measured value and the natural one.