A coincidence that is a factor of fourteen

Two complaints are made about the cosmological constant and they are usually made in the same breath, which obscures that they are different complaints with different statuses.

The first is that the natural theoretical estimate of a vacuum energy density is larger than the observed one by something like $10^{120}$. That is a genuine and unresolved problem in field theory, it is about a number that has never been computed correctly, and nothing in this essay touches it.

The second is the coincidence problem: that the density of the cosmological constant and the density of matter are within a factor of two of each other now, when one of them has been falling as the cube of the expansion for thirteen billion years and the other has not moved. Over a logarithmic history spanning sixty decades in the scale factor, they cross once, and the present sits close to the crossing.

That second complaint has a status the first does not, because it can be turned into arithmetic. If a much larger constant would have left no galaxies, then the range of constants compatible with anybody being present to notice the coincidence is narrow — and the coincidence is not a fact about the universe, it is a fact about where in the range of possible universes an observer can be.

Why growth stops #

The mechanism is simple and it is the whole of the argument.

An overdensity grows because its extra gravity pulls material in faster than the expansion pulls it apart. During matter domination the two are finely balanced in a way that makes the growth a power law: a perturbation’s amplitude grows in proportion to the scale factor, steadily, for ever.

Once the cosmological constant dominates, the expansion becomes exponential. An overdensity that has not already collapsed is now being pulled apart faster than it can pull itself together, and its growth stops. Not slows — stops. The linear growth factor approaches a finite limit:

$$D(a) = \frac{5\Omega_{\rm m}}{2}E(a)\int_0^a \frac{da'}{\left(a'E(a')\right)^3},$$

and the integral converges as $a\to\infty$ because the integrand falls as $a^{-3}$ once $\Lambda$ dominates.

Normalise so that $D = a$ while matter dominates, and $D_\infty$ is a pure number depending only on the ratio $\Omega_\Lambda/\Omega_{\rm m}$ — not on either separately, and not on the Hubble constant. It is 1.11 for the universe there is.

That number is the whole of what the cosmological constant does to structure. Everything that was going to collapse, collapses by the time the growth factor reaches its limit; everything that has not is frozen at whatever amplitude it had reached, for ever.

From a growth factor to a fraction #

Turning $D_\infty$ into a fraction of matter that collapses requires one more ingredient and a standard piece of statistics.

The primordial density field is close to Gaussian, so at any scale the overdensity is a normal variable with some dispersion $\sigma$. A region collapses when its linearly extrapolated overdensity exceeds a threshold, which spherical collapse puts at 1.686. The fraction of the mass in regions above that threshold is then the tail of a Gaussian:

$$F = \mathrm{erfc}\!\left(\frac{\delta_c}{\sqrt{2}\,\sigma_\infty}\right),\qquad \sigma_\infty = \sigma_{\rm prim}D_\infty .$$

That is the Press–Schechter form, and it is approximate — it is known to overpredict the abundance of small haloes and underpredict the largest — but nothing in this argument depends on the approximation, because what is being compared is the same expression evaluated at different values of $\Lambda$.

The primordial amplitude $\sigma_{\rm prim}$ is held fixed. That is the substantive assumption: the question being asked is what happens if only the cosmological constant is varied, with the same initial fluctuations. Vary the fluctuation amplitude as well and the bound moves, which is a complication the last section returns to.

What the number is, and what it is not #

The computed bound is a factor of about fourteen, at the fluctuation amplitude the observed universe has and for the collapsed fraction falling to a tenth of its present value.

Two qualifications attach to it immediately.

The threshold is a choice. “Enough structure for an observer” has no precise definition, and the answer depends on where the line is drawn — a tenth of the present collapsed fraction gives fourteen, a hundredth gives more, and a definition in terms of the mass scale that forms rather than the fraction gives something else again. The order of magnitude is robust and the number is not.

It is a bound, not a prediction. Nothing in the calculation says the cosmological constant should be near its maximum; it says only that it cannot be far above it. Weinberg’s original point in 1987 was sharper than a bound: if the constant varies across some ensemble with a distribution that is smooth near zero — as any distribution with a natural scale of $10^{120}$ times the observed value would be — then a typical observer sees a value of the order of the largest that permits structure, because that is where the volume of the distribution is. The prediction is therefore not “small” but “near the bound”, and the observed value was found, ten years later, to be exactly that.

That is a prediction that was made and confirmed, and it is worth stating because the anthropic argument is usually described as unfalsifiable. This instance of it was not: it predicted a nonzero constant of a particular order of magnitude at a time when the accepted value was zero, and the supernova measurements of 1998 found one.

What a universe with fourteen times as much would look like #

It helps to say concretely what the bound excludes, because “no structure” is doing a lot of work in the sentence.

At fourteen times the observed constant, the crossing between the matter and vacuum densities happens at a redshift near 2.4 rather than 0.29 — roughly eleven billion years earlier. Everything that had collapsed by then would still be there: the earliest galaxies, forming at redshifts of six to ten, would exist. What would not exist is everything assembled since.

That matters more than it sounds, because most of the mass in bound objects assembled late. The abundance of massive haloes grows steeply with time during matter domination, and a galaxy of the Milky Way’s mass reached it only in the last several billion years. Freeze the growth at $z = 2.4$ and the largest common objects are dwarf galaxies; clusters never form; and the merger history that built anything larger simply stops.

There would also be less time. The universe would be younger at any given redshift and would begin expanding exponentially before most of the stars that have ever formed had formed — the cosmic star-formation rate peaks near $z = 2$, which is almost exactly where this universe’s growth would freeze.

Push to a hundred times and the picture is harsher: the crossing is at redshift 6.5, before the bulk of reionisation, and the collapsed fraction the calculation returns is a thousandth of the present one.

So the bound is not a cliff at fourteen. It is a steep fall beginning around ten and complete by a few hundred, and where it is called a bound depends on how much structure is deemed enough — which is the qualification the last section is about.

The argument against the argument #

The anthropic reading is contested, and the objections are worth stating properly rather than dismissed.

The ensemble is not observed. The reasoning requires that the cosmological constant actually takes different values somewhere — in other regions, other vacua, other universes — and that is a hypothesis with no independent support. Without an ensemble there is nothing to be typical within, and the calculation becomes a statement about a counterfactual.

The measure is undefined. Even granting an ensemble, computing what a typical observer sees requires a way of counting observers, and in an infinite or eternally inflating spacetime every such counting scheme gives a different answer. This is the measure problem and it is unsolved. A prediction whose value depends on an arbitrary choice of counting is not a prediction.

Varying one parameter is not the ensemble. The calculation holds everything else fixed and varies $\Lambda$. If the fluctuation amplitude, the baryon-to-photon ratio and the matter density also vary across the ensemble, the joint distribution matters and the bound on $\Lambda$ alone is not the relevant quantity. The second figure shows how much a single other parameter moves it.

And the coincidence may not need explaining. The crossing of two densities that fall at different rates had to happen at some time, and structure formation requires a long matter-dominated era, since growth only runs while matter dominates, so any observer capable of asking exists within a few e-folds of the crossing more or less by construction. That argument does not need an ensemble at all — it needs only that observers take a few billion years to appear.

The last objection is the strongest and it is also the least satisfying, because it explains the coincidence by an anthropic step of its own.

Why it was proposed before the constant was found #

The chronology is the strongest thing about this argument and it is usually left out.

Through the 1970s and 1980s the cosmological constant was assumed to be exactly zero. Nobody could compute why, but zero is the kind of number a symmetry might produce, and a small nonzero value is not. The theoretical position was that some mechanism would eventually be found setting it to zero exactly.

Weinberg’s 1987 paper accepted that no such mechanism was known and asked a different question: if the constant varies across some larger structure, what values are compatible with there being anybody to measure it? The calculation is the one in this essay, and the answer was a bound of order a hundred times the matter density.

He then made the additional step that turns a bound into a prediction. If the underlying distribution of possible values is smooth on the scale of the bound — and any distribution whose natural scale is $10^{120}$ times larger is extremely smooth on that scale — then almost all of the probability compatible with observers sits near the top of the allowed range, not near zero. So the prediction is that the constant should be comparable to the matter density rather than negligible against it.

At the time, that prediction was in conflict with the accepted value. Eleven years later the supernova measurements found a constant of about twice the matter density, which is to say inside the predicted range and of the predicted order.

A prediction made from an argument widely regarded as unfalsifiable was published, was in conflict with the consensus, and turned out to be right. That does not establish the ensemble or the measure; it does mean the argument cannot be dismissed as unable to say anything.

What is actually measured here #

It is worth separating which parts of this essay are observations and which are arithmetic about hypotheticals, because the ratio is unusual for this collection.

Observed: the present densities of matter and of the cosmological constant, to a couple of per cent; the primordial fluctuation amplitude, from the microwave background, to under a per cent; the present collapsed fraction, from galaxy surveys, to a factor of maybe two depending on what counts as collapsed.

Computed from those: the asymptotic growth factor, exactly; the collapsed fraction at any other $\Lambda$, given the Press–Schechter form.

Assumed: that $\Lambda$ could have been otherwise; that the other parameters would not have been; and that structure is what an observer requires.

The first two are solid and the third is not an empirical claim at all. The value of the calculation is that it converts a rhetorical objection into a number, and the number is small enough to be worth arguing about — which a hundred and twenty orders of magnitude is not.

Under that reading the collapsed fraction is not an abstraction: it counts the systems that reached the state the Local Group is already in. What remains is to ask how well the component doing the freezing is actually known, and the answer is thinner than the argument built on it.

The other side of the bound #

The calculation above is one-sided, and the missing side is worth a paragraph because it makes the allowed window a window rather than a ceiling.

A negative cosmological constant is also possible in principle, and it is excluded far more sharply than a large positive one. A universe with negative vacuum energy decelerates increasingly, reaches a maximum size and recollapses, and the time it has before doing so is set by how negative the constant is. At a magnitude comparable with the observed one the universe would have recollapsed before now; at ten times it, within a couple of billion years of the big bang.

So the allowed range runs from a few times the observed value in the negative direction to about fourteen in the positive, and the observed value sits inside a window roughly twenty times wide. Against a prior spanning $10^{120}$, that is a very narrow window, and the anthropic reading is that the observed value being inside it is the only fact requiring explanation.

The window is narrow on both sides and the observed value is not in the middle of it, which is the detail the prediction turns on: a smooth prior puts most of its probability near the edges of the window rather than at zero, and the observed value is nearer the positive edge than the centre.

Choosing the criterion #

There is a choice buried in “how much structure counts”, and the alternatives give different bounds, which is worth laying out because the spread between them is the honest uncertainty on the number.

The collapsed fraction, used here, asks what share of the matter ends up in bound objects of any size. It is the most conservative criterion because small haloes are easy to make and the fraction falls slowly.

The largest halo mass asks instead how big the biggest bound objects get. That falls much faster with $\Lambda$, because the massive end of the halo distribution is the exponential tail, so a criterion demanding galaxy-sized haloes rather than any haloes tightens the bound by several times.

The total mass turned into stars is closer to what the argument wants and hardest to compute, because it needs a model of how gas cools and forms stars in haloes of each mass — and that model carries its own dependence on the cosmology.

The three criteria span roughly an order of magnitude in the bound, which is comparable with the effect of varying the fluctuation amplitude in the second figure. The number is a factor of ten to a hundred, and any single value quoted for it is a choice of criterion, which is why the useful statement is the order of magnitude rather than the digit.

If it is not a constant #

The whole argument assumes the dark energy is a cosmological constant, and that assumption is doing more work than it appears to.

If instead the density evolves — if the equation of state is not exactly $-1$ — then the growth of structure does not freeze at a fixed amplitude but at one depending on the history, and the bound changes. More importantly, a component whose density tracks the matter density would remove the coincidence entirely: if the ratio is fixed by a dynamical mechanism rather than by where in history the observation is made, there is nothing to explain.

That is the appeal of the tracking models, and it is why whether $w$ is exactly $-1$ is a question about the coincidence as well as about the component. A measurement of $w$ differing from $-1$ would not merely add a parameter; it would remove the motivation for the anthropic reading.

The current constraints put $w$ within a few per cent of $-1$ and are consistent with a constant. They are not precise enough to exclude the tracking behaviour that would matter, and improving them is the stated purpose of the surveys now running.

Still open: what has not been examined #

Four essays have taken the budget apart: what its denominator is, where the ordinary matter actually sits, what the one component that changes category does to structure, and how far the largest component could have been otherwise.

What has not been examined is the curvature. Every figure here assumes the total is exactly one, and the measurement that gives $\Omega_{\rm total} = 1.000 \pm 0.002$ is a single angle interpreted through an assumed expansion history — so the flatness is conditional in a way the other entries are not. Releasing curvature alone widens the inferred expansion rate by a factor of several, and the degeneracy that does it has a shape worth drawing.