Cosmology

The darkness has a number in it

The night sky is not black. It carries about sixty nanowatts per square metre per steradian, and that number is the sum of every photon every star has ever emitted, redshifted and added up over thirteen billion years.

Assumes Olbers's paradox and Star formation.

The resolved paradox makes a prediction, and the prediction is not that the sky is black. It is that the sky is faint by a specific and computable factor — some fourteen orders of magnitude below the surface brightness of a star — which leaves a residue. That residue has a name, an absolute value in watts, and a spectrum. Measuring it is the only way to count photons whose sources are individually too faint to see, and it is the one observation that tests the resolution of the paradox rather than merely restating it.

The quantity is the extragalactic background light: the surface brightness of the sky in every direction, after the atmosphere, the Solar System’s dust, the Galaxy’s own emission and every resolved source have been taken out. What survives that subtraction was emitted by galaxies, and by all of them — including the ones no survey has ever detected, which is what makes the number worth having.

The darkness, itemised. The extragalactic background light as a spectrum: νIᵥ against wavelength, in nanowatts per square metre per steradian, with wavelength logarithmic. The total under the two humps is not read off a plot — it is the integral (c/4π)∫ε(z)(1+z)⁻¹|dt/dz|dz of the Madau–Dickinson star formation history, taking a continuously star-forming population to radiate 10¹⁰ solar luminosities per solar mass per year, and it comes to 37.5 nW m⁻² sr⁻¹. Half of it was emitted beyond z = 0.91, which is the figure's real content: the dark sky is dominated by light released when the universe was 45 per cent of its present age. The split between the two humps is an assumption rather than a result — 50 per cent of the starlight is taken to be absorbed by dust and re-radiated near 140 µm, and the shapes are lognormals of about the observed widths — but the AREA under each is the computed quantity. Measured, the two come to about 24 and 26: the calculation falls 25 per cent short, and whether the missing light is real or a residual foreground is unsettled.
Fig. 1 The background as a spectrum rather than a single number. Two humps: starlight that escaped its galaxy, peaking near a micron because a redshifted population of stars peaks there, and starlight that dust absorbed and re-radiated, peaking in the far infrared near 140 µm. The total under both is not read off a plot — it is the integral (c/4π)ε(z)(1+z)1dt/dzdz(c/4\pi)\int \varepsilon(z)(1+z)^{-1}|dt/dz|\,dz over a measured star formation history, and it comes to 37.5 nWm2sr1\mathrm{nW\,m^{-2}\,sr^{-1}} against a measured total near 50. Half of it was emitted beyond z=0.91z = 0.91, when the universe was 45 per cent of its present age.

One integral, and only one term in it is not geometry

The arithmetic is short enough to state in full, which is unusual for a cosmological observable and is most of why this one is worth trusting.

A comoving volume emits energy at a rate ε(z)\varepsilon(z) per unit volume. Each photon is redshifted on the way here, losing energy by a factor (1+z)1(1+z)^{-1}; the second factor of (1+z)(1+z) that appears in any flux calculation is the reduced rate of arrival, and that one is already inside the conversion from redshift to elapsed time. Sum over cosmic time, divide by 4π4\pi for the solid angle the light is spread over, and multiply by cc to turn an energy density into an intensity:

I=c4π0ε(z)1+zdtdzdz.I = \frac{c}{4\pi}\int_0^\infty \frac{\varepsilon(z)}{1+z}\left|\frac{dt}{dz}\right|dz .

Everything in that expression is geometry or an expansion history except ε\varepsilon, and ε\varepsilon factorises into two pieces of very unequal reliability.

The first is the cosmic star formation rate density — how many solar masses of new stars a cubic megaparsec makes per year — which is measured, rises to a broad peak near z=2z = 2, and falls by a factor of ten between that peak and the present day. It is among the best-determined functions in extragalactic astronomy, fitted across eight orders of magnitude of survey depth in the ultraviolet, the infrared and the radio.

The second is how much energy a solar mass of new stars radiates over its life. That is the whole uncertainty, and it is not small.

The answer is exactly proportional to it. A continuously star-forming population is taken here to radiate 101010^{10} solar luminosities per solar mass per year, which amounts to about 7×1047\times10^{-4} of the rest mass converted to light — roughly a tenth of the available hydrogen burned at the 0.7 per cent efficiency of the proton–proton chain. Move that number and the whole background moves with it, linearly, with nothing anywhere in the integral to damp the change.

The darkness, itemised. The extragalactic background light as a spectrum: νIᵥ against wavelength, in nanowatts per square metre per steradian, with wavelength logarithmic. The total under the two humps is not read off a plot — it is the integral (c/4π)∫ε(z)(1+z)⁻¹|dt/dz|dz of the Madau–Dickinson star formation history, taking a continuously star-forming population to radiate 1.7·10¹⁰ solar luminosities per solar mass per year, and it comes to 63.8 nW m⁻² sr⁻¹. Half of it was emitted beyond z = 0.91, which is the figure's real content: the dark sky is dominated by light released when the universe was 45 per cent of its present age. The split between the two humps is an assumption rather than a result — 50 per cent of the starlight is taken to be absorbed by dust and re-radiated near 140 µm, and the shapes are lognormals of about the observed widths — but the AREA under each is the computed quantity. Measured, the two come to about 24 and 26: the calculation exceeds the measurement, which for this quantity is the harder result to explain.
Fig. 2 The same integral with the energy yield raised by seventy per cent, to 1.7×10101.7\times10^{10} solar luminosities per solar mass per year. The total rises in exact proportion and now sits above the measurement rather than below it, and nothing else in the calculation has moved. That is the sense in which the background measures the star formation history only once the stellar population model is believed: the two enter as a product, and the sky measures the product.

The conversion factor depends on the initial mass function, and it depends on it strongly, because the light comes overwhelmingly from stars above two solar masses while the mass comes overwhelmingly from stars below one. A population weighted towards massive stars is brighter per unit mass formed by a factor no photometry of a distant galaxy can measure directly — the individual stars are not resolved, and the integrated colour of a young population is nearly independent of the exact slope at the top end.

That is the same degeneracy a mass-to-light ratio carries at the other end of the same problem, arriving from the other direction. There, a measured luminosity has to be turned into a mass; here, a measured mass formation rate has to be turned into a luminosity. Both conversions run through the same stellar population synthesis, and both inherit its assumptions whole.

When the darkness was emitted

The more interesting output of the integral is not the total but the distribution over redshift, because that part depends only on the shape of the history and not on the conversion factor at all.

Because the emissivity peaks near z=2z = 2 and the redshift factor penalises early light, the background is dominated neither by the present day nor by the peak. Half of the sky’s residual brightness was emitted beyond z=0.91z = 0.91 — when the universe was less than half its present age — and rather less than a third of it came from beyond z=2z = 2, because by then the (1+z)1(1+z)^{-1} has already taken two thirds of each photon’s energy.

That distribution has a consequence worth stating carefully. The dark sky is a photograph of the epoch of galaxy formation taken with no angular resolution whatever. Every galaxy in the exposure is unresolved by construction, since any galaxy bright enough to resolve has already been subtracted out of it. What is left is the population’s integral, and the integral is weighted towards exactly the redshifts at which individual galaxy counts begin to fail.

The darkness, itemised. The extragalactic background light as a spectrum: νIᵥ against wavelength, in nanowatts per square metre per steradian, with wavelength logarithmic. The total under the two humps is not read off a plot — it is the integral (c/4π)∫ε(z)(1+z)⁻¹|dt/dz|dz of the Madau–Dickinson star formation history, taking a continuously star-forming population to radiate 10¹⁰ solar luminosities per solar mass per year, and it comes to 15.0 nW m⁻² sr⁻¹. Half of it was emitted beyond z = 0.91, which is the figure's real content: the dark sky is dominated by light released when the universe was 45 per cent of its present age. The split between the two humps is an assumption rather than a result — 50 per cent of the starlight is taken to be absorbed by dust and re-radiated near 140 µm, and the shapes are lognormals of about the observed widths — but the AREA under each is the computed quantity. Measured, the two come to about 24 and 26: the calculation falls 70 per cent short, and whether the missing light is real or a residual foreground is unsettled.
Fig. 3 The background a universe forming stars at forty per cent of the observed rate would carry. The shape is untouched — the redshift at which half the light was emitted has not moved, because that is a property of the history’s shape rather than of its normalisation — and the total falls in proportion. Which is the useful separation: the spectrum’s peak positions constrain when stars formed, the height constrains how many, and only the second depends on the stellar population model.

It also explains a fact about the measurement that would otherwise look like a coincidence. The optical hump peaks near one micron rather than near the half-micron where a stellar population’s own light peaks, and the shift is not reddening — it is the redshift of the epoch the light mostly came from. A background emitted at z1z \approx 1 arrives with its peak moved by a factor of two, which is precisely what the drawn spectrum shows.

Why there are two humps and not one

Roughly half of all the starlight ever emitted has been absorbed by dust inside the galaxy that emitted it, and re-radiated at wavelengths a hundred times longer. That is not an inference from theory; it is read directly off the background, from the fact that the far-infrared hump is about as large as the optical one.

The consequence is that the two halves of the background are the same photons counted at two different wavelengths, and any accounting that uses only one of them is out by a factor of two. An optical survey that adds up its galaxies and compares the total to the optical background is comparing a half to a half, which happens to work; the same survey compared to the bolometric background is comparing a half to a whole, and will conclude that half the light in the universe is unaccounted for.

The darkness, itemised. The extragalactic background light as a spectrum: νIᵥ against wavelength, in nanowatts per square metre per steradian, with wavelength logarithmic. The total under the two humps is not read off a plot — it is the integral (c/4π)∫ε(z)(1+z)⁻¹|dt/dz|dz of the Madau–Dickinson star formation history, taking a continuously star-forming population to radiate 10¹⁰ solar luminosities per solar mass per year, and it comes to 37.5 nW m⁻² sr⁻¹. Half of it was emitted beyond z = 0.91, which is the figure's real content: the dark sky is dominated by light released when the universe was 45 per cent of its present age. The split between the two humps is an assumption rather than a result — 20 per cent of the starlight is taken to be absorbed by dust and re-radiated near 140 µm, and the shapes are lognormals of about the observed widths — but the AREA under each is the computed quantity. Measured, the two come to about 24 and 26: the calculation falls 25 per cent short, and whether the missing light is real or a residual foreground is unsettled.
Fig. 4 The same total with only a fifth of the starlight reprocessed. The optical hump grows, the infrared one collapses, and the area under the pair is unchanged — because the area is what the integral computed and the split is an assumption laid on top of it. A universe with this little dust would have its entire energy budget visible to an optical telescope, and the measured infrared background says it does not.

The dust half of the background was the surprise of the 1990s, and it arrived as a number rather than as an object. Until then the optical background was assumed to be most of it. The COBE satellite’s far-infrared measurement found a comparable amount at wavelengths where nobody had a galaxy count at all, which meant that half the star formation in the universe had been happening somewhere nobody had looked.

The galaxies responsible were found afterwards, at submillimetre wavelengths, and they turned out to be among the most luminous objects there are — obscured, dusty, and forming stars at hundreds of solar masses a year behind enough dust to hide almost all of it. The background had counted them a decade before anyone imaged one, which is the clearest demonstration there is of what an integrated measurement is for.

The darkness, itemised. The extragalactic background light as a spectrum: νIᵥ against wavelength, in nanowatts per square metre per steradian, with wavelength logarithmic. The total under the two humps is not read off a plot — it is the integral (c/4π)∫ε(z)(1+z)⁻¹|dt/dz|dz of the Madau–Dickinson star formation history, taking a continuously star-forming population to radiate 10¹⁰ solar luminosities per solar mass per year, and it comes to 37.5 nW m⁻² sr⁻¹. Half of it was emitted beyond z = 0.91, which is the figure's real content: the dark sky is dominated by light released when the universe was 45 per cent of its present age. The split between the two humps is an assumption rather than a result — 50 per cent of the starlight is taken to be absorbed by dust and re-radiated near 90 µm, and the shapes are lognormals of about the observed widths — but the AREA under each is the computed quantity. Measured, the two come to about 24 and 26: the calculation falls 25 per cent short, and whether the missing light is real or a residual foreground is unsettled.
Fig. 5 The peaks moved: the optical hump pushed to 1.6 µm, as a more redshifted or more reddened population would put it, and the infrared one to 90 µm, as warmer dust would. Those positions are the part of the spectrum carrying information about when the light was emitted and at what temperature it was reprocessed — and they are also the part this drawing assumes rather than derives. The humps are lognormals of about the observed widths, and only their areas were computed.

The hardest photometry there is

Everything above is a prediction. The measurement is harder than any other quantity in this subject, and for a reason that has nothing whatever to do with cosmology.

Sunlight scattered off interplanetary dust — the zodiacal light — is about two orders of magnitude brighter than the extragalactic background at optical wavelengths. Removing it means modelling a dust cloud whose density and scattering properties are known to perhaps ten per cent. A ten per cent error in a foreground a hundred times the signal is a ten-fold error in the signal, so the measurement is not photometry at all; it is foreground subtraction with a photometric residual.

The airglow of the Earth’s own upper atmosphere is worse still, which is why no ground-based measurement has ever been competitive. And the Galaxy’s diffuse starlight and infrared cirrus sit on top of both, varying across the sky in a way that at least offers a handle: a genuine extragalactic signal is isotropic, and a Galactic one correlates with Galactic latitude.

The cleanest way out is to leave. The New Horizons spacecraft, past the orbit of Pluto, sits where the zodiacal foreground is negligible rather than merely modelled, and its camera has made the most direct optical measurement of the background there is. It finds more light than the counted galaxies add up to — by roughly a factor of two, depending on the analysis.

The darkness, itemised. The extragalactic background light as a spectrum: νIᵥ against wavelength, in nanowatts per square metre per steradian, with wavelength logarithmic. The total under the two humps is not read off a plot — it is the integral (c/4π)∫ε(z)(1+z)⁻¹|dt/dz|dz of the Madau–Dickinson star formation history, taking a continuously star-forming population to radiate 10¹⁰ solar luminosities per solar mass per year, and it comes to 37.5 nW m⁻² sr⁻¹. Half of it was emitted beyond z = 0.91, which is the figure's real content: the dark sky is dominated by light released when the universe was 45 per cent of its present age. The split between the two humps is an assumption rather than a result — 50 per cent of the starlight is taken to be absorbed by dust and re-radiated near 140 µm, and the shapes are lognormals of about the observed widths — but the AREA under each is the computed quantity. Measured, the two come to about 34 and 26: the calculation falls 37 per cent short, and whether the missing light is real or a residual foreground is unsettled.
Fig. 6 The same calculation set against a larger measured optical component, as the spacecraft measurements prefer. The computed integral now falls thirty-eight per cent short rather than twenty-five. Whether that gap is real is unsettled, and the two possibilities are entirely different statements: either there is a population of sources nobody has counted, or there is a residual foreground nobody has removed. The figure cannot choose between them, and neither can the measurement on its own.

The candidates for a genuine excess are not exotic. Intra-halo light stripped from galaxies during mergers and left floating between them; the faint outskirts of ordinary galaxies, below any survey’s surface-brightness limit; a population of dwarf galaxies the counts have missed. Each of them is precisely the kind of object the luminosity function’s faint end is hardest to measure, and that is not a coincidence. A photon is easier to collect in aggregate than a galaxy is to detect individually, and the background is the aggregate.

The competing explanation is duller and probably more likely: a zodiacal model wrong at the few-per-cent level in the direction that produces the excess. The dust cloud has structure, including a band of material associated with asteroid families and a ring of dust trapped in resonance with the Earth, and the versions of the model that include them differ from the versions that do not by about the size of the discrepancy.

Three ways to measure a background, and they disagree

There are exactly three methods, and their disagreements are structured rather than random.

Direct photometry points a calibrated detector at a blank field and subtracts every foreground it can model. It measures the whole background, including any component made of sources too faint to resolve, and it is limited entirely by the foreground model. This is the method that finds an excess.

Integrated galaxy counts resolve every source a deep image contains, sum their fluxes, and extrapolate the count below the detection limit. It measures only light in sources, it has no foreground problem at all, and its weakness is the extrapolation — a count whose faint end rises steeply enough carries an appreciable fraction of its total below any limit, which is the same worry the faint-end slope raises in a different currency. This is the method that finds no excess.

Fluctuation analysis measures not the mean brightness but its variance from place to place on the sky. Unresolved sources clump the way galaxies do, so a diffuse background made of sources has a characteristic angular power spectrum, and a truly smooth foreground does not. The method throws away the mean — which is what the other two are arguing about — and keeps only the structure, so it can say whether an excess is made of objects without saying how much of it there is.

The three are not independent measurements of one number. They are measurements of three different numbers that coincide only if the census is complete, and the interesting statement is that the second and third nearly agree with each other while the first does not.

The floor under every deep image

One consequence of the background is entirely practical and rarely stated alongside the cosmology.

A telescope pointed at empty sky does not see nothing. It sees the background, and the background is a source of photons that arrive with Poisson noise like any other. The extragalactic background light is therefore the irreducible noise floor of deep imaging — an exposure can be extended indefinitely and the signal-to-noise rises only as the square root of the time, because the sky underneath is still delivering photons.

In practice the zodiacal light is the larger part of that floor inside the Solar System, which is why a space telescope in low Earth orbit is not as much better than a ground-based one as the absence of an atmosphere suggests. Beyond the zodiacal cloud the floor drops to the extragalactic background itself, and at that point the limit on detecting a faint galaxy is the accumulated light of every other galaxy in the universe.

There is a pleasing circularity in that. The background exists because the paradox was resolved rather than abolished — the sky is faint, not dark — and the residue that the resolution predicts is exactly what sets the limit on finding the faint sources the residue is partly made of. A survey trying to resolve the background is working against the background.

What the darkness is against

It is worth restoring the scale the resolved paradox established, because an absolute brightness in nanowatts per square metre per steradian has no intuitive size at all.

Two reasons the sky is dark, and only one of them is the expansion. The surface brightness of the sky accumulated out to a given distance, in units of the surface brightness of a star, both axes logarithmic. Every shell of thickness dr contributes the same amount, because the number of stars in it grows as r² and each one's flux falls as r⁻², so the straight rising line is the paradox: in an infinite static universe the sky reaches stellar surface brightness at the mean free path to a stellar surface, λ = 1/nσ, which for 10⁹ stars per cubic megaparsec of radius 0.6 R☉ is 1.74·10¹⁸ Mpc. The two suppressions are then computed separately, and they are wildly unequal. The finite age cuts the integral off at the particle horizon, 1.41·10⁴ Mpc, which is a factor of 1.23·10¹⁴ short of λ. The expansion then dims what is left by a further factor of 6.03, computed as the mean of (1+z)⁻² over the comoving distance to the horizon. The horizon does 14 orders of magnitude and the redshift does less than one. The common answer that the sky is dark because the universe is expanding is therefore very nearly the wrong answer: the sky is dark because the universe is young, and the expansion is a small correction on top.
Fig. 7 The accumulation that produced the paradox, for comparison. A sky saturated at the surface brightness of a star sits at the top of this plot; the horizon cuts the integral off fourteen orders of magnitude below it. The extragalactic background light is the residue at that cut — a real, positive, measured number sitting exactly where the accumulation stops. Olbers’ argument said the sky should be at the top; the resolution says where it should be instead, and the measurement agrees to within the factor by which the stellar population model is uncertain.

Sixty nanowatts per square metre per steradian is, for comparison, about a twentieth of the microwave background’s thousand. The sky’s brightest component is not starlight at all, and never was. The cavity radiation left over from recombination outweighs everything every star has ever emitted by a factor of twenty; all the light from all the stars in thirteen billion years amounts to five per cent of what one moment of the early universe left behind.

That comparison has a second reading, and it is the one that connects back to the paradox proper. Olbers asked why the sky is not as bright as a stellar surface and got the answer “not enough time”. The thermodynamic restatement asked why the sky is not as bright as the walls of the cavity it sits inside and got the answer “it is, and the walls have cooled”. Both are true, and the two numbers above are the two answers in absolute units: the residue from the stars, and the residue from the walls, differing by a factor of twenty in favour of the walls.

Four instruments, and only one points at the sky

The numbers in this essay come from four kinds of instrument and it is worth being explicit about which does what, because they are quoted together as though they were one measurement.

The star formation history is a compilation: ultraviolet luminosity densities from deep imaging, corrected for dust by an assumed attenuation law; infrared luminosity densities from far-infrared and submillimetre surveys, which measure the part the ultraviolet lost; and radio and hydrogen-alpha measurements as a cross-check at low redshift. The compilation agrees with itself to about a factor of 1.5 at the peak and rather better nearby, and the dust correction is where most of the disagreement lives.

The infrared background is COBE’s, refined since: an absolute photometer, cooled, with its own internal blackbody reference, measuring the total sky brightness and subtracting a zodiacal model and a Galactic cirrus model. It is the most secure of the direct measurements because the zodiacal cloud is faint at 140 µm in a way it is not at 0.6 µm.

The optical background is the contested one, and the spacecraft measurement is the newest entry rather than the only one. Earlier attempts used the Hubble Space Telescope with a careful zodiacal subtraction, and the dark side of the Moon as an occulting screen, and each of them was limited by exactly the foreground the outer-Solar-System measurement escapes.

And the gamma-ray constraint below is a fourth kind entirely — not a photometric measurement of the sky at all, but a measurement of what the sky removes from a spectrum. It has no foreground because it has no aperture pointed at the background.

Three simplifications, all in the same direction

The emissivity is taken to be instantaneous. A population is credited with radiating while it is being formed, where a real one radiates over the following ten billion years — most of it within the first hundred million, which is why the approximation works at all. Because the star formation history is itself broad in time, the error in the total is of order ten per cent; the error in the shape is larger, since delayed light is redshifted less than prompt light.

The two humps are shapes rather than derivations. Their areas are computed; their widths and peak positions are put in by hand at approximately the observed values. A real background spectrum has structure that no two-component fit contains — the stellar photospheric bump near 1.6 µm, and polycyclic aromatic hydrocarbon emission features near 8 µm that are a diagnostic of the dust itself.

And no figure here separates a source from a foreground, which is the entire difficulty of the measurement. Every number drawn as “measured” is the output of a foreground subtraction, and the disagreements in this field are disagreements about that subtraction rather than about photometry. A curve drawn through the measured points would imply an agreement about the data that does not exist.

The same integral read backwards

There is an elegance worth noticing in how one expression is used in two directions, and the second direction is much the more valuable.

Run forwards, with a star formation history assumed, it predicts a sky brightness — which is what this essay has done, and what makes the calculation checkable. Run backwards, with the sky brightness measured, it constrains the star formation history at redshifts where no galaxy survey is complete. A survey misses faint sources; an integrated brightness does not.

And the constraint it gives is a bound in one direction only, which happens to be the useful direction. Any population of sources that has ever emitted light contributes to the background; nothing at all subtracts from it. So a measured background is an upper limit on all the star formation nobody has counted — and the fact that the counts nearly account for it is a statement that the census is nearly complete, arrived at without counting anything.

That inference is the reason the residue is worth measuring at all, and it is why the New Horizons excess matters more than its size suggests. A factor of two in the background is a factor of two in the light from sources that were never in anybody’s catalogue, and there is no other measurement that would notice them.

Still open: whether the extra light is there

The disagreement is between a direct photometric measurement and a sum over counted galaxies, and the two have completely different failure modes. The first fails through an unremoved foreground; the second fails by missing sources. Both failures push in the direction observed, which is why twenty years of better data have not settled it.

What would settle it is a measurement of the background made without any foreground at all — and there is one, at a wavelength no telescope collects. A distant source of very high energy gamma rays loses part of its spectrum on the way here, to collisions with exactly the background photons the zodiacal light is hiding. That absorption weighs the dark sky by what it removes from something else.

About the same objects

Not linked from either essay — found by the objects both name.

What links here

Essays that link to this one from their own argument.

The objects this essay names

Each one links to every other essay that touches it.

Comoving distanceCosmological dimmingExtragalactic background lightLookback timeLuminosity densityMass-to-light ratioOlbers's paradoxStar formation historySurface brightnessZodiacal light