A forest with no continuum left

Light from a distant quasar passes through the intergalactic medium, and every parcel of neutral hydrogen along the way absorbs at its own redshifted Lyman-alpha wavelength. The result is a forest of absorption lines blueward of the quasar’s own emission, and the amount of light removed measures how much neutral gas there is.

The measurement is a fraction: the observed flux divided by what the quasar would have emitted. The numerator is measured well. The denominator is not observed at all.

The difficulty is not a small correction on an otherwise clean measurement. Above redshift five it is the dominant error, it grows without bound as the forest saturates, and it is the reason a quantity that could in principle be measured to a per cent is quoted as a limit.

Why the level is not observable #

At low redshift the forest is thin and there are stretches of spectrum between the lines where the quasar’s own continuum shows through. Drawing a curve through those stretches is straightforward and the result is good to a per cent or two.

At high redshift there are no such stretches. By redshift five the mean transmission is a few per cent and the absorption is continuous rather than discrete — every pixel is absorbed, and the highest points of the spectrum are not the continuum but the least-absorbed parts of the forest.

So the continuum has to be extrapolated from the red side of the Lyman-alpha emission line, where there is no forest, across the emission line itself, into the region being measured. That extrapolation spans a region where the quasar’s own spectrum has real structure: the broad emission line, the smaller lines around it, and a continuum slope that varies from object to object.

The extrapolation is the same operation as predicting a magnitude in a band the source was never observed in, and it fails in the same way: the prediction is only as good as the template’s applicability to this object, and the objects that most need it are the ones the template describes worst.

The standard method is to build a set of principal components from low-redshift quasars whose full spectra are visible, fit the components to the red side of a high-redshift quasar, and predict the blue side. The prediction is good to five to ten per cent, and that number is the systematic on every high-redshift forest measurement.

There is a second difficulty that compounds it and is worth naming separately: quasars are not a homogeneous population. Their continuum slopes, emission-line strengths and the relative prominence of their broad lines vary substantially from object to object, and the variation correlates with luminosity — which correlates with redshift in any flux-limited sample. So the template built from low-redshift quasars may not describe high-redshift ones, and the mismatch would be a systematic rather than a scatter, in a direction that depends on how the population evolves. That is the hardest part of the error to bound, because bounding it requires knowing the evolution the measurement is being used to study.

Why the error grows as the forest darkens #

The optical depth is minus the logarithm of the transmitted fraction, so an error in the fraction propagates as

$$\delta\tau = \frac{\delta F}{F},$$

and $F$ is falling exponentially with redshift. A fixed fractional error in the continuum is a fixed fractional error in $F$, so the error in $\tau$ grows as one over the flux.

At redshift three, with $F\approx0.7$, a five per cent continuum error is $\delta\tau\approx0.07$ against $\tau\approx0.35$ — twenty per cent, unpleasant and survivable. At redshift six, with $F\approx0.03$, the same error is $\delta\tau\approx1.7$ against $\tau\approx3.5$ — half the measurement.

Beyond that the flux is consistent with zero and the optical depth has no upper bound at all. That is the reason measurements above redshift six are quoted as lower limits on the optical depth rather than as values.

What the measurement is trying to say #

The reason the optical depth matters is that it constrains the neutral fraction of the intergalactic medium, and the neutral fraction is what reionisation is about.

The relation is unhelpfully steep in the other direction. Because the Lyman-alpha cross-section is enormous, an intergalactic medium with a neutral fraction of even $10^{-4}$ is completely opaque — so a saturated forest says only that the neutral fraction exceeds about that, which is a very weak statement. A trough proves the forest survived rather than proving reionisation had not happened.

So the measurement’s useful regime is narrow: below about redshift five and a half the forest is transmitting and the optical depth is a measurement, and above about six it is saturated and only limits are available. The interesting epoch is exactly at the boundary.

There is a further use of the forest that is much less sensitive to the continuum and is worth naming because it is now the main one: the flux power spectrum. The fluctuations of the transmitted flux along a line of sight trace the density fluctuations of the intergalactic gas, and their power spectrum measures the matter power spectrum on scales far smaller than any galaxy survey reaches. A multiplicative continuum error changes the mean flux and largely divides out of the fluctuations’ shape, so the power spectrum is a far more robust observable than the mean. It is the forest’s main contribution to cosmology today — a constraint on the neutrino mass and on the small-scale power — and it is robust for exactly the reason the mean flux is not.

There is a third robust statistic worth naming, and it is the one that turned the saturated regime from useless into informative: the length distribution of dark gaps. Where the flux is consistent with zero the optical depth is unmeasurable, and how far a dark stretch extends is still measurable — it is a length on the sky and needs no photometric calibration at all. Long dark gaps require large regions of enhanced neutral fraction or suppressed ionising background, and their distribution constrains the patchiness directly. That statistic uses only where the flux crosses a threshold, so a continuum error shifts the threshold slightly and changes the gap lengths hardly at all. A trough that proves the forest survived becomes a measurement rather than a limit when it is the trough’s length being measured.

The continuum comes back through the normalisation #

The flux power spectrum is described above as robust because a multiplicative continuum error divides out of the fluctuations’ shape. That is true of the shape and it is not the whole story, and the way the error returns is worth following because it is how a systematic survives a statistic designed to remove it.

The forest’s power spectrum is not interpreted directly. It is compared against hydrodynamic simulations of the intergalactic medium, and those simulations have a free parameter that nothing predicts: the intensity of the ultraviolet background, which sets how ionised the gas is. The standard procedure is to fix it by rescaling the simulated optical depths until the simulated mean flux matches the observed one.

So the observed mean flux — the quantity the continuum error lives in — enters the interpretation as the normalisation of the model the power spectrum is compared against. A continuum error that changed the mean flux by five per cent changes the rescaling, which changes the simulated fluctuation amplitude, which changes the inferred amplitude of the matter power spectrum. The statistic is robust and the inference is not.

The size of the leak is modest and it is not negligible against what the forest is used to constrain. A neutrino mass of a tenth of an electron volt suppresses small-scale power by a few per cent, which is the same order as the mean-flux systematic — so the forest’s neutrino limit carries a continuum error inside it, arriving three steps from where it was introduced. An amplitude and a depth that arrive multiplied is the same structure in the microwave background: a nuisance parameter that multiplies the quantity of interest and is fixed by a separate, weaker measurement.

Two further leaks travel with it. The gas temperature enters the small-scale power through thermal broadening, and it is constrained largely by the shape of the same power spectrum, so temperature and amplitude trade against each other. And the quasar sample is flux-limited, so the objects measured at high redshift are the most luminous ones — which have systematically different continua and larger proximity zones than the population as a whole. A sample brighter than the population it came from is not usually thought of as a spectroscopic problem, and here it decides which template applies.

The general lesson is that a statistic immune to a systematic is not the same as an inference immune to it. The immunity has to hold all the way through to the number being quoted, and a model normalised on the contaminated quantity gives it a way back in.

What was actually measured #

There are three, and the third is the one that changed the picture.

The evolution of the mean flux. Measured over redshifts two to five with continuum errors under five per cent, the mean transmitted flux falls smoothly and is well described by a slowly evolving photoionisation rate. That is the regime where the method works, and it constrains the intensity of the ultraviolet background over three billion years.

The first complete troughs. Quasars beyond redshift six show stretches with no detected flux at all, extending over hundreds of megaparsecs. That was initially read as the end of reionisation, and it is consistent with a neutral fraction anywhere above $10^{-4}$ — which includes fully ionised.

And the scatter between sight lines. Different quasars at the same redshift show very different amounts of transmission, far more than density fluctuations alone predict. That scatter is now the primary evidence about the end of reionisation: it indicates that the ultraviolet background is patchy, which is what a process ending at different times in different places produces. The scatter is a ratio between sight lines, so the continuum errors partly cancel — which is why it has become the preferred statistic.

One more thing worth recording, because it is a rare case of the problem being solved by better data rather than by a better statistic. The templates used to predict a quasar’s unabsorbed spectrum are built from low-redshift quasars, and their quality is limited by how many such quasars have been observed with the same instruments and the same signal-to-noise. Large spectroscopic surveys have now measured hundreds of thousands, which has reduced the prediction error from ten per cent to five and, more usefully, has made the error’s distribution measurable — so an analysis can now marginalise over a realistic continuum uncertainty rather than adopting a single prediction. That is a modest improvement in a number and a substantial improvement in what can be claimed from it.

Where the picture stops #

Three of them, and the second is the one that motivates the alternatives.

The continuum error is correlated across a spectrum. A principal-component prediction that is wrong is wrong smoothly, so the error does not average down over a spectrum’s length. It behaves like a multiplicative offset applied to the whole forest of one quasar, and combining many quasars reduces it only if the prediction errors are independent between objects — which they partly are and partly are not, since they share a template.

Saturation destroys the information rather than degrading it. Where the flux is consistent with zero there is nothing to correct. No improvement in the continuum recovers a measurement from a saturated region, and the only routes into the fully neutral epoch are different observables entirely — the damping wing of a quasar’s own absorption profile, the twenty-one centimetre line, or the microwave background’s optical depth.

And the quasar’s own neighbourhood is not typical. A quasar ionises its surroundings, so the forest immediately blueward of its emission line is more transmitting than average. That proximity zone has to be excluded, and how far to exclude depends on the quasar’s luminosity and lifetime, which are not known independently.

A fourth belongs with them, and it concerns the resolution of the spectrum. Everything above treats the transmitted flux as a smooth quantity, and it is not: at high resolution the forest resolves into individual lines with cores and wings, and the mean flux measured at low resolution is an average over structure the instrument did not resolve. Since the relation between flux and optical depth is nonlinear, an average of the flux is not the flux at the average optical depth — so a low-resolution measurement and a high-resolution one of the same sight line give different mean optical depths, by an amount that depends on the distribution of line strengths. That is a second denominator problem, arriving at the level of the instrument rather than the source.

Why a denominator nobody measured is a familiar problem #

The general shape is one this collection has met in a different subfield, and the parallel is exact.

An equivalent width is an area measured against a continuum that is drawn rather than observed, and in a crowded spectrum the drawn level is systematically too low. Here the same operation is performed across an entire spectral region rather than around a line, and the level is extrapolated from outside rather than interpolated from within — which is worse, because an extrapolation has no local check.

The remedies are the same in both cases and they are worth stating together. Work differentially, comparing objects that share the systematic: the sight-line-to-sight-line scatter here, and differential abundance analysis there. Move to a statistic that does not need the level: the flux power spectrum, whose shape is nearly independent of a multiplicative continuum error. And find a different observable that measures the same physics without the denominator, which for reionisation means the damping wing and the twenty-one centimetre line.

What none of the remedies does is measure the continuum. It is not measurable, and every technique that has made progress here has made it by not needing to.

A final observation about how the field’s framing changed. For thirty years the question was “when did the forest first appear”, and the answer was pursued by finding higher-redshift quasars and measuring their troughs. The troughs arrived, they were consistent with everything above a neutral fraction of $10^{-4}$, and the question stopped being answerable that way. What replaced it was a set of questions about variance — how much sight lines differ, how large the ionised bubbles were, how patchy the background was — all of which are answerable with the same data and none of which needs the continuum to better than the scatter being measured. Reionisation ends when the walls meet, and measuring the meeting turned out to require statistics of the differences rather than a measurement of the mean.

It is worth noting one more thing the continuum problem has cost, since it is quantifiable. Published values of the mean optical depth at redshift five from different groups differ by twenty to thirty per cent, with individually quoted uncertainties of ten. The difference is almost entirely continuum methodology — which template set, which region of the red side was fitted, how the proximity zone was excluded. That spread between analyses is the field’s own measurement of the systematic, and it is larger than any single analysis’s error bar. The disagreement between analyses is the honest uncertainty wherever a reference has to be constructed rather than observed, and it is worth looking for before believing an error bar.

Where the ladder goes next #

The rung above this one is the damping wing: the Lorentzian absorption profile a neutral medium imprints redward of a quasar’s Lyman-alpha line, which is unsaturated even when the forest is black and which is currently the best probe of the neutral fraction above redshift six. The rung beyond it is the twenty-one centimetre line — an emission or absorption signal from the neutral gas itself, requiring no background source and no continuum at all.