A prior the survey measures through its own noise

A radial-velocity signal near the noise can be judged in two ways. The first asks how often noise alone would produce a peak as tall, which gives a false-alarm probability that depends only on the noise. The second asks how much more probable the data are with a planet than without one, which gives a Bayes factor — and the Bayes factor depends on what amplitude a planet would be expected to have. A prior that allows planets of any size up to large amplitudes spreads its belief thinly and penalises a small signal; a prior concentrated at small amplitudes favours it. The same data can give odds of several to one either way.

The obvious remedy is to make the prior a measurement. The distribution of planet masses and periods has been measured by the radial-velocity surveys themselves, and converted to semi-amplitudes for a given star it is exactly the prior the Bayes factor needs: how large are planets, around stars like this, actually found to be? The difficulty is that the population’s small end — the part that matters for a marginal signal — was measured from marginal signals. The prior for judging a weak detection comes from weak detections, whose reality was judged with some earlier prior. This essay is about what that circle does to the numbers and how it is broken.

An amplitude is biased before it is selected #

The first effect is present even before a threshold is applied, and it comes from what an amplitude is. At a known period, a radial-velocity signal is fitted as a sine and a cosine, each with its own noise; the amplitude is the length of the vector they make. A length cannot be negative, and noise added to both components makes the vector longer on average, not shorter.

A planet with no signal at all — a star with nothing orbiting at that period — is measured, on average, with an amplitude of 1.25 times the noise. A planet with an amplitude equal to the noise is measured as 1.55. The distributions are Rice distributions, the same that describe the amplitude of any sinusoid measured through Gaussian noise, and they approach the truth only when the signal is several times the noise. Every catalogue of amplitudes contains this bias for its weakest entries, and it is quite separate from the question of whether those entries are planets.

The same bias in every length #

The upward bias of a noisy amplitude is not peculiar to radial velocities. It afflicts every quantity that is the length of a vector estimated from noisy components. An orbit’s eccentricity is one: it is the length of a vector whose two components are fitted from the data, and an eccentricity fitted to noisy data never comes back zero, even for a perfectly circular orbit — the classic form of the problem, noticed in spectroscopic binaries half a century ago. The degree of polarisation of faint light is another, and so is the amplitude of any periodic signal found in a time series, from a pulsating star’s to the reflex motion of a planet’s host.

The remedies are the same in every case. A single measurement can be corrected only with a prior on the true values, and the prior is only as good as the population it comes from; a population can be estimated without bias only by modelling the measurement process, not by averaging the measured values. The eccentricity distribution of exoplanets has been revised for exactly this reason: the fraction of apparently eccentric orbits among planets with small amplitudes was inflated by the same arithmetic as their amplitudes, and hierarchical fits that model the bias find the small planets’ orbits closer to circular than the raw catalogues suggest.

The threshold keeps the lucky ones #

A detection threshold selects the upper part of each distribution. A planet whose measured amplitude clears the threshold is, near the threshold, a planet whose noise happened to lengthen its amplitude; the ones whose noise shortened it fell below and were not counted. Since the population contains more small planets than large ones, more are promoted across the threshold than demoted, and the detections just above it are systematically too large — the same arithmetic that inflates marginal transit depths, and every flux-limited count near its limit.

In the simulated survey, the planets detected within one unit of the threshold have measured amplitudes a third larger than their true ones, on average. Since the minimum mass of a planet is proportional to its semi-amplitude, a minimum mass that is itself a lower bound is, for a marginal detection, inflated by the same third before the unknown inclination is even considered. Well above the threshold the detections sit on the diagonal and the bias disappears. Below it the measured amplitudes level off near the noise’s own contribution, whatever the true amplitude, and the undetected planets are indistinguishable from stars without planets.

What a population prior does to one planet #

If the population were known, each marginal planet’s amplitude could be corrected by asking what true amplitudes, weighted by how common they are, would produce the measured one. That is the posterior mean under the population prior.

The correction is a shrinkage: the measured amplitude is pulled towards the part of the population where true amplitudes are common, which, for a population rising towards small planets, is downward. At the threshold, a measured amplitude of four times the noise corresponds to a true amplitude of about 3.3 for the population drawn — a fifth smaller — and to 2.9 for a steeper population. Well above the noise the prior no longer matters and the measurement stands on its own. This is the individual-planet side of the population prior: it does to each marginal detection what the survey’s statistics say should be done to it. It also shows why the circle matters. The correction depends on the steepness of the population, and the steepness is measured from the detections being corrected.

One marginal planet, corrected #

The size of the correction for a single planet is easiest to see with numbers attached. Suppose a star’s radial velocities, a hundred measurements with a metre per second of scatter each, show a signal at some period with a fitted semi-amplitude of 0.6 metres per second, where the amplitude’s uncertainty is about 0.14 — a signal at a little over four times its noise, just past a typical threshold. Taken at face value, around a Sun-like star at a period of a few weeks, that is a minimum mass of a couple of Earth masses. Under a population prior of the steepness drawn, the posterior mean amplitude is about a fifth lower, and so is the minimum mass. The planet is still probably there — its evidence is enough to clear the threshold — but its most probable mass is smaller than the number quoted, and its uncertainty, once the prior’s own uncertainty is included, is larger.

For a single planet that is a modest adjustment. For a catalogue it is not, because the adjustment is always in the same direction. A list of marginal detections quoted at their fitted amplitudes is a list of planets all somewhat too massive, and a mass–radius relation, an occurrence rate or a comparison with formation models built on it inherits the shift. The correction is also what changes how many planets a star appears to have: a system whose third planet sits at the threshold looks like a two-planet system in some analyses and a three-planet one in others, and the population of multiplicities measured from such systems depends on how marginal members were treated.

The slope of a population read from its detections #

The natural way to measure the population is to fit a power law to the detected planets’ amplitudes, above the threshold where the survey is complete. It is also wrong, in a predictable direction.

The direct fit comes out steeper than the truth, and the error grows with the steepness. Planets scattered up from below the threshold crowd the amplitudes just above it, so the detected sample looks richer in small planets than the population really is, and the fitted law rises too fast towards the threshold. For a flat population the effect vanishes, because there are as many planets above the threshold to be scattered down as below to be scattered up; for the steep populations actually observed, it is a third of the index or more.

The repair is to fit not the detections but the survey. For each trial population, simulate the survey — the same stars, the same noise, the same threshold, the same fitting — and compare the simulated detections with the real ones; the population whose simulated detections look like the observed ones is the estimate. The forward model puts the noise and the selection where they belong, inside the model, instead of treating the measured amplitudes as exact and the sample as complete. In the simulated surveys it recovers the input index to within its statistical error at every steepness drawn.

The count below the threshold #

The quantity surveys most want is not the slope but what the slope implies for planets too small to detect — Earth-mass planets, whose semi-amplitudes around Sun-like stars are a few tenths of a metre per second, below the reach of all but the most precise instruments.

Extrapolating the naive law over the factor of three to eight in amplitude between the detected planets and the band of interest multiplies its error in the slope. For the steeper populations the naive prediction of how many small planets there are is more than twice the truth; the forward-modelled one is within a quarter. That is the planets that were not seen counted by extrapolation, and the method of the extrapolation matters more than the number of detections it rests on — on top of the part of any such rate that is a definition, the choice of which box of masses and periods is being counted. It is also exactly the prior that a Bayes factor for a new marginal signal would use. A survey that measured its population naively would assign every new small signal a prior that expects too many small planets, and would be too ready to believe in them.

Why the naive estimate lasted #

The direct fit to detected amplitudes was the standard method for years, and not through carelessness. Its errors are invisible in the data it uses. A power law fitted to detections describes the detections well; residuals from the fit look like noise; the fitted slope has a small statistical error when there are many detections. Nothing inside the analysis signals that the slope is biased, because the bias lives in the relation between the detections and the planets that were not detected, and the naive method never models the latter. Only a simulation of the survey, with known inputs, shows the gap — the same lesson the luminosity functions of galaxies taught, where the scatter of measured brightnesses about true ones steepened the bright end of the count until it was modelled.

The forward model’s cost is that it must be told everything about the survey: which stars were observed, how often, with what noise, with what search and what threshold, and how candidates were vetted. A published catalogue of detections does not contain that information, which is why population analyses by groups outside a survey have tended to use the naive method, and why the most reliable population estimates have come from the surveys themselves, which could simulate their own pipelines. That dependence is itself a reason for surveys to publish their non-detections and their sensitivity alongside their planets.

Measuring the population and the planets together #

The circle is broken by not separating the two steps. A hierarchical model treats the population’s parameters — its slope, its normalisation, its dependence on period and on the star — as unknowns alongside every planet’s amplitude, period and phase, and fits them all to all the data at once: every star’s radial velocities, including the stars with no detection, whose velocities say that any planet they have is small. The population informs each planet’s amplitude through the prior; each planet’s data inform the population through its likelihood; and nothing is thresholded, because a marginal signal enters with exactly the weight its data give it, neither promoted to a detection nor discarded.

The result is a population that is consistent with the individual planets and a set of individual planets consistent with the population, and the marginal signals are handled by their probabilities rather than by a cut. It is computationally heavy — tens of thousands of stars, each with its own period search — and it depends on modelling the noise correctly for every star, which is its real weak point. But it answers the question the Bayes factor left open: the prior for a marginal planet is the population, and the population is measured, not guessed, with the marginal planets contributing only what they are worth.

What the simulations leave out #

The simulated surveys make the problem cleaner than real ones in three ways. Each star has exactly one planet at a known period, so the amplitude is fitted at the right period; a real signal has to be found first, and a search over many periods raises the threshold and adds false detections whose amplitudes are pure noise, which bias the population further towards small amplitudes. The noise is white and identical for every star; real radial-velocity noise is dominated at the metre-per-second level by the star’s own surface — granulation, oscillations, spots rotating across the disc — which is correlated in time, different for every star, and can imitate a planet at the star’s rotation period and its harmonics. And the population is a single power law in amplitude, when the real one depends on period and on the star’s mass and composition, so that a survey’s population prior for one star is a slice through a many-dimensional distribution.

None of these changes the direction of the effects drawn — amplitudes near the noise are inflated, detections near a threshold more so, and a law fitted to detections is too steep — but each changes their size, and the correlated stellar noise changes it most, because a signal whose noise is not the assumed noise has a detection significance that is not the computed one.

Why the number at the bottom matters #

The occurrence of Earth-like planets around Sun-like stars is one of the handful of numbers that determine how to design the next generation of instruments and missions. Direct imaging of such planets requires knowing how many there are within reach, and radial-velocity campaigns designed to find them need to know how many stars to observe and for how long. The number currently comes partly from transit surveys, with their own extrapolations, and partly from radial-velocity surveys whose smallest detections sit at their thresholds. The figures say that the method of extrapolation can change that number by a factor of two, independently of the data, and that forward-modelled estimates are the ones to use — which is the practice the recent analyses have adopted, and the reason their answers differ from the earliest ones.

Still open: a prior for planets nobody has measured #

The instruments now reaching ten centimetres per second in principle are aimed at planets whose amplitudes are below any population anyone has measured by radial velocity: Earth analogues around Sun-like stars, at nine centimetres per second. For those, there is no population to use as a prior except one extrapolated from larger planets — or one borrowed from transit surveys, which measure radii rather than masses and relate to amplitudes only through a mass–radius relation with its own scatter. The first detections at that level will be judged with a prior that is itself an extrapolation, and the population measured from them will inherit whatever that prior assumed. Whether the hierarchical approach can be carried into a regime where every signal is marginal — where the population and the planets must be measured from data none of which, alone, would count as a detection — is the question the next decade of radial-velocity surveys will have to answer.