At a year, most candidates are noise

An occurrence rate is a count divided by a probability, and the careful work in measuring one has almost always gone into the probability — the chance that a planet of a given size and period would transit, would be detected, and would survive the vetting. The count was treated as the easy half. A candidate that passed every test was a planet, and the number of planets in a bin was the number of candidates in it.

That is nearly true at short periods and badly untrue at long ones. Near the detection threshold, for orbits of about a year — exactly the orbits where Earth analogues round Sun-like stars would be — more than half the candidates a transit survey reports are not planets and not astrophysical impostors either. They are the noise of the instrument, arranged by chance into something that looks like two or three transits, and nothing in any of their light curves says so.

The problem is distinct from the one usually meant by “false positives”, and it is worth separating them at the start. An astrophysical false positive is a real dip in a real star’s light caused by something other than a planet — an eclipsing binary blended with the target, a background star. It is a genuine signal with the wrong explanation, and it can be tested for: by the shape of the dip, by a shift in the image’s centre of light, by the depth in different colours. A false alarm is not a signal at all. It is a coincidence in the noise that crosses the threshold, and it cannot be tested by looking harder at itself, because there is nothing there to look at.

A threshold set for the wrong noise #

The detection threshold of the Kepler pipeline was 7.1 — a candidate was kept if the combined signal of its transits exceeded 7.1 times the noise. The number was not arbitrary. It came from a calculation about how many chances the search had to be fooled.

A transit search tries every period, every phase and every duration on every star, and each combination is a separate trial in which pure noise might, by chance, line up into something transit-shaped. Across 150,000 stars and four years of data the number of effectively independent trials is of order $10^{12}$. For Gaussian noise, the probability that a single trial exceeds 7.1 standard deviations is about $6\times10^{-13}$, and the expected number of false alarms in the whole survey is then below one. The peak that appears in nothing at all — the highest point of a periodogram of pure noise — is the same problem in the frequency domain, and the same answer: the threshold has to rise with the number of places searched, and a threshold set that way is right for exactly the noise distribution it was calculated for.

Real noise has a heavier tail than a Gaussian. A space photometer’s light curves carry the jitter of the spacecraft’s pointing, the thermal flexing of the telescope, cosmic-ray hits, electronic crosstalk between detector channels, and the variability of the stars themselves — spots, flares, granulation, the same stellar surface that can imitate a planet in a radial-velocity curve. Most of that is removed or averaged down. What survives is a small population of events that are individually large and not Gaussian at all, and a threshold chosen for a Gaussian passes them. The figure’s comparison is not a claim about the actual shape of Kepler’s noise, which was not a Student distribution; it is a demonstration that a change in the far tail far too small to see in the core is enough to turn less than one false alarm into tens of millions of candidate events, of which the pipeline’s later stages must then reject all but a few.

Why long periods are where the noise wins #

Most of those events are rejected easily, because a genuine transit has to recur. A search for a ten-day planet in four years of data requires about 140 transits, each at the right moment and each of the right depth and duration, and a coincidence of noise that satisfies all of those at once is extraordinarily unlikely. A few large glitches can cross the threshold in a single-event statistic; they cannot line up into 140 evenly spaced copies of themselves.

A one-year planet in four years of data has three or four transits. Three glitches lined up at equal intervals, with the right depth and duration, is a much less demanding coincidence, and the pipeline sees very many opportunities for it. Worse, the glitches are not independent of the calendar. The Kepler spacecraft rotated by ninety degrees every quarter to keep its solar panels on the Sun, so each star moved from one detector to another four times a year, and detector-specific artefacts — patterns of electronic interference that drifted across the image in bands — recurred on that cycle. Some of the noise was periodic, at periods near a year, precisely where the planets sought were.

The shape of the two curves explains why raising the threshold does not simply solve the problem. The planets are spread across a wide range of the statistic, falling slowly as it rises, because planets come in a range of sizes and a larger planet gives a stronger signal. The false alarms are piled up against the threshold, falling steeply above it. Raising the threshold removes false alarms faster than planets, which is good, but it removes the smallest planets first, and the smallest planets at long periods are the ones the whole exercise is about. An Earth-size planet on a one-year orbit round a Sun-like star, observed for four years, produces a statistic of seven to ten. The planets sought and the noise that imitates them occupy the same narrow strip of the detection axis, and no threshold separates them.

Why the candidate cannot vouch for itself #

A transit candidate is vetted by asking its light curve questions that a real planet answers one way and most impostors another. The four contact points of a transit give it a characteristic shape — a flat bottom with sloping sides — that a grazing eclipsing binary does not have. The odd-numbered and even-numbered transits of a planet are identical, where an eclipsing binary mistaken for a planet at half its true period shows alternating depths. The duration is fixed by the orbit and the star’s density, and a duration inconsistent with the star is suspicious.

Every one of those tests needs signal. The shape of a transit, the difference between odd and even depths, the precise duration — each is measured from the same few transits that make up the detection, with the same noise, and at a detection statistic of eight the shape is known to perhaps a quarter of the depth. A false alarm at that statistic passes the shape test for the same reason it passed the detection: its noise happened to form a dip, and a dip of that size in that noise is roughly transit-shaped because nothing about the noise can be measured more finely than the dip itself. A candidate near threshold has spent all its signal on being detected and has none left over to be examined with.

That is the deep reason the problem cannot be solved one candidate at a time. It is a property of the population near threshold, and it can only be measured on a population — which is what the next step does.

Measuring a false-alarm rate with no planets in it #

If a false alarm cannot be recognised by looking at it, the rate of false alarms has to be measured on data in which every detection is known to be false. The Kepler team’s final catalogue did this in two ways, and between them they are the most elegant measurement in the whole census.

The first is inversion. Turn every light curve upside down, so that a dip becomes a bump and a bump a dip, and run the entire pipeline on the inverted data — detection, vetting, everything. A real transit becomes a brightening, which no transit search will pick up. But noise that is symmetric in sign — as much likely to go up as down — looks the same inverted, and every candidate the pipeline reports from the inverted data is a false alarm of that kind. The count of inverted candidates at each period and statistic is a direct measurement of the false-alarm rate.

The second is scrambling. Keep the light curves the right way up but shuffle the order of the observing seasons — put the data from the third year first and the first year third. A real planet’s transits, strictly periodic in the true order, are no longer evenly spaced in the scrambled one and are not detected at their period. Noise that is not periodic across seasons is unaffected. Every candidate from the scrambled data is again a false alarm.

The two methods miss different things, which is why both were used. Inversion misses noise that is not symmetric: a sudden drop in a pixel’s sensitivity after a cosmic-ray hit produces only dips, and inverted it becomes only bumps. Scrambling misses noise that repeats with the season, since it rearranges seasons whole. The measured false-alarm rate is taken from the pair, and the uncertainty in it is set largely by how different the two are.

The result could then be folded into the rate: each candidate weighted by the probability that a candidate with its period and statistic is real. That probability is the reliability, and at the long-period, low-statistic corner it came out well below one half.

How much it moves the answer #

A rate that counts every candidate as a planet, and corrects only for completeness, is not noisy near the threshold. It is biased, and in one direction.

The figure also shows why the bias escaped notice for so long. An analysis that raised its threshold to be safe got a lower rate, and one that used all the candidates got a higher one, and each could read the difference as the effect of completeness — the survey’s declining sensitivity near threshold, which every analysis corrected for. The two corrections push in the same direction near the threshold and cannot be separated without an independent measurement of the false alarms. Early estimates of the rate of Earth-size planets in the habitable zones of Sun-like stars that used the whole candidate list were among the highest published; the reliability corrections made later brought the rates in that region down by up to about a factor of two, the same size as the spread that comes from how the region is defined.

When the noise is worse than modelled #

The false-alarm tail is the least certain part of all of this, and the conclusions should be checked against a noisier instrument than the one assumed.

That asymmetry is a general property, not a feature of the model’s parameters. Short-period candidates are protected by the number of transits they require; long-period ones are exposed to whatever the noise does. A survey that is excellent at short periods can be very poor at long ones with no visible change in its data, and the only instrument that reveals the difference is the inverted-and-scrambled run — which is why later transit surveys, with shorter observing windows on most stars and far fewer long-period candidates, have had to build their own.

The same trick, in other searches for rare things #

The inversion and the scramble are instances of a method every field that searches for rare signals in large data has had to invent, and the other inventions make the logic clearer.

The gravitational-wave detectors search the output of two or three interferometers for coincident chirps, and their noise is badly non-Gaussian — glitches from the instruments and their environment arrive far more often than a Gaussian tail allows. Their false-alarm rate is measured by time slides: shift one detector’s data against another’s by more than the light travel time between them, many times over, and search the shifted data exactly as the real data are searched. No real astrophysical signal can be coincident in shifted data, so every coincidence found there is noise, and the rate of such coincidences is the background against which a detected merger is judged. It is the scramble, applied across detectors instead of across seasons.

Particle physics met the same problem as the look-elsewhere effect. A search for a new particle scans a range of masses, and a local excess of three standard deviations somewhere in the range is expected by chance once the number of places looked at is counted; the conventional demand of five standard deviations for a discovery exists partly because of it. Several excesses of three or four standard deviations have appeared and vanished with more data, each of them real noise, correctly computed, arising in a search that looked in many places.

And the timing of millisecond pulsars established its background of nanohertz gravitational waves in the same way: the signature is a correlation between pulsar pairs that depends on their angular separation, and its significance was measured by scrambling the pulsars’ sky positions or shifting their phases, destroying the correlation a real background would produce while keeping every property of the noise.

In each case the method is the same: build a copy of the data in which the signal is known to be absent and the noise is known to be preserved, and count what the search finds there. The hard part is always the second clause — preserving the noise — and each field’s version fails for the kind of noise its transformation does not preserve.

One candidate, examined #

The effect is clearest in a single case. In 2015 a planet was announced round a Sun-like star, about one and a half times the Earth’s size on a 385-day orbit — as close to an Earth twin as the survey had found. Its signal came from four transits, and its detection statistic sat not far above the threshold.

When the final catalogue’s reliability calculation was applied to it, with the improved stellar parameters and the inverted-and-scrambled false-alarm rates, the probability that its signal was a false alarm came out at a level that could not be dismissed. The planet may well be real. What changed is that its reality became a number rather than a fact, and a number that depends on the false-alarm rate of a noise distribution at a period and a statistic where that rate is least well known. A single candidate of that kind cannot be confirmed by the survey that found it; it needs another instrument — a radial-velocity signal that for an Earth-mass planet round a Sun-like star is nine centimetres a second, below what any spectrograph has yet achieved, or a future transit observed from elsewhere.

The Earth analogues of the transit census are therefore a population and not a list. The rate of them is known, with an uncertainty that includes the reliability; the identity of any particular one is known only to a probability. That is the same situation as the rates themselves, which are sums over probabilities, carried one level down into the candidates.

What the model leaves out #

The planets and the false alarms are modelled, not measured. The planet curve is a falling power law in the detection statistic above a completeness ramp, and the false-alarm curve an exponential above the threshold whose length and amplitude rise with period; both are chosen to reproduce the published pattern — reliability near one at short periods, falling below a half at a year near threshold — and not fitted to the published tables.

The noise figure’s heavy tail is a Student distribution, chosen because it has a single parameter that moves the tail while holding the variance. Real photometric noise is not drawn from any named distribution, and its tail varies from star to star and from season to season. The figure demonstrates sensitivity, not the actual number.

Reliability depends on the vetting as well as on the noise. A false alarm that fails a later test — a transit whose shape is wrong, or whose depth differs between odd and even events — is removed before it becomes a candidate. The curves drawn are for what survives the vetting, and a better vetting moves them without changing the noise at all. The comparison stars used to correct the photometry are part of the same correction and carry their own share of the systematics.

And the false alarms are treated as independent of the planets. A glitch on a star that also has a real planet can alter that planet’s measured depth or period rather than creating a new candidate, which is a different error with a different correction.

Still open: the corner that one survey had to itself #

The long-period, small-planet corner of the Kepler census was measured by one instrument, and its reliability was measured by running that instrument’s pipeline on that instrument’s data turned upside down. Nothing else has observed the same stars long enough, and precisely enough, to check it independently. The next transit survey designed for long periods round Sun-like stars will stare at a field for years again, with different detectors, a different pointing strategy and a different noise distribution, and its catalogue will carry its own false alarms in its own corner. Whether the two surveys agree on the rate of Earth-size planets on year-long orbits — after each has measured its own reliability — is the first test of the reliability correction that does not depend on the correction itself. If they agree, the rate of Earth analogues will be a measurement of the sky. If they disagree by about a factor of two, it will have been, all along, partly a measurement of the noise.