A magnitude measured where nothing was measured

A magnitude is supposed to be a property of a star. What a telescope records is a property of a star, an atmosphere, a mirror, a filter and a detector, and only the first of those is wanted.

The last three are removable in principle: they are stable, they can be characterised in daylight, and they change slowly. The atmosphere is not. It is different tonight from last night, different at the start of the night from the end, and different in every direction the telescope is pointed. Removing it is the central problem of ground-based photometry, and it sits underneath every distance the ladder is built out of, and the way it is removed is worth looking at directly, because the answer is arrived at from a place nobody has ever observed.

Airmass #

The quantity on the horizontal axis is the length of the sight line through the atmosphere, in units of the vertical path. For a plane-parallel atmosphere it is exactly $\sec z$ with $z$ the zenith angle, and that is right to a per cent out to about $60°$.

Past that the Earth’s curvature matters, and the secant — which diverges at the horizon — stops being usable. The standard replacement is a fit that reaches about 38 at the horizon rather than infinity.

What is doing the absorbing #

Atmospheric extinction is not one process, and the three that make it up behave differently enough that they have to be separated before anything can be said about how a night’s slope will change.

That division is the reason for the practice. If extinction were Rayleigh alone it would be a site constant, measured once and looked up thereafter. The aerosol term is what forces it to be remeasured, and the aerosol term is also the one that varies most between the beginning and the end of a night.

The intercept is the measurement #

The Bouguer method is Pierre Bouguer’s, from the 1720s, and it is the oldest surviving piece of quantitative photometry. Observe the same star at several airmasses, plot instrumental magnitude against $X$, fit a line, take the intercept.

The assumption underneath is that the atmosphere is stable over the run of observations, so that all the variation with $X$ is geometric. That is the assumption that fails, and it fails in a way that is hard to see: if the aerosol content is slowly rising through the night while the star is also rising, the fitted slope absorbs part of the change and the intercept is wrong by an amount nothing on the plot reveals. The standard defence is to observe the star both rising and setting, so that a monotonic drift in the atmosphere produces a loop in the plot rather than a line — which is visible.

The second assumption is that the extinction is grey across the passband, and it is not.

It also explains a rule of thumb that sounds arbitrary. An observer is told to keep to airmasses below about 2, and the reason is not that the correction becomes large — it is that the correction becomes uncertain: past $X = 2$ the plane-parallel approximation is failing, the aerosol layer is being traversed obliquely enough that its horizontal structure matters, and the star is looking through air that a nearby comparison star is not looking through. The linear model breaks in three ways at once.

Standard stars, and what they really provide #

In practice nobody determines an absolute intercept from scratch. What is done instead is to observe a set of standard stars — objects whose magnitudes in a defined system have been established by somebody else, over many nights, with great care — at a range of airmasses through the night, and to solve for the transformation between instrumental magnitudes and the standard system at the same time as the extinction coefficients.

That turns the problem into a fit with several unknowns per band: a zero point, an extinction coefficient, a colour term, and a transformation coefficient relating the instrument’s own passband to the standard one. Six to ten parameters for a night’s data, solved by least squares.

It also relocates the honesty. The standard system is itself defined by a set of measurements someone made — for the $UBV$ system, by Johnson and Morgan in the 1950s, with a particular photomultiplier and a particular set of filters on a particular mountain — and every magnitude in that system since has been tied to that. A magnitude has to say which light, and what it is ultimately saying is the light that instrument would have measured.

The ultraviolet, where it hurts most #

The $U$ band’s extinction coefficient is about four times the $V$ band’s, so the extrapolation from $X = 1.2$ to $X = 0$ is four times as long. It is also the band in which detectors are least sensitive and stars are usually faintest, so the photon noise on each point is largest.

Those two compound. The uncertainty on an extrapolated intercept goes as the extinction coefficient times the uncertainty in the slope times the lever arm, and all three are worst in the ultraviolet. Ground-based $U$-band photometry has always been the least trustworthy part of any magnitude system, and the discrepancies between different observatories’ $U$ magnitudes for the same stars are several times larger than in $V$.

The ozone edge makes it worse still: below about 340 nm the atmosphere becomes opaque quickly, so the effective wavelength of the $U$ band as it reaches the ground is not where the filter puts it — the same trap that makes a planet’s radius depend on the colour it is measured in, and it moves with airmass as the short-wavelength side is cut away more than the long. That is a colour term that depends on airmass in a way the linear model does not describe.

Two ways the slope is got, and what each assumes #

The Bouguer method is not the only one, and the alternative is worth setting beside it because they fail differently.

One star, many airmasses. The classical method: watch a single star rise or set and fit the line. It assumes the atmosphere is constant in time and gives a slope with a long lever arm, so the slope is well determined — but it takes hours, during which the assumption is being strained.

Many stars, one moment. Observe a field containing stars at a range of airmasses simultaneously, or two standard fields at very different altitudes within a few minutes. This assumes the atmosphere is the same in different directions rather than at different times, and the run of airmass is usually shorter, so the slope is less well determined but the assumption is better.

Which is preferable depends on the night. A stable night favours the first; a night with drifting aerosol favours the second. Modern wide-field surveys use a third approach that is really the second taken to its limit — every exposure contains hundreds of stars with known standard magnitudes, so the extinction, the zero point and the flat field are solved simultaneously across the whole survey by a global fit with millions of parameters, and no single night’s coefficients are ever quoted.

What makes a night photometric #

The whole method rests on the atmosphere holding still, and observers use a word for a night on which it does: photometric. It is worth saying what that word is actually claiming, because it is a stronger claim than “clear”.

A cloudless night is not necessarily photometric. What is required is that the extinction coefficient be constant across the sky and constant in time, to a fraction of a per cent, for hours. Thin high cirrus invisible to the eye reduces the transmission by a per cent or two and moves; a haze layer that forms as the temperature falls changes the aerosol term through the night; and dust transported from a continent away can raise the red extinction by a factor of two while leaving the sky looking perfectly clear.

So the condition is tested rather than assumed, and the tests are all internal to the data. A standard star observed repeatedly at the same airmass should return the same instrumental magnitude every time; a plot of residuals against time should be flat rather than trending; and a star observed both rising and setting should trace a line rather than a loop. Failing any of those means the night’s coefficients cannot be trusted and the data are usable only differentially.

The practical statistics are sobering. At a good site perhaps a third to a half of clear nights meet the criterion, and at a mediocre one far fewer — which is why absolute photometric calibration is done on a small number of carefully chosen nights and everything else is tied to it.

There is a further subtlety in what “the same airmass” buys. Two stars at the same airmass in different directions are looking through different columns of air, and if the aerosol layer is patchy their extinctions differ. That is not detectable from a Bouguer plot of either star, and it is the residual that limits absolute photometry at a good site to about one per cent rather than to the photon noise — the sky is not a function of zenith angle alone, and the entire method assumes it is.

The other extinction #

The word means something else two hundred parsecs further out, and the two are routinely used within a sentence of each other.

The order of operations is not negotiable and is worth stating. An observed magnitude is corrected for atmospheric extinction to give a magnitude above the atmosphere; that is transformed to a standard system; only then is interstellar extinction considered, and it cannot be measured from a single star at all — it needs a colour excess, which needs an intrinsic colour, which needs a spectral type.

The other multiplicative terms #

Extinction is not the only factor standing between a count of electrons and a magnitude, and the others are worth naming because they enter the same way and are removed by different means.

The flat field is the detector’s own response, which varies across the chip by several per cent and is not the same at every wavelength. It is measured by exposing on a uniformly illuminated surface — the twilight sky, or a screen inside the dome — and dividing it out. The difficulty is that neither illumination has the same spectrum as a star, so a flat field taken on the twilight sky corrects a red star and a blue star by slightly different amounts.

The aperture correction accounts for the fact that a star’s light is spread by the atmosphere into a profile with no edge, so any finite measuring aperture misses some of it, and how much depends on the seeing at that moment. Measuring bright stars in a large aperture and faint ones in a small one, then correcting the second to the first, is standard — and the correction changes through the night as the seeing changes.

And the atmospheric transmission is not stable even on a photometric night at the per-mille level, because the water vapour column varies. That matters not at all in $V$ and a great deal in the near infrared, where the bands are placed between water absorption features and their effective transmission depends on how much water is overhead.

The reason for listing them together is that all four — extinction, flat field, aperture, water — are multiplicative factors that a logarithmic scale turns into additive offsets, and all four are estimated from the same data they are applied to. A calibration is therefore a simultaneous solution rather than a sequence of independent corrections, and the covariances between the terms are where the residual error lives.

A per-cent magnitude is not one measurement made carefully; it is half a dozen nuisance parameters solved for at once, and the extrapolation to zero airmass is only the one with the most striking geometry.

The two extinction laws the essay uses can each be read at parameters well outside the standard values, and the difference between them is what a sight line’s dust actually says.

What space changed, and what it did not #

Above the atmosphere the first correction disappears entirely, and with it the largest single systematic in ground-based photometry. That is most of the reason space photometry reaches millimagnitude precision on time series while ground-based work struggles below a per cent.

What does not disappear is the definition problem. A space telescope’s passband is its own, and turning its measurements into a standard system still requires a transformation with a colour term. Gaia’s photometry is calibrated on its own internal system and tied to ground-based standards, so the chain still ends where Johnson and Morgan left it.

The measurement that does not need the extrapolation #

There is a way of avoiding the whole construction, and it is worth stating because it is what most photometry actually does now and because it shows exactly which part of the problem the extrapolation was solving.

If two stars are close together on the sky and observed in the same exposure, their light passes through very nearly the same column of atmosphere. The extinction affects both by the same factor, so the ratio of their brightnesses is uncorrupted, and the difference of their magnitudes is free of the atmosphere without any coefficient having been determined.

That is differential photometry, and its precision is limited by photon noise and by scintillation rather than by the atmosphere’s transmission. It is how every transit light curve is measured, how variable stars are monitored, and how the millimagnitude time series that modern astrophysics runs on are obtained from the ground at all.

What it does not give is a magnitude. It gives a difference, and turning a difference into a number requires the comparison star’s own magnitude — which somebody had to obtain by the method this essay is about, on a photometric night, with an extrapolation.

So the two techniques are not competitors. The extrapolation establishes a small number of anchors and the differential method propagates them, and the accuracy of the anchors and the precision of the propagation are separate quantities that get quoted together and mean different things. A light curve good to a hundred parts per million, tied to a comparison star whose own magnitude is uncertain at the per-cent level, is a perfectly ordinary and perfectly useful object — the shape is measured far better than the level.

The atmosphere is removed either by modelling it or by arranging for it to cancel, and almost everything that requires precision does the second while depending on somebody having once done the first.

What the ladder has established #

The first rung of this anchor was about dust: an absorption that is a property of the space between stars, measured in magnitudes per kiloparsec, and separable from distance only through its colour dependence.

This rung is about a second absorber sitting between the observer and everything, including the dust — one that is thicker, more variable, and removed by a method with a peculiar property: the answer is at a point outside the data. That is not a flaw in the method. It is what an extrapolation is, and the reason it is trustworthy here is that the functional form being extrapolated along is known exactly from the geometry, so the only question is whether the coefficients held still.

The general lesson is worth stating in the site’s own terms. The observation behind the number is not the observation the number describes. A published $V$ magnitude of 12.437 was never measured; five measurements of something else were, and 12.437 is where the line through them crosses an axis.

And the extrapolation itself read over a shorter run of airmass, which is the practical constraint on every observing night.

One more reading shows where the light removed by extinction actually goes.

Where the ladder goes next #

The rung above is the modern replacement for all of this: differential photometry with an ensemble of comparison stars in the same field, in which the extinction is never determined at all because it cancels between the target and the references. That works superbly for variability and not at all for absolute calibration, so the two approaches coexist — and it is why a transiting planet’s depth to a hundred parts per million from the ground while the star’s own magnitude stays uncertain at the per-cent level. The first is a ratio taken at one airmass; the second is an extrapolation to none.