A response measured pixel by pixel

Every published brightness is an extrapolation and every extrapolation starts from a measured count. This essay is about what happens between the photons and the count.

The first thing done to any astronomical image is division by a flat field. The purpose is stated in every observing manual: pixels differ in sensitivity, so divide by an exposure of something uniform and the differences go away.

That description is correct and it describes the smaller half of the problem. Dividing by a flat field removes two things at once, and only one of them is a property of the detector.

The distinction between them is not pedantic. One is a few per cent, uncorrelated, and disappears the moment several pixels are added together. The other is a few tenths of a per cent, smooth, and survives everything. The larger number is the harmless one.

Two things with the same name #

Pixel response is a property of the silicon. This pixel converts photons to electrons two per cent more efficiently than that one, because its gate structure is slightly different or its surface layer slightly thinner. It is uncorrelated from pixel to pixel, it is stable over months, and any uniformly illuminated exposure measures it.

Illumination is a property of everything in front of the detector. The telescope’s optics vignette the edges of the field; the filter’s transmission varies across its own diameter; dust on a window casts a soft shadow; scattered light adds a diffuse background that is not uniform. It is smooth, it varies over the field on scales of the field itself, and — the crucial point — it depends on where the light came from.

That last dependence is why the problem is hard. A dome flat is a screen a few metres away lit by lamps; a twilight flat is the sky at dusk, which is bright, gradient-ridden and polarised; a night-sky flat is the accumulated background of many science exposures. These three illuminate the pupil differently, they scatter differently inside the instrument, and they measure three different illumination patterns. All three measure the same pixel response.

There is also a fourth source, and it is the one that has become standard for the highest-precision work: the sky itself, used not as a flat but as a reference. Observing the same stars at many places in the field and demanding that they give the same magnitude everywhere measures the illumination pattern with the only source guaranteed to be at infinity. That is the technique the last section is about, and mentioning it here is worth doing because it reframes the whole problem: the difficulty is not measuring a response, it is finding something uniform to measure it against, and nothing on the ground is uniform in the way required.

Why the small term is the one everybody removes #

Pixel response is a few per cent and illumination error is a few tenths of a per cent, so the reflex is to worry about the larger number. The reflex is wrong, and the reason is that a photometric measurement is a sum over many pixels.

A photometric aperture contains fifty to a few hundred pixels. Pixel-to-pixel response scatter is uncorrelated, so summing $N$ pixels reduces it by $\sqrt N$: a two per cent per-pixel scatter becomes under three tenths of a per cent in the aperture, and it is random from star to star, so it looks like noise and is treated as noise.

An illumination error does none of that. It is the same for every pixel in the aperture, so summing does not reduce it. It is the same for every star in that part of the field, so it does not average over stars. And it varies smoothly with position, so it looks like a real gradient in the sky rather than like an error.

There is an exception worth recording, because it is where pixel response does matter. Undersampled data — where the point spread function is comparable to a pixel — puts most of a star’s light into a handful of pixels, so the averaging that saves aperture photometry does not happen, and the response scatter enters at nearly its full per-pixel value. That is the regime of a space telescope with a small mirror and large pixels, and it is why such missions dither obsessively: not to average the illumination, which is stable, but to average the pixel response, which for them is not the small term at all.

What a residual gradient is confused with #

A smooth magnitude offset that depends on position in the field masquerades as several genuinely interesting things, and the list is worth having.

A colour term. If the illumination error differs between filters — and it does, because vignetting is achromatic and filter transmission is not — the residual is a position-dependent colour, which is indistinguishable from a real colour gradient across a cluster or a galaxy.

Extinction structure. A radial gradient in a field containing a dark cloud looks like a gradient in the reddening, and reddening is how dust is measured.

A zero-point drift. Fields observed at different pointings have their stars at different field positions, so their zero points differ, and the difference varies from night to night with whatever changed in the optics.

A stellar population gradient. In a crowded field — a globular cluster, a nearby galaxy — the stars near the field centre and the stars near its edge are at different physical radii, and a photometric gradient becomes a population gradient with a plausible astrophysical story attached.

A transit depth. The effect reaches time-series work too. A transit is measured as a ratio to comparison stars, and if the telescope drifts during the night then the target and the comparisons move across the illumination pattern by different amounts, producing a smooth trend in the ratio over exactly the timescale a transit lasts. This is the single most common source of spurious shallow transits from the ground, and the standard defence — keeping the star on the same pixels all night — is a defence against this and nothing else.

How it is actually solved #

Three techniques, in increasing order of both effort and reliability.

Dithering. Observe the same field at several telescope positions, so each star lands on several parts of the detector. This averages the illumination error over the positions used, and it reduces the residual by roughly the square root of their number — which is a real gain and not a solution.

A star-flat. Observe a dense star field at many offsets and require that each star’s measured magnitude be the same wherever it lands. That is a fit for the illumination pattern using the sky itself as the uniform source, and it measures exactly the quantity wanted rather than a proxy for it. It costs a night.

Self-calibration, or ubercalibration. Extend the same idea to a whole survey. Every star observed more than once, at different positions and on different nights, contributes an equation; the unknowns are the stars’ magnitudes, the nightly zero points and the illumination pattern; and the system is solved globally. The overlaps between fields are what make it determined.

The last of these is what modern wide-field surveys do, and its effect was dramatic: applied retrospectively to a survey already considered well calibrated, it removed systematic patterns of a few per cent across the sky and brought the photometric consistency from about two per cent to a few millimagnitudes.

And there is a fourth technique that is not a correction at all and is often the right answer: give up on absolute photometry and work differentially. If the quantity wanted is a change in a star’s brightness rather than its brightness, then comparing it against stars in the same part of the field cancels the illumination pattern exactly, because they share it. That is why differential photometry reaches a floor thousands of times below the absolute calibration, and it is a reminder that the right response to a systematic is sometimes to design the measurement so that it cancels rather than to measure it.

What was actually measured #

Three results establish the size of the effect and each was obtained differently.

The star-flat measurement. Observing a dense field at dozens of dither positions and requiring internal consistency measures the illumination pattern directly. For typical wide-field imagers it comes out at two to five per cent from centre to corner, and it disagrees with the dome flat by that amount — which is the measurement that the dome flat is wrong rather than an argument that it might be.

The survey self-calibration. A large imaging survey re-reduced with a global solution showed residual patterns correlated with position in the focal plane, with amplitude around one per cent and a shape that repeats field to field. The pattern is a property of the instrument and it was invisible in any single field.

The disagreement between flat-fielding methods. Comparing dome, twilight and night-sky flats from the same instrument gives three illumination patterns differing by a few per cent, in a way that is reproducible and that correlates with the angular size of the source used. That is the direct demonstration that what is being measured depends on where the light came from.

The consequence for a survey is best seen at the faint end, where the same illumination error that shifts a bright star’s magnitude decides whether a faint one is catalogued at all.

Where the picture stops #

The picture stops in three places, and the second is the one that has grown in importance.

Scattered light is not multiplicative. Everything above treats the illumination as a factor that multiplies the true image. Scattered light adds instead, and an additive term divided by a flat is not corrected — it is spread around. Distinguishing the two requires measuring the instrument’s scattering directly, usually by imaging a very bright star well outside the field and mapping what arrives inside it.

The detector is not a simple linear device. Modern CCDs show brighter-fatter effects, in which accumulated charge deflects later photoelectrons and makes bright stars broader than faint ones; tree rings and edge distortions, in which the pixels are not the same size; and non-linearity near full well. None of these is a response error in the sense a flat field corrects, and dividing by a flat treats all of them as though they were.

And the sky is not uniform either. A night-sky flat assumes the background is flat, and airglow has real structure at the per cent level on scales of degrees, varying through the night. That structure is imprinted on the flat and then divided out of the data, putting a real sky gradient into every science image as a spurious instrumental one.

And a fourth, and it is about time rather than position. Everything above treats the illumination pattern as fixed. It is not: dust accumulates on the corrector plate, the filter’s position in its wheel repeats to a fraction of a millimetre rather than exactly, the telescope flexes with elevation, and the whole optical train is realigned after every mirror recoating. So a pattern measured in one semester is not the pattern in the next, and a survey spanning years has to solve for it repeatedly. The self-calibration approach handles that naturally, because it fits the pattern from the data themselves; a stored flat field does not, and the standard failure mode of a long imaging programme is a calibration that was correct when it was measured.

There is also the awkward case of a filter that is not in a collimated beam. In many instruments the filter sits in a converging beam, so a photon’s path through it depends on where in the field it is going, and the effective bandpass therefore varies across the field. That is not an illumination error at all — it is a different photometric system in different parts of the detector — and no flat field of any kind corrects it. It shows up as a position-dependent colour term, and it has to be modelled as one.

Why this belongs to photometric systems #

The reason to file this argument with what a magnitude has to say about which light rather than with detectors is that the failure is a failure of definition rather than of hardware.

A photometric system is a claim that a magnitude means the same thing everywhere: the same number for the same star, whichever telescope, whichever night, wherever in the field. Everything in this essay is an obstacle to that claim, and the obstacles are not in the detector — they are in the fact that a telescope’s response depends on where in its field a photon lands, and that measuring that dependence requires a uniform source that does not exist.

The eventual solution is characteristic of the subject and worth noticing. Nobody found a uniform source. What happened instead was that the sky itself was used: observe the same stars in many places, demand that they agree, and solve for whatever pattern is needed to make them agree. That converts an instrumental calibration into an over-determined system of equations, and it works because the sky is full of objects that do not change.

The same manoeuvre appears wherever a calibration is hard: rather than measure the instrument, measure something the instrument should report consistently and solve for the inconsistency. It requires redundancy, it requires the redundancy to be designed in advance, and it is the reason a modern survey’s observing strategy is a calibration strategy first and a sky-coverage strategy second.

One observation to end on about how such errors are found, because it generalises. Nothing in a single well-calibrated image reveals an illumination error: the image looks right, the stars have plausible magnitudes, and the internal scatter is what the photon noise predicts. The error becomes visible only in comparisons — the same star at two field positions, the same field on two nights, the same sky in two overlapping surveys. Every one of the diagnostics above is a comparison, and every one requires that the observations have been arranged so that the comparison is possible.

That is the practical lesson worth carrying, and it is not about flat fields. A systematic that is constant within a dataset is invisible inside it, and the only way to see it is redundancy that was designed in before the data were taken. A magnitude extrapolated to zero airmass is measurable because observations were deliberately taken at several airmasses; an illumination pattern is measurable because stars were deliberately put in several places. Neither would be recoverable from a perfectly efficient observing programme that never repeated anything.

There is a diagnostic worth naming because it distinguishes the two quantities in practice rather than in principle. A pixel-response error is uncorrelated between neighbouring pixels and therefore between two exposures in which the star lands on different pixels; an illumination error is smooth, so it is nearly identical for the two. Observing the same star at many positions across the field and looking at how the measured magnitude varies therefore separates them immediately: the scatter between nearby positions is the pixel term and the trend across the field is the illumination term. That measurement is what a dithered survey performs automatically as a by-product of its own observing pattern, which is why the surveys with the best large-scale photometric uniformity are the ones that dithered for reasons of coverage rather than of calibration. The calibration came free with a decision made about something else, which is an unusually lucky arrangement and worth recognising when designing the next one.

Where the ladder goes next #

The natural next rung is the additive half: scattered light, its measurement, and why an additive error survives a multiplicative correction. The rung after it is the global solution itself — how a survey’s magnitudes, zero points and illumination pattern are fitted together, and what has to be true of the observing pattern for the system to have a unique answer.