Skip to content

Latest commit

 

History

History
281 lines (243 loc) · 16.9 KB

README.md

File metadata and controls

281 lines (243 loc) · 16.9 KB

CI Coverage License

eiger-correction

What Where
Source https://github.com/DiamondLightSource/eiger-correction
Install pip install git+https://github.com/DiamondLightSource/eiger-correction.git
Docker docker run ghcr.io/diamondlightsource/eiger-correction:latest
Releases https://github.com/DiamondLightSource/eiger-correction/releases

A headless workflow that corrects polarisation and detector-response artefacts in EIGER2 CdTe 500K scans. The eiger-correction command is defined in __main__.py. The extraction stages are in extract_eiger_corrections.py, scientific routines in eiger_workflow.py, JPEG generation in eiger_diagnostics.py and the Markdown report in eiger_report.py. No Jupyter installation is needed.

Quick start

eiger-correction /dls/i15-1/data/2026/cm44163-4/i15-1-100605.nxs --output-dir /some/output/directory

This writes the following to the output directory:

detector_corrections.h5   # detector calibration
pattern.h5                # final integrated pattern
pattern_tth.xy            # two-column 2theta (degrees) / intensity
pattern_q.xy              # two-column Q (angstrom^-1) / intensity
angle_grouping.jpg
azimuth_geometry.jpg
detector_response.jpg
final_xy.jpg
overlap_final_abcd.jpg
position_gains.jpg
sampling_q.jpg
report.json               # machine-readable run record
report.md                 # human-readable report with embedded JPEGs
run.log                   # full process log

See Outputs and unattended operation for details.

Interactive notebook

notebooks/extract_eiger_corrections.ipynb runs the same two-stage extraction step by step with the package functions, showing every intermediate printout and plot inline. Settings live in the cell tagged parameters, so it can also be run headless with papermill:

papermill notebooks/extract_eiger_corrections.ipynb run.ipynb \
    -p SCAN_PATH /dls/i15-1/data/2026/cm44163-4/i15-1-100605.nxs \
    -p OUTPUT_DIRECTORY /some/output/directory

It writes calibration and dual-grid pattern HDF5 files with WRITE_CALIBRATION=True and separate _tth.xy / _q.xy files with WRITE_XY=True, to OUTPUT_DIRECTORY (default ./outputs). run.ipynb keeps all the figures. Notebook parameters FIT_Q_RANGE, Q_STEP_SIZE, TWO_THETA_STEP_SIZE and Q_MAX use the same units and final-only cutoff semantics as the CLI. Overlap diagnostics remain full-range.

Environment and invocation

Python 3.11 or newer is required. Install the package and its dependencies with uv sync (or pip install .). The dev container does this automatically. Then run eiger-correction SCAN (or python -m eiger_correction SCAN) with the source NeXus path. By default, the goniometer calibration is decoded directly from the JSON blob at /entry/plan_metadata/geometry_calibration in that scan. An explicit --geometry GEOMETRY_JSON overrides the embedded geometry; an unreadable or invalid override is an error, never a silent fallback. Missing/invalid embedded JSON requires an explicit file. Geometry compatibility checks still apply. The selected source and checksum are recorded in the JSON report and the source is shown in the Markdown overview; exported HDF5 retains the selected geometry. --help describes every option and --version prints the installed version. --output-dir DIRECTORY chooses a new run directory; otherwise a unique UTC timestamped directory is created under outputs. Existing directories are rejected, even if empty: outputs are never overwritten and simultaneous jobs cannot collide.

For example, the real-data smoke test used scan 100605 from visit cm44163-4, the bundled October 15 geometry (src/eiger_correction/data/), and --raw-audit. All source inputs are opened read-only. Companion detector (STEM-fastcs_eiger_000001.h5), I0 (STEM-i0.h5) and metadata (STEM-fastcs_eiger_meta.h5) filenames are inferred from the NeXus stem; --detector, --monitor and --metadata override them. If the detector file is absent, frames are read from entry/fastcs_eiger/data in the NeXus file itself.

Defaults matching the latest extraction notebook

Setting Default CLI control
Azimuth Single inclusive pyFAI chi wedge 67.5–112.5° --azimuth-range LOW HIGH; --all-azimuths disables
Detector-fit Q interval 10–40 Å⁻¹ --fit-q-range LOW HIGH
Fit and final Q target step 0.025 Å⁻¹ --q-step-size
Fit iterations 10 --iterations
Processing and final 2theta target step 0.025° --two-theta-step-size
Final Q range 0 to NeXus Q maximum, or unmasked pyFAI maximum --q-max (Å⁻¹)
Angle grouping tolerance 0.001° --angle-tolerance; zero means exact repeats
Raw saturation threshold 10⁷ --saturation
Overlap support 100 pixel weight, 20 shared bins --overlap-min-pixels, --overlap-min-bins
Overlap fractional floor 0.003 --overlap-floor
Diagnostic scattering interval 30–80° --diagnostic-range LOW HIGH
Independent validation One held-out detector fold --validation basic|holdout|full
Polarization factor 0.95 --polarization
CdTe density 6.2 g/cm³ --density
JPEG resolution 140 dpi --jpeg-dpi

--q-max sets the final maximum Q in Å⁻¹; the corresponding 2theta maximum is calculated using the scan wavelength: 2theta = 2 asin(Q * wavelength / (4*pi)). Omitting it (Python None) reads /entry/plan_metadata/experiment_definition/data/q_max in the input NeXus file, also in Å⁻¹. Missing or empty metadata (null dataset, empty array or blank string) falls back to the maximum finite unmasked pyFAI pixel-centre angle across the scan, including the per-position azimuth selection. Nonempty invalid metadata fails clearly; an explicit --q-max takes precedence. The chosen limit and its source are recorded in the reports. This limit affects only the final output integrations, never detector fitting or overlap scales. Restrictive azimuth wedges can still disconnect the overlap graph: this is a hard failure, with no silent threshold relaxation or fallback. Chi bounds are inclusive; seam-crossing bounds such as 170, −170 are supported. The opposite wedge is not implicitly included.

Bin counts are derived as ceil((upper - lower) / step), retaining exact range endpoints. Uniform bin widths are therefore at most the requested step, apart from pyFAI's small floating-point expansion of integration limits. The fit grid must contain 16–65535 bins. The former --fit-bins and --theta-bins options are replaced by the step-size options; --theta-max is replaced by --q-max.

Outputs and unattended operation

Both HDF5 exports include NeXus NXentry/NXdata, root/entry default, signal and axes tags for automatic plotting in readers such as DAWN. The pattern defaults to intensity versus 2theta (degrees), with conditional Poisson sigma linked as its uncertainties. The corrections file defaults to the combined pixel × sensor divisor image, with detector column/row pixel-index coordinates. Legacy dataset-level plot tags are included for older readers. Existing dataset paths remain valid; additional hard links do not copy arrays.

Every run directory contains:

  • Calibration HDF5 (detector_corrections.h5): detector/pixel/sensor divisors, fixed masks, geometry, physical model, fit history and settings, held-out results, final source-only position gains, and separately labelled stage-1 nuisance gains.
  • Pattern HDF5 (pattern.h5): independently integrated final 2theta (degrees) and Q (Å⁻¹) grids in entry/merged_2theta and entry/merged_q (q_angstrom axis), intensity, conditional Poisson sigma, signal/normalization/variance/pixel-weight accumulators per position and merged, fit-range flags, angles, I0, gains and applied corrections.
  • Two-column XY (pattern_tth.xy, pattern_q.xy): 2theta degrees or Q (Å⁻¹) and intensity, ten significant digits; unsupported bins are omitted, not filled or interpolated.
  • Seven JPEGs: actual angle grouping; geometry-only azimuth masks; response maps/sensor gains/monitors; Q sampling; stage-1/final gains; full/zoom intensity; final overlap panels A–B after final applied gains, using the full processing range, independent of the output crop. Unsupported intervals are recorded as skips and warnings rather than invalidating an otherwise supported extraction. The overlap figure notes that individual curves may spike where few pixels contribute; A retains all supported bins, while B uses the diagnostic pixel-weight threshold. The historical JPEG filename is retained for existing links.
  • JSON run report (report.json): status, inputs, software versions and source/geometry hashes, settings, per-stage timing, validation metrics, fit history, all gains, graph edges/support, residual summaries, output names, warnings and exceptions. It is atomically checkpointed before/after each stage, not only on success.
  • Markdown run report (report.md): a human-readable companion to the same JSON snapshot, including a run overview, warnings/failures, validation and timing tables, processing settings, product links and an embedded, labelled JPEG gallery. Open the Markdown preview in VS Code to see the images; the relative links remain valid when the complete run directory is moved. Both reports are updated at every checkpoint, including failed and interrupted runs. Detailed arrays and graph edges remain in JSON rather than overwhelming the overview.
  • Text run log (run.log): Python stdout/stderr, fit progress, logging and failure tracebacks. Python warnings also appear in the report. No successful processing step depends on reading terminal stdout.

The report status is running, completed, completed_with_warnings, failed, or interrupted. Exit status is zero for completed runs (including warnings), nonzero for failure/interruption. A non-improving held-out metric is a warning, not an invented acceptance threshold: automation should inspect the report's warnings as well as the exit code. Argument-parser errors and inability to create the run directory necessarily occur before a report can be created. A forced kill/power loss leaves the last checkpoint marked running; it cannot finalize a report or guarantee scratch cleanup. Normal exceptions and keyboard interruption retain the traceback and clean up temporary grouped detector data. Failed runs can contain partial products; only accept a completed report.

Method and speed choices

  1. Stream raw counts once into summed angle groups. Groups are anchored at their minimum motor reading (not chained). Geometry uses the arithmetic mean of motor readings, one equal contribution per raw frame, never I0-weighted. I0 is summed. Any nonfinite/negative/saturated member rejects that group pixel.
  2. Verify geometry/physical normalization, then build a sparse fit-only cache. Stage 1 estimates a common log-Q profile, pixel/sensor response and nuisance position scales, using the existing robust/shrunk fit unchanged.
  3. Freeze the detector map and integrate each group once in 2theta at unity position scales. The same read supplies integer pixel-centre sampling counts. These counts are not photons or pyFAI's fractional pixel-splitting weights.
  4. Solve one positive scalar per position jointly from same-bin full-range overlaps. The graph must connect every position; the equal-position geometric mean scale is one. Final scales replace, not multiply, nuisance scales. Apply them to normalization accumulators only and rebuild the merge.
  5. Reintegrate the requested final range on independent uniform Q and 2theta grids, apply the frozen gains, export, read back/check outputs, and render full-range JPEGs.

Removed notebook-only work: the low-angle raw-image mask preview, full raw-data reread for sampling plots, separate-versus-summed exploratory integrations, ungrouped benchmark and duplicate grouping preview algorithm. Residual diagnostics are reused rather than recomputed for the stack. One raw source pass is necessary for aggregation; later fitting/validation and integration read the much smaller temporary grouped file. --raw-audit explicitly adds a small independent source reread. No smoothing, tapering, background subtraction, additive offset, Q-dependent scale or new fit model was introduced.

Validation strategy

Every run (--validation basic or stronger): check input shape/metadata, positive I0, complete grouping/mean-angle/summed-I0 identities, vertical geometry, constant-pattern physical normalization, fit support, connected overlap graph, positive finite gains, mean-log gauge, invariant signal/variance/pixel counts, correct normalization-only scale application, accumulator conservation, and intensity/sigma identities. Calibration and pattern HDF5 are reopened and checked; XY is read back against the integrated data. Unsupported bins are reported.

Default --validation holdout: one additional detector fit trains on alternating contributing groups, holding the others out entirely. Compare physical-only versus detector-corrected held-out overlap (absolute and centered) and higher-Q RMS. No gain is inferred on the test data, avoiding that extra diagnostic fit/integration. Half-data fits use four observations per pixel, versus eight for the full fit; this never weakens the full-fit support requirement.

--validation full: reverse the train/test roles for a second fold, retain the notebook's lower-Q-inferred gain / higher-Q check, and report fold-to-fold sensor log-gain RMS. This costs two extra fits rather than the default one.

--raw-audit: independently reconstruct the highest-position group from raw frames, reject invalid constituents and reintegrate it with the final gain. Compare all four accumulators against scalar application (3 × 10⁻⁶ relative tolerance). Useful after geometry, masking or integration-code changes; normally unnecessary for every scan after acceptance testing.

Interpretation: held-out tests probe detector generalization within this scan, not transfer to another scan. Stage-2 overlap residuals are in-sample descriptive diagnostics, not independent validation or calibrated uncertainties. Recommended further acceptance testing is independent scans or a known reference and, if stage-2 generalization must be quantified, a separately designed disjoint-overlap-bin evaluation accounting for pixel-splitting covariance. No arbitrary scientific pass/fail thresholds are imposed here.

Scope and tests

This supports the documented EIGER2 CdTe 500K (512 × 1028) and matching energy, wavelength, thickness, material, orientation and geometry convention. CdTe absorption is a modeled Beer–Lambert correction, not measured photon-counting QE. The azimuth wedge reduces azimuth mixing but does not model beam/sample anisotropy. Response use outside the fit-Q interval is extrapolation. Poisson sigma omits gain, calibration, monitor and physical-model uncertainty and neighboring-bin covariance. Treat the method as exploratory until independently validated.

Run the synthetic test suite with pytest; it does not require beamline data. tox -p runs the tests together with pre-commit (ruff lint/format) and pyright. The real scan-100605 smoke test (2026-11-10; Python 3.14.6, pyFAI 2026.5.0) completed in 89.6 s with held-out validation and raw audit: 3608 raw frames → 41 groups, 334 graph edges, all 2001 bins supported, all JPEGs, no reported warnings. Absolute held-out overlap changed from 1.054% to 0.247%. These are a single-run observation, not a speed guarantee or scientific certification.

The Q-output smoke check (2026-11-12, pyfai_new) on the same scan completed with --q-max 20, default held-out validation and --raw-audit: all 41 positions remained connected, the raw audit passed, and both XY files matched their HDF5 grids. A second run without a cutoff read NeXus q_max=25 Å⁻¹. Detector divisors, bad-pixel masks and final position gains were bit-identical between the two runs. The processing range stayed 0–87.133° while the 20 Å⁻¹ output stopped at 29.820°.

Raw data and generated products are not versioned. Share them separately subject to institutional policy; do not commit generated detector data or results.