NeXus/HDF5/CBF Integration

This site contain materials relating to the proposed integration of NeXus, HDF5 and CBF.

A good starting point would be the presentation foils

The software supporting this project is being gathered at the cbflib.sf.net web site

Here is the most recent report:

Dear Colleagues,

Starting in 2010 a collaborative effort has been under way to achieve inter-operation of CBF (the Crystallographic Binary Format, see IUCr International Tables for Crystallography, Volume G, chapters 2.3, 3.7, 4.6, 5.6) and NeXus ("A common data format for neutron, x-ray and muon science", see http://www.nexusformat.org/).

The most recent formal report on the effort was presented as a poster at the meeting of the American Crystallographic Association in May 2014, see

https://sites.google.com/site/nexuscbf/home/presentation-foils/Integration_Poster_10May14_w_overlays.pdf

and the general state of the effort is updated on

https://sites.google.com/site/nexuscbf

The work discussed on this site is the result of contributions by many people, including the IUCr Committee of the Maintenance of the CIF Standard (COMCIFS, http://www.iucr.org/resources/cif/comcifs), chaired by James Hester, the NeXus International Advisory Committee (NIAC, http://wiki.nexusformat.org/NIAC), chaired by Mark Koennecke, Graeme Winter, Tobias Richter and Jonathan Sloan of Diamond Light Source (DLS), Nicholas Sauter and Aaron Brewster of Lawrence Berkeley National Laboratory. Our thanks for years of supporting efforts at the Brookhaven National Laboratory PXRR Group: Robert M. Sweet, Dieter Schneider, Howard Robinson, John Skinner, Matt Cowan, Leonid Flaks, Richard Buono; at DLS: Alun Ashton, Bill Pulford; and at the Dowling College ARCiB Lab Group: Mojgan Asadi, Kostandina Bardhi, Keti Bardhi, Limone Rosa, Our thanks to DECTRIS, BIOIHDF and the HDF Group. Our thanks to Frances C. Bernstein. Over the years the work has been supported in part by funds from NIGMS, DOE, NSF, PaNdata ODI (EU 7th Framework Programme).

As noted in the poster:

The new generation of high performance x-ray detectors requires integration of HDF5, NeXus and CBF.

    • The DECTRIS workshop in Baden, Switzerland in January 2013 established the parameters of the integration.
    • NIAC and COMCIFS are working together to ensure interoperability.
    • Use of compressions helps to control storage volumes.
    • Use of HDF5 helps to reduce high file-count burdens on facility file systems.
    • CBFlib 0.9.5
    • Can store arbitrary CBF files in HDF5 and recover them.
    • Supports the NeXus NXmx Application Definition for single crystal MX data.
    • Supports use of all CBFlib compressions in HDF5 files.
    • Provides minicbf2nexus to convert sets of minicbf files to a single NeXus file.
    • Provides cbf2nexus to convert a CBF file describing a single scan to a single NeXus file containing the same data.
    • Provides nexus2cbf to convert back from a NeXus file to a CBF file.
    • A simplified functional mapping for single crystal MX has been prepared.
    • A full mapping extending the functional mapping to the general case is being finished.
    • Updated CBF dictionary has been prepared.
    • There is much work still to be done – collaborators welcome.

This is an interim report on the work done in June and July 2014.

The effort up to this point has resulted in reliable inter-operation for files of images from detectors that produce single rectangular images, such as the Pilatus 6M. Now we are working to extend the mapping to support multi-module detectors in which each frame is composed of multiple images, each with its own geometric description. Examples of such detectors are the CSPAD FEL detectors and the semi-cylindrical DECTRIS Pilatus 12M-DLS. The DECTRIS Eiger NeXus/HDF5 format introduced some support for such detector images in HDF5. CBF supports such images in CIF with what it calls "array sections" and "detector elements". NeXus supports such images with the "NXdetector_module" group, based on the DECTRIS Eiger approach. Full general support for more complex multi-module detectors in NeXus requires use of a more general naming scheme for groups and fields than what we used for inter-operation for single module detectors.

The code in CBFlib has now been adapted to support the more general naming scheme in the mapping from CBF to NeXus and that code has been uploaded to the CBFlib_bleeding_edge module in the SourceForge cbflib svn.

The necessary changes for the NeXus to CBF mapping have been started. Some extensions and changes to the NeXus NXmx application definition to clearly support the very large number of axes with unique names and the use of more than 2-dimensional image arrays with the CSPAD will be necessary. When that direction is complete, a CBFlib 0.9.6 release kit will be posted on SourceForge and this interim report will be replaced with a more detailed one.

-- Herbert J. Bernstein, Dowling College