ACE Level 2 Data Implementation in HDF






Some Links to Background Information


General Organization of ACE SIS Level 2 Data

All ACE data available to the scientific community (Browse, Level 1, Level 2, ancillary) are formatted using HDF. Here, we attempt to describe how ACE Level 2 data are implemented in HDF, using the data from the SIS instrument as an example.

The Solar Isotope Spectrometer (SIS) is designed to provide high resolution measurements of the isotopic composition of energetic nuclei from He to Ni (Z=2 to 28) over the energy range from ~10 to ~100 MeV/nucleon. For more information about the SIS instrument, visit the SIS Home Page, at Caltech.

ACE Level 2 data are suitable for serious scientific study and are made available to the scientific community for that purpose. The data are organized by instrument (there are 9 science instruments aboard the spacecraft), and by time-averaging periods. Data from each instrument are stored in separate HDF data files, and separate HDF files also contain the data from the different averaging periods. For SIS, the averaging periods are

The hourly, daily and 27-day averaging periods are common to the other ACE instruments, while the 256-second average is the natural instrument cycle for SIS.

All level 2 data records for all ACE instruments contain timing information in the same format. The format of the timing information is described here.


Implementation of SIS Level 2 Data in HDF Format

In cooperation with the SIS team, a subset of the full SIS dataset was identified as being appropriate for inclusion in the SIS Level 2 data. This subset consists of the fluxes of 9 elements, in 8 energy bands. The energy bands are different for each element. The SIS team delivers these fluxes in 256-second averages in ASCII text format to the ASC, and the ASC performs the longer-term time averaging and the HDF formatting.

The first step in implementing these data in HDF is to define a SIS Level 2 data record. The ASC uses the C programming language, so this record is defined by a C data structure. The data structure for the SIS hourly averages is shown here:

struct SIS_data_1hr {

  /* UT time at the start of the averaging period */
  int32   year;                         /* integer year */
  int32   day;                          /* integer day of year */
  int32   hr;                           /* hour of day */
  int32   min;                          /* min of hour */
  float32 sec;                          /* seconds */
  float64 fp_year;                      /* floating point year */
  float64 fp_doy;                       /* floating point Day of Year */ 
  float64 ACEepoch;                     /* UT time in sec since 00:00:00 1/1/96 */

  float32  flux_He[NUMBER_OF_ENERGY_BANDS]; /* He flux in eight energy ranges,
  					     * particles/(cm2 Sr sec MeV/nucleon) */

  float32  flux_C[NUMBER_OF_ENERGY_BANDS];  /* Carbon flux */
  float32  flux_N[NUMBER_OF_ENERGY_BANDS];
  float32  flux_O[NUMBER_OF_ENERGY_BANDS];
  float32  flux_Ne[NUMBER_OF_ENERGY_BANDS];
  float32  flux_Mg[NUMBER_OF_ENERGY_BANDS];
  float32  flux_Si[NUMBER_OF_ENERGY_BANDS];
  float32  flux_S[NUMBER_OF_ENERGY_BANDS];
  float32  flux_Fe[NUMBER_OF_ENERGY_BANDS];

  float32  cnt_He[NUMBER_OF_ENERGY_BANDS]; /* He counts in eight energy ranges,to be used
  					    * for computing statistical uncertainties in the fluxes */
  float32  cnt_C[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_N[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_O[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_Ne[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_Mg[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_Si[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_S[NUMBER_OF_ENERGY_BANDS];
  float32  cnt_Fe[NUMBER_OF_ENERGY_BANDS];

  int32   solar_activity_flag;  /* = 0 less than threshold    */
                                /* = 1 greater than threshold */
};
The data structures for the other time-averages are exactly the same, except for the name of the data structure.

Implementing SIS Level 2 Data in HDF

The details of reading and writing the data from/to HDF files are rather cumbersome. Many calls to the HDF API are required to initialize the HDF file, and to read/write the individual elements of the data structure.

To ease the pain, the ASC has developed a perl program - hdfgen - which takes a C-structure as input and generates all the HDF API calls needed to read and write the entire C structure. The user interacts with the data at the level of read_Cstruct and write_Cstruct instead of the individual data element level that the HDF API requires. Use of hdfgen helps to reduce software development time and minimize coding errors. Read all about this nifty perl program here.

Skipping the details, the hdfgen routines create two basic HDF building blocks - Vgroups and Vdatas. Vgroups are generic grouping elements allowing a user to associate related objects within an HDF file. As Vgroups can contain other Vgroups, it is possible to build a hierarchical file. Vdatas are generic list objects. Data is organized into "fields" within each Vdata. Each field is identified by a unique "field name". The type of each field may be any of the basic number types that HDF supports. Fields of different types may exist within the same Vdata. hdfgen can also handle SDS objects, but we don't use them for ACE level 2 data.

For SIS Level 2 hourly-average data, the hdfgen routines create a Vgroup labelled "VG_SIS_data_1hr". Within the Vgroup, a Vdata called "SIS_data_1hr" is defined whose fields correspond to the elements of the C data structure listed above. Data processing programs write data to the HDF files by repeatedly filling up the C data structure and then calling a hdfgen-created routine to write a new record to the Vdata.

A second Vdata is also defined for SIS, in which we record the energy ranges, in MeV/nucleon, for each element (the energy ranges are different for each element). Of course, we only write one record to this Vdata.

A third Vdata is also defined for SIS, which is only used by our web-based data browser. This Vdata records a mapping between Year/Day_of_year and record number in the SIS_data_1hr Vdata. This information enables us to give users the option to subset data based on Date rather than record number. This Vdata contains one record per day, so a user can't subset the data into chunks smaller than one day (unless she resorts to using record numbers to perform the subsetting).


SIS Level 2 Metadata (Annotations and Attributes)

We use HDF File Annotations and Object Annotations to record metadata within HDF files. Both kinds of annotations are basically containers into which one can dump ASCII-formatted information.

The hdfgen routines automatically add an object annotation whenever a Vdata or an SDS is defined in an HDF file. The object annotation containes a copy of the C data structure used as input by hdfgen, and is added to the Vgroup containing the Vdata (or SDS).

File annotations are used to record global information about the dataset. At least three file annotations are added to each ACE Level 2 data file:

HDF also provides Attributes, which can be set for either a Vdata, or for any of the fields within a Vdata. In the future, the ASC may utilize HDF attributes to record information such as the physical units associated with the data contained within a Vdata field, etc.


Email Address: ad@srl.caltech.edu

Last Updated: 23 November, 1999

Return to ASC Home Page