Description of CRIS Level-One Processing The purpose of this document is to describe the processing that occurs on the CRIS data during the level-one processing at the ACE Science Center (ASC). I. Overview of ASC Instrument Level-One Processing The data that is the starting point of the level-one processing has the following characteristics: 1. The data consists of a single stream in chronological (S/C clock) order. The raw S/C data can reach the Mission Operation Center (MOC) and eventually the ASC by several paths. The level-zero processing merges the different paths into a single stream and inserts fill data in places where there is missing data. 2. Each S/C minor frame has associated with it a one-bit quality flag. The quality indicates whether the level-zero processing encountered uncorrectable errors in processing the data for that minor frame. If uncorrectable errors are encountered, the level-zero processing removes the minor frame and inserts fill data. 3. The data stream is expected to contain few dropouts, and the dropouts that occur are expected to cluster during time intervals when receiving conditions are poor. The level-one processing of instrument data is performed inside a loop that calls four C functions per instrument for each S/C minor frame. The four functions associated with CRIS are 1. accum_cris_level1, 2. accum_cris_browse, 3. output_cris_level1, 4. output_cris_browse. From the names, it is obvious that accum_cris_level1 and output_cris_level1 are associated with the generation and output of CRIS level-one data. The other functions are associated with the generation and output of browse data and will not be discussed in this document. The function, output_cris_level1, directly controls the structure and content of the CRIS data that is written to the level-one data set which will be distributed to the instrument teams. It has only indirect control of the format of the data written to the CDROM. The format is controlled by the eight functions 1. write_cris_hskp_rate, 2. write_cris_events, 3. write_cris_diagnostic_event, 4. write_cris_command_echo, 5. write_cris_command_table, 6. write_cris_incomplete_event, 7. write_cris_unprocessed_minor_frame, 8. write_cris_minor_frame_summary. Each of the above write functions is written by a PERL script that is furnished by the ASC. The PERL script takes as input the definition of a C data structure and generates a write function with embedded HDF calls. When that PERL-generated write function is called it writes the data to the level-one data set. The ASC will furnish the inverse read functions that will allow the user to recover the original C data structure from the level-one HDF data set. In addition to the instrument data, the level-one product includes S/C housekeeping data. The ASC will furnish functions that given the level-one S/C housekeeping structure, the functions will return the attitude and postion of the S/C. Also the ASC will furnish a function that converts the S/C clock to utc. II. Overview of CRIS Telemtry CRIS operates on a 256 instrument minor frame cycle. During each instrument cycle, CRIS has a sync bit which remains low for the first 128 instrument minor frames and high for the next 128 instrument minor frames. During each instrument cycle, CRIS samples each housekeeping parameter once, each low priority rate once, and each high priority rate sixteen times. CRIS during its normal processing, outputs a collection of compressed events during the instrument cycle. In general, the compressed events will span instrument minor frame boundaries, but during periods of normal processing will not span instrument cycle boundaries. CRIS contains an 844 byte command table which defines the instrument status. During each instrument cycle, 66 bytes of the command table is readout as part of the housekeeping data. It takes about 13 instrument cycles to readout the entire table. When CRIS receives a command, it puts a command response in its telemetry. In general, command response spans minor frames and can span cycle boundaries. Finally, CRIS can be put into a diagnostic mode. During diagnostic mode, CRIS suspends putting compressed events into its telemetry stream, and substitutes diagnostic event data. The housekeeping/rate and command responses are uneffected. The most important point from the above discussion is that CRIS produces five different types of data: housekeeping/rate, compressed event, command table, command responses, and diagnostic event. The CRIS level one processing reflects this data organization. III. S/C Minor Frames and Instrument Minor Frames The ACE S/C produces one minor frame of data per second. Within each S/C minor frame there is a 58 byte allocation for CRIS data. CRIS produces one 58 byte instrument minor frame per second. CRIS uses its 58 byte allocation in the S/C minor frame to output the last four bytes of the previous instrument minor frame and the first 54 bytes of the current instrument minor frame. The ACE S/C maintains a major frame counter and a minor frame counter. The S/C housekeeping data is tied to these counters, and most instruments' telemetry are also tied to the S/C minor and major frame counters. CRIS does not use these counters. IV. Overview of CRIS Level-One Processing The CRIS level-one processing parses the natural occuring data structures from the raw CRIS data. Each data structure which is built contains a S/C clock time tag that allows the data to be traced to the original S/C minor frame and to correlate it with other data. The data inside the structures is left in raw units, i.e. ADC values, PHA value, and counts. The details of each structure is covered in the next section. The level-one processing is done on an instrument cycle basis. In other words, 256 instrument minor frames (spanning 257 S/C minor frames) are processed at a time. The criteria used to determine whether a block of minor frames contains a valid instrument cycle is the following: 1. the S/C clock of each S/C minor frame increments by 1 for each of the 257 minor frames present, 2. the S/C major and minor frame counters increment properly for each of the 257 S/C minor frames, 3. the quality flag is low for each of the 257 S/C minor frames present, 4. the first 128 instrument minor frames have their sync bits set low, 5. the next 128 instrument minor frames have their sync bit set high, 6. the sync bit of the next instrument minor frame is low, 7. all events in the event are self-consistent in length, i.e. the sum of the length of the components is consistent with the length of the event. If the block passes the test, then the data is processed and put in the appropriate data structure. If the block fails, the following happens: 1. the oldest S/C minor frame is written to the level-one data set as unprocessed data, 2. another S/C minor frame is added to the end of the block, 3. the block is retested using the above criteria. If the number of unprocessed minor frames grows unacceptably large, then somewhere in the level-two processing the problem can be addressed. The unprocessed minor frames are part of the level-one product. By including the unprocessed minor frames in the product, it is possible to recover the original CRIS data stream from the level-one product; however, no software is planned for such a recovery. V. CRIS Level One Data Structures For each of the write functions mentioned in section I of this document, there is a corresponding data structure which will be written to the level-one data set using HDF functions. The purpose of this section is to define these data structures and provide some explanation for their existence and form. Each subsection will begin with the actual C structure followed by a brief explanation. A. Compressed Event Data Structure #define MAX_NUM_CRIS_EVENTS 454 #define NUM_CRIS_EVENT_BUFFER_BYTES 14336 /* 55*256 + 256 */ typedef struct { uint32 ClockCycle; uint32 ClockMinorFrame[MAX_NUM_CRIS_EVENTS]; uint16 EventIndex[MAX_NUM_CRIS_EVENTS]; uint16 NumberEvents; uint8 EventBuffer[NUM_CRIS_EVENT_BUFFER_BYTES]; uint8 ContinuationFlag; } L1CrisEventStruct; /* Definition: */ /* ClockCycle = S/C clock of the first minor frame of the cycle */ /* ClockMinorFrame = S/C clock of the minor frame containing the */ /* start of each event */ /* EventIndex = Starting byte of each event strored in the event */ /* buffer */ /* NumberEvents = Number of events stored in the event buffer */ /* EventBuffer = Event buffer */ /* ContinuationFlag = Flag indicating events from this cycle are */ /* continued in the next structure */ /* The size of a range 2 event with 6 SOFT clusters is 31 bytes. If we assume that this is the smallest event that is put into the buffers, then the maximum number of events that can be stored in one CRIS cycle is 454. */ The CRIS compressed events occupy most of the CRIS data stream. These events come from CRIS in a compressed form, and this form is maintained during the level-one processing. Even though it would have been possible to define a fixed data structure to store each event, it was felt that this would result in greatly enlarged data files. Since the work to expand a compressed event should be small compared to the work of analysing the event, it was felt that doing the expansion at analysis time was more appropriate. Standard functions to expand the compressed event into a fixed structure can be furnished to the CRIS team members. The actual events have an internal time tag with a resolution of 256 seconds. The S/C clock associated with an event is the time the event was put in the telemetry stream; it is not the time that CRIS processed the event. For low priority events, the readout time can be significantly different from the time the event was processed. The continuation flag in the data structure is not expected to be used. It only exists to deal with the pathological case, that the average length of an event is less than 31 bytes. B. Housekeeping/Rate Structure #define NUM_CRIS_HSKP_RATE_BUFFER_BYTES 768 typedef struct { uint32 ClockCycle; uint8 HskpRateBuffer[NUM_CRIS_HSKP_RATE_BUFFER_BYTES]; } L1CrisHskpRateStruct; /* Definition: */ /* ClockCycle = S/C clock of the first minor frame of the cycle */ /* HskpRateBuffer = Housekeeping/rate buffer for the cycle */ In the CRIS telemetry stream, three bytes from each minor frame are used to output housekeeping and rate quantities. The pattern is the same for each instrument cycle, so a data structure with fixed fields could have been substituted for the byte array in the structure. This would make given standard names to all the data quantities and made the the data slightly more accessible. However, most of these same benefits can be obtained by furnishing a C function that expands the data. The main argument for leaving it as an array is that the expansion can take place at the same time as the conversion to physical units; there is very little benefit in introducting another structure containing raw data. C. Command Table Structure #define NUM_CRIS_COMMAND_TABLE_BYTES 844 typedef struct { uint32 ClockMinorFrame0; uint32 ClockMinorFrame1; uint8 CommandTable[NUM_CRIS_COMMAND_TABLE_BYTES]; uint8 MissingDataFlag[NUM_CRIS_COMMAND_TABLE_BYTES]; } L1CrisCommandTableStruct; /* Definition: */ /* ClockMinorFrame0 = S/C clock of the minor frame containing the */ /* first byte of the command table */ /* ClockMinorFrame1 = S/C clock of the minor frame containing the */ /* last byte of the command table */ /* CommandTable = CRIS command table */ /* MissingDataFlag = Missing data flag for each byte of the */ /* command table */ The data in the CRIS command table structure is duplicated in the housekeeping/rate structure. In general, duplicating data is undesirable. However, since each housekeeping/rate structure contains only 66 bytes of the command table, the command table is much easier to use if there is a structure that contains the whole table. Since the readout of the command table spans about an hour, it is assumed the occasionally a minor frame of data will be missing. For places where there is missing data, the missing data flag is set. The level-two processing will have to decide what to do with command tables with missing data. The two clocks associated with the command table only indicate when the readout of the command table started and when it ended. They should not be interpreted to mean that in the time interval the command table's value was stable. D. Command Echo Structure #define MAX_NUM_CRIS_COMMAND_ECHO_CHARACTERS 54 typedef struct { uint32 ClockMinorFrame; uint8 NumberChars; char8 CommandEcho[MAX_NUM_CRIS_COMMAND_ECHO_CHARACTERS]; } L1CrisCommandEchoStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame containing the */ /* command echo */ /* NumberChars = Number of characters in the command echo */ /* CommandEcho = Command echo characters */ Whenever a command is received, CRIS puts a command response in its telemetry. There is no agreed-upon limit on the size of the CRIS command response. Also, the form of the command response is ASCII characters with embedded linefeeds and carriage returns. The command response was designed to be displayed on a "dumb" terminal and to be interpreted by humans. Since command responses usually span minor frames and also can span instrument cycles, there is no simple way to determine where one command response ends and another begins. To keep the level-one processing simple, it was decided to call the portion of the command response that appears in a instrument minor frame a command echo and define a structure to contain it. E. Diagnostic Event Structure #define NUM_CRIS_DIAG_EVENT_BUFFER_BYTES 6440 typedef struct { uint32 ClockMinorFrame; uint32 NumberBytes; uint8 DiagEventBuffer[NUM_CRIS_DIAG_EVENT_BUFFER_BYTES]; uint8 ContinuationFlag; } L1CrisDiagnosticEventStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame containing the */ /* start of the diagnostic event */ /* NumberBytes = No. of bytes of the diagnostic event in this */ /* structure */ /* DiagEventBuffer = Diagnostic event buffer */ /* ContinuationFlag = Flag indicating the diagnostic event is */ /* continued in the next buffer */ The size of a CRIS diagnostic event is dominate by the portion containing the raw SOFT data. The maximum length of the SOFT data is controlled by a firmware limit which is current set at 6 Kbytes. There is also a maximum hardware limit of 32 Kbytes. If the firmware limit is never revised upward, all diagnostic events will fit in the buffer. If the limit is revised upward, then the continuation flag will be used. F. Unprocessed Minor Frame Structure #define NUM_CRIS_TELEMETRY_BYTES 58 typedef struct { uint32 ClockMinorFrame; uint8 FormatId; uint8 MajorFrameCount; uint8 MinorFrameCount; uint8 quality; uint8 MainBusVoltage; uint8 CrisCurrent; uint8 CrisInternalTemperature1; uint8 CrisInternalTemperature2; uint8 CrisInterfaceTemperature; uint8 NumberBytesUsed; uint8 CrisData[NUM_CRIS_TELEMETRY_BYTES]; } L1CrisMinorFrameStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame */ /* FormatId = Format of the minor frame */ /* MajorFrameCount = Major frame count of the minor frame */ /* MinorFrameCount = Minor frame count of the minor frame */ /* quality = Quality flag associated with the minor frame */ /* MainBusVoltage = Main bus voltage value (0=no data) */ /* CrisCurrent = CRIS I/F & survial heater current (0=no data) */ /* CrisInternalTemperature1 = */ /* CRIS internal temperature 1 (0=no data) */ /* CrisInternalTemperature2 = */ /* CRIS internal temperature 2 (0=no data) */ /* CrisInterfaceTemperature = */ /* CRIS interface temperature (0=no data) */ /* NumberBytesUsed = Number of CRIS data bytes used */ /* CrisData = CRIS data */ Whenever the CRIS level-one processing encounters a S/C minor frame which contains data that is not part of a valid CRIS instrument cycle, it writes the above structure to the level-one data set. Most of the fields in the structure are self-explanatory. Because of the relationship of an instrument minor frame to a S/C minor frame, the values which NumberBytesUsed can assume the values 0, 4, and 54. G. Minor Frame Summary Structure typedef struct { uint32 ClockMinorFrame; uint8 FormatId; uint8 MajorFrameCount; uint8 MinorFrameCount; uint8 quality; uint8 MainBusVoltage; uint8 CrisCurrent; uint8 CrisInternalTemperature1; uint8 CrisInternalTemperature2; uint8 CrisInterfaceTemperature; uint8 NumberBytesUsed; } L1CrisMinorFrameSummaryStruct; /* Definition: */ /* ClockMinorFrame = S/C clock of the minor frame */ /* FormatId = Format of the minor frame */ /* MajorFrameCount = Major frame count of the minor frame */ /* MinorFrameCount = Minor frame count of the minor frame */ /* quality = Quality flag associated with the minor frame */ /* MainBusVoltage = Main bus voltage value (0=no data) */ /* CrisCurrent = CRIS I/F & survial heater current (0=no data) */ /* CrisInternalTemperature1 = */ /* CRIS internal temperature 1 (0=no data) */ /* CrisInternalTemperature2 = */ /* CRIS internal temperature 2 (0=no data) */ /* CrisInterfaceTemperature = */ /* CRIS interface temperature (0=no data) */ /* NumberBytesUsed = Number of CRIS data bytes used */ The purpose of this data structure is two-folded. The first is to give a summary of the amount of data was processed out of each S/C minor frame. The second is to hold the CRIS-specific S/C housekeeping values. H. Incomplete Event Structure typedef struct { uint32 ClockMinorFrame; uint8 NumberEventBytes; uint8 EventBytes[256]; } L1CrisIncompleteEventStruct; The main purpose for the existence of this data structure is to be able to claim that 100% of the CRIS data is in the level-one data set. Incomplete events can arise in two different manners. The first way occurs when CRIS is in normal mode. Since compressed events are of variable length, there is usually one event which will not fit into the event space of the instrument cycle. Since events are not allowed to span cycle boundaries, the event gets truncated if CRIS attempts to put it into the instrument minor frame. The second way happens when CRIS switches from normal mode to diagnostic mode and back to normal mode. When CRIS changes to diagnostic mode, it suspends its normal event processing and starts inserting frames with diagnostic event data. When it resumes normal mode, it will continue at the exact byte that it had last outputted. If for some reason, there is missing data in the cycle that CRIS resumes normal operations, the last event will not be completed by the level-one processing.