A.1 ARCHIVE INPUT DATA FORMATS

Datasets received by ST DADS on the SOGS interface are in VAX GEIS format or 'other' formats where other formats are ASCII text files or binary data representations. The following sections provide an overview of the expected formats.

A.1.1 GEIS Format

GEIS is a SOGS specific data format derived from the FITS data format standard. The GEIS format separates the header and data portions of a FITS format file into a separate header file and data file. The header file contains parameters which describe the contents of the data file, as well as additional parameters. SOGS documentation refers to the combination of the header file and data file as a file type. An example of a Calibrated Science dataset in GEIS format is illustrated in Figure A-1. This figure shows that the dataset is made up of pairs of files where each file pair is a file type. Calibrated Science data is described in detail in the Section A.3.1.

Figure A-1.

Illustration of a GEIS Dataset

(See file entitled "Landscape Files")

A.1.1.1 GEIS Header Files. GEIS header files are based on the standard FITS format described in Reference 7 and the FITS extension for grouped data described in Reference 8. Reference 2 describes the SOGS GEIS header format.

FITS headers are comprised of records containing information describing the structure of the data, information describing the data, and textual comments. The header consists of a number of 80-character card images. The syntax of a card image is given by:

KEYWORD = VALUE / COMMENT

1 2 3 4 5 8

12345678901234567890123456789012345678901234567890 ... 0

The keyword shall be a left justified, 8 character, blank filled, ASCII string with no embedded blanks. All digits (0-9) and upper case alphabetic characters (A-Z) are permitted; no lower case characters shall be used. The underscore ("_") and hyphen ("-") are also permitted. An "=" sign occurs in column 9 and an ASCII blank occurs in column 10 to indicate that an associated value or comment field follows. If no values are present, columns 9-80 may contain any ASCII text. Logical, integer, or real data shall be right-justified in columns 11-30 and any imaginary part shall be right-justified in columns 31-50.

Character strings shall be delineated by single quote marks ("'") with the first single quote in column 11 followed by the string beginning in column 12. The closing single quote shall not occur before column 20 or after column 80. Leading blanks are significant; trailing blanks are not. ST DADS shall conform to the FITS standards for processing single quotes and double quotes when handling GEIS character strings. Specifically, two contiguous single quotes are required to denote a single quote. Double quotes are allowed.

Comments may be preceded by a "COMMENT" keyword, a "HISTORY" keyword, or a blank keyword field. In these cases columns 9-80 may contain any ASCII text. Any number of comment rows may appear in a header. Comments may also be included on a line following other keyword values. The value and the comment shall normally be separated by a "/" in column 32 (or column 51 in the case of imaginary data) to denote the beginning of the comment. The last card in a header shall always contain an "END" keyword.

In addition to FITS keywords specified in References 7 and 8 which described the structure of the data GEIS includes the DATATYPE keyword which specifies the data type of the data items contained in the data file of each file type pair and the PDTYPEi keyword which specifies the data type of FITS group parameters where 'i' indicates the specific group parameter. ST DADS supports the following values for the DATATYPE keyword:

INTEGER*2

UNSIGNED*2 (non-FORTRAN)

INTEGER*4

REAL*4

REAL*8

ST DADS supports the following values for the PDTYPEi keywords:

LOGICAL*4 CHARACTER*8

INTEGER*4 CHARACTER*24

REAL*4 CHARACTER*48

REAL*8 CHAR*8

CHARACTER*4

Appendix B provides additional detail on the usage of structural keywords.

A.1.1.2 GEIS Data Files. The size and structure of binary data files are described by the keywords in the corresponding header files. As shown in Figure A-2, the binary data file consists of one or more fixed blocks of data called groups. Each group consists of an array of binary data followed by an associated group parameters block. The minimum structure of a data file consists of a single data array and no parameters.

The data within each record of the data array is organized as an n-dimensional array of data stored in "words" of a fixed number of bits. The structural keywords in the header describe the size and dimensions of the array in addition to the datatype of the array (see Appendix B). The data is represented in either VAX real or integer formats.

The associated group parameter block contains zero or more parameters. The datatype of each parameter in the group parameter block is defined by the PDTYPEi keywords found in the GEIS header file. The actual group parameter values are contained within the parameter block of the GEIS data file.

A.1.2 Other Formats

ST DADS shall also accept ASCII and other binary format files for ingest.

A.2 ST DADS DATA FORMAT CONVERSIONS

ST DADS shall archive most data files received in the FITS format; however some files shall be archived in the same format as received. ST DADS converts most GEIS format files received from SOGS into standard FITS format. In addition, some ASCII format files shall be converted by ST DADS into FITS table extension format. Table A-1 provides format details for each file ingested by ST DADS from SOGS. When validation override is set by SOGS and a dataset fails validation the dataset shall not be converted to FITS by ST DADS, but shall be archived in its original format.

Figure A-2.

GEIS Ddata File Structure

(See file entitled "Landscape Files")

The following sections provide an overview of the format conversions performed by ST DADS on ingested datasets. As specified in Section 3, ST DADS shall provide all datasets in their original format when retrieved by SOGS. Therefore, the data format conversion processes are considered to be internal ST DADS processes and the details are not specified in this ICD. Reference 18 describes the details of ST DADS format conversion processing.

A.2.1 GEIS TO FITS CONVERSION

ST DADS performs the following activities when converting from GEIS to FITS:

a. ST DADS combines the separate GEIS header and associated data files into single, standard FITS files.

b. ST DADS converts the VAX representation of the GEIS data arrays into standard FITS representations. Real data is converted to IEEE-754 floating point. Integer data is converted to place the highest order byte in the lowest byte address.

c. ST DADS reverses the order of the data arrays and associated group parameter blocks to conform to FITS and ensures that the group parameters and data array block are of the data type as required by FITS.

d. GEIS allows for a non-standard order of keywords, including the use of the keywords DATATYPE and PDTYPEi.

e. ST DADS sets the keyword SIMPLE=T.

Table A-1. ST DADS Archive Classes Format Description

(part 1 of 2)

(See file entitled "Landscape Files")

Table A-1. ST DADS Archive Classes Format Description

(part 2 of 2)

(See file entitled "Landscape Files")

A.2.2 ASCII TO FITS EXTENSIONS

ST DADS converts certain ASCII format files to FITS table extension format conforming to Reference 9. Specific files that are converted to FITS are identified under Section A.3 of this Appendix. Reference 18 describes the details of this conversion process. These ST DADS FITS tables have a fixed record length of 132 bytes. Any records in the original source files which exceed 132 bytes will be represented by multiple records in the corresponding ST DADS FITS table and will not be restored to a single record on retrieval .

A.3 ARCHIVE CLASS DESCRIPTIONS

This section describes the file naming convention and data format for datasets of each archive class provided by SOGS to ST DADS. Table A-1 summarizes the file organization and record structure for the files in each of these archive classes. Unless otherwise specified, information extracted for the ST DADS catalog from datasets of archive classes described in the following sections shall be treated as optional keywords. In cases where the information presented in this section is not identical to the current version of sources cited, the original source cited in the reference is to take precedence.

A.3.1 CALIBRATED SCIENCE DATA

SOGS shall send Calibrated Science Data to ST DADS in the GEIS format as described in Section A.1.1. Calibrated Science files which contain raw data that have been reformatted into the GEIS format are labeled as the Raw Portion of the Calibrated Science dataset. File types which contain data which have been reformatted into the GEIS format and calibrated are labeled as the Calibrated Portion of the Calibrated Science dataset. Each file that is contained in the calibrated portion of a dataset contains a file extension that begins with the letter "C".

Each Calibrated Science dataset will also contain an ASCII format trailer file that has the file extension ".TRL". This file does not belong to the raw or calibrated portion. Calibrated Science datasets for the WF/PC and FOC instruments will also include an ASCII format group parameter file. ST DADS shall convert each of these files (TRL, DGR, and CGR) into corresponding FITS table extension format files and then store the resulting FITS file in the archive.

File types previously illustrated in Figure A-1 include the Standard Header Packet (SHP) composed of the SHH/SHD file pair, the Unique Data Log (ULH/ULD file pair), Science Data (D0H/D0D), Data Quality (Q0H/Q0D), and Calibrated Science Data (C0H/C0D, C1H/C1D). Reference 2 provides a detailed description of these and other file types that comprise Calibrated Science datasets for the different instruments. ST DADS shall combine each of these GEIS file pairs into a single FITS file and store the resulting FITS file in the archive. Appendix B of this ICD specifies the required files for each instrument's Calibrated Science datasets.

The format of the file name stem used for all files in a Calibrated Science dataset is given in Figure A-3. This figure illustrates that the file name stem is dependent on the instrument used to produce the data, and the HST observation which the data represents.

Figure A-3. Calibrated Sceince Data File Naming Convention

A.3.1.1 Calibrated Science Header Files. Appendix B specifies the required keywords for Calibrated Science dataset header files. As noted in Appendix B, some keywords must fall within a specified value range. Reference 2 specifies the complete list of possible files and the keywords contained in those files. Appendix B specifies only those files and keywords that are validated by ST DADS.

A.3.1.2 Calibrated Science Data Files. The size and structure of binary data files are described by the keywords of the header files.

A.3.1.3 Group Parameter Files. For FOC and WF/PC Calibrated Science datasets SOGS shall provide a group parameter file which includes the keywords extracted from the data file parameter block as described above. These files shall have the same filename stem as the other files in the Calibrated Science dataset and shall have the extension .DGR and .CGR where the .DGR file provides group parameters applicable to the .D*D files and the .CGR file provides group parameters applicable to the .C*D files. ST DADS shall convert .CGR and .DGR file into FITS table extension format.

A.3.2 EDITED SCIENCE DATA

SOGS shall send Edited Science datasets to ST DADS in binary format. Edited Science Data is the raw packetized data for science instruments that is supplied to SOGS by the ST Data Capture Facility (DCF). It is similar to the raw portion of Calibrated Science Data, except it is in packetized format. Edited Science datasets do include one GEIS format file type, the DGH/DGD file pair, where the .DGH file is an ASCII format GEIS header as described in Section A.1.1.1.

The Edited Science dataset is comprised of multiple files that relate to a SOGS observation. The file name stems for Edited Science datasets follow the IPPPSSOOT convention as described for Calibrated Science Data (Figure A-3).

ST DADS does not extract specific header information for the ST DADS catalog for this archive class since it is associated with a Calibrated Science dataset. ST DADS performs no format conversion on any files in Edited Science datasets.

A.3.3 ASTROMETRY DATA

SOGS shall send Astrometry datasets to ST DADS consisting of three GEIS format astrometry file pairs (A*H/A*D), a proposal information file (DMH), and a message initiation file (DBI). An Astrometry dataset represents the data produced by all of the Fine Guidance Sensor (FGS) instruments for each HST observation. There are three FGS instruments with one astrometry file pair for each. The files comprising an Astrometry dataset are listed in Table 3-4. The A*D portion of the astrometry file pairs is required to be present only when the keyword DATAFILS='T'. ST DADS shall check the contents of only one of the three A*H files to obtain the value of DATAFILS. These files follow the IPPPSSOOT file naming convention shown in Figure A-3 for GEIS datasets where the instrument portion, I, of the file name always has a value of 'F' for Fine Guidance Sensor. Reference 4 describes Astrometry Data file formats and file naming conventions in detail. The required keywords for Astrometry datasets are presented in Appendix B. ST DADS shall convert all files in the dataset to FITS equivalents:

a. Each of the A*H/A*D GEIS file pairs shall be converted into a single FITS format file.

b. The GEIS format DMH file shall be converted into a FITS format file.

A.3.4 SCIENCE MISSION SPECIFICATION

SOGS shall send Science Mission Specification (SMS) datasets to ST DADS in ASCII format. An SMS dataset shall consist of a single file with the file extension POD. The file name stem is Y* where * is eight characters in the form:

* = YMDHHMMT

where,

Y= Year (1-9,A-Z) beginning in 1981

M= Month (1-9,A-C)

D= Day (1-9,A-V)

HH= Hour (00-23)

MM= Minute (00-59)

T= Qualifier

Note: This file name is the PODPS time stamp and this indicates the time in which the dataset is placed on the SOGS disk.

An example of the SMS header is shown in Figure A-4.

SMS0001

:SMSHDR,SMS_ID(SD003000), PATCH_ID( )

;;

,START=90Y003D00H00M00.000S,END=90Y005D00H00M00.000S

;;

,CREATED(1989Y301D04H00M37.120S),CALENDAR(A218501)

;;

,PDB_ID(22.00)

;;

;SMSTIME=1990.003:00:00:00.000

Figure A-4. Sample SMS Data

ST DADS shall catalog the values for the following keywords from the ASCII format SMS header:

SMS_ID()

START=

END=

CALENDAR()

PDB_ID()

ST DADS shall extract the values from within the '()' or following the '='as appropriate. ST DADS shall convert the START and END times to VAX date/time format as specified in Appendix H.

ST DADS shall convert ASCII format SMS files into FITS table extension format.

A.3.5 MISSION SCHEDULE DATA

SOGS shall send Mission Schedule datasets to ST DADS in ASCII format similar to that of SMS datasets. Mission schedule data describes the sequence of mission events transmitted to SOGS in response to an SMS. Mission Schedule datasets shall be identifiable by their file name stem and file extension. Mission Schedule datasets shall consist of a single file with the file extension POD. The file name stem is U* where * is the PODPS time stamp as defined in Section A.3.4. An example of the mission schedule is shown in Figure A-5.

RUN SMS: SD0030000

MS RUN NAME: S0147430, START=1990.002:23:55:00.000, END=1990.005:00:00:00.000

Figure A-5. Sample Mission Schedule

ST DADS shall catalog the values for the following keywords from the first portion of the ASCII format MSC header:

START=

END=

ST DADS shall extract the values following the '='as appropriate converting the START and END to VAX date/time format as specified in Appendix H.

ST DADS shall convert ASCII format MSC files into FITS table extension format.

A.3.6 MISSION TIMELINE DATA

SOGS shall send Mission Timeline datasets to ST DADS in ASCII format. Mission Timeline datasets shall consist of a single file with the file extension POD. The file name stem is V* where * is the PODPS time stamp as defined in Section A.3.4. ST DADS shall extract Mission Timeline Start Time and Mission Timeline Stop Time using a context search in the first 40 lines of the header for the line:

MISSION TIMELINE REPORT FOR THE PERIOD:yy.ddd hh:mm:ss.sss TO yy.ddd hh:mm:ss.sss

PAGE 001

ST DADS shall convert ASCII format MTL files into FITS table extension format.

A..3.7 OSS REAL-TIME ACTIVITY LOGS

SOGS shall send Observation Support System (OSS) Real-Time Activity (RTA) Log datasets to ST DADS in binary format. RTA datasets shall consist of a single file with the file extension POD. The file name stem is Z* where * is the PODPS time stamp as defined in Section A.3.4. The qualifier, T, at the end of the file stem is a "1" if the log is for the ScI facility or a "2" if the log is for ST Science Support Center (SSC) or "C" if it is RTA Catalog Data. The file name extension is POD. ST DADS extracts the RTA start time from the file name stem for the catalog. ST DADS shall archive the binary format RTA files in the same format as received from SOGS.

A.3.8 EPHEMERIS (DEFINITIVE ORBIT) DATA

SOGS shall send Ephemeris Data datasets to ST DADS in binary format. Ephemeris datasets shall consist of a single file that contains HST (2 Day) definitive orbit data. The file name extension for all Ephemeris data is ORX as defined in Reference 2. The file name stem is P*R where '*' is the time of the start of orbit data (as defined for the first seven characters of the PODPS time stamp described in Section A.3.4). References 1 and 3 define the format of the ephemeris files. ST DADS extracts orbit date from this file for the catalog converting it to VAX date/time format as specified in Appendix H. ST DADS shall archive the binary format ORB files in the same format as received from SOGS.

A.3.9 ON BOARD COMPUTER (OBC) DUMPS

SOGS shall supply Onboard Computer (OBC) Dump datasets to ST DADS in binary format. OBC Dump datasets shall contain four files with the extensions indicated in Table A-1. The file name stem shall have the form:

NQYMDHHZZ

where:

N - Denotes Dump Data (constant character "G")

Q - Denotes the Dump Type, and can take the values:

T -Executive Status Buffer Dump (NSSC-I)

U -NSSC-I Memory Dump

V -High Speed Photometer Microprocessor Memory Dump

X -Faint Object Camera Microprocessor Memory Dump

Y -Faint Object Spectrograoph Microprocessor Memory Dump

A -Unknown Dump Type

Y - Denotes the year of first minor frame (1-9,A-Z, beginning with 1981)

M - Denotes the month of first minor frame (1-9,A-C,

where January is 1)

D - Denotes the Day of the first minor frame (1-9,A-V,where 1 denotes the first day of month)

HH- Denotes the hour (00-23) of the day for the first minor frame

ZZ- Denotes the minute (00-59) of the hour for the first minor frame

ST DADS extracts the minor frame time from the file name stem for the catalog. ST DADS shall convert the minor frame time to VAX date/time format as specified in Appendix H. ST DADS shall archive the binary format DMP files in the same format as received from SOGS.

A.3.10 ENGINEERING DATA

A.3.10. ENGINEERING DATA

SOGS shall supply Engineering Data datasets to ST DADS. Engineering datasets shall consist of two required files, a data file and a PODPS generated message initiation file. The data file shall have an extension of DAT and shall have an ASCII file header containing information about the data followed by any number of minor frames of binary format data. The message initiation file extension is DBI and is a binary format file. Engineering dataset files shall have file name stems conforming to the following convention:

IYMDHHXXT

where

I=E

Y=Year (1-9,A-Z, where 1981 = 1)

M=Month (1-9,A-C, where January =1)

D=Day (1-9,A-V, where first day of the month = 1)

HH=Hour (00-23)

XX=Minute(00-59)

T=M (merged real time and ETR telemetry)

Reference 4 describes the format of the these files. ST DADS shall extract spacecraft start time, spacecraft stop time and engineering telemetry format from these ENG datasets for the catalog. Spacecraft start time is located in bytes 25-36 of the .DAT file and has the format YYDDDHHMMSS. Spacecraft stop time is located in bytes 37-48 of the .DAT file and has the format YYDDDHHMMSS. Both spacecraft start time and spacecraft stop time are ASCII, left justified, and spaced filled. ST DADS shall convert the start and stop time to VAX date/time format as specified in Appendix H. Engineering telemetry format is located in bytes 23-24 of the .DAT file and is of data type C*2. ST DADS shall archive the binary format ENG files in the same format as received from SOGS.

A.3.11 ENGINEERING SUBSET DATA

SOGS shall supply Engineering Subset datasets to ST DADS in binary format. Engineering Subset datasets may consist of one or more files. All files in a dataset shall have the same file name stem, but different file extensions. Engineering Subset dataset file names shall conform to the following convention:

IYMDHHPXX.EXT

where:

I= S

Y= Year (1-9,A-Z, where 1981=1)

M= Month (1-9,A-C, where January=1)

D=Day (1-9,A-V, where first day of the month = 1)

HH= Hour(00-23)

P= Subset Definition Origin where P can have the values:

P - PDB

W - Wildcard

XX= Subset Identifier (00-ZZ) assigned. This identifies the requestor.

and the file extension is named as:

EXT=ABC

where

A=Measurement Value Conversion

R=Raw

V=EU converted

B=Format:

C=Compressed

U=Uncompressed

C=Sequence number (0-9,A-Z) to eliminate file name duplication.

Each Engineering Subset file includes a header and minor frame subset records. The file header contains information that describes the file. The minor frame subset records contain subset measurements selected from the associated telemetry minor frame. Reference 4 describes these files in detail.

ST DADS shall extract the information indicated in Table A-2 from the first file of a Engineering Subset datasets where the first file is the one having the lowest sequence number in the file name extension. ST DADS shall archive the binary format SUB files in the same format as received from SOGS.

Table A-2. Engineering Subset Catalog Information

A.3.12 ANCILLARY DATA

SOGS shall supply Ancillary datasets to ST DADS. Ancillary datasets may consist of one or more files that may have any valid VMS file name stem up to 31 character and any valid VMS file name extension that does not exceed three characters. File extension shall not be required to make the dataset unique from other datasets. ST DADS shall determine the uniqueness of Ancillary datasets by the combination of the archive class (ANC), the file name stem, and the date of creation provided in the corresponding Archive Input Request. These datasets shall be cataloged using only dataset name, archive class, and generation date. There are no associated keywords. If ST DADS receives an Ancillary dataset that is not unique using these criteria, the dataset will not be archived by ST DADS. ST DADS shall store Ancillary datasets in the binary format in which they are received by ST DADS. All VMS file structures shall be maintained and supplied for distribution.

A.3.13 GUIDE STAR POSITION PRODUCTS

SOGS shall send Guide Star Position Products to ST DADS as two archive classes: Guide Star Data and Maneuver Verification Data. Both of these archive classes shall contain a single binary format file with file extensions of GSD and MVD respectively. These files are described in Table A-1 and Reference 4. The file naming convention is:

File name Stem = Tymdhmmss

where:

y= Year (1-9,A-Z, where 1981=1)

m= Month (1-9,A-C, where January=1)

d= Day (1-9,A-V, where first day of the month =1)

h= Hour (0-9,A-N, where first hour of the day=0)

mm= Minute (00-59)

ss= Second (00-59)

For either GSD or MVD archive class ST DADS shall extract the first 17 bytes of the file header which represents a UTC in ASCII format in the form:

yyyy.ddd hh:mm:ss

This UTC represents the predicted start time of the first data record in the file for GSD files and the maneuver start time of the first data record in the file for MVD files. ST DADS shall convert these times to VAX date/time format as specified in Appendix H.

ST DADS shall archive the binary format GSD and MVD files in the same format as received from SOGS.

A.3.14 OBSERVER COMMENTS

SOGS shall send Observer Comments received by PODPS from OSS to ST DADS in ASCII format. Observer comments are sometimes received as part of the TRL file of Calibrated Science datasets or the TRX file of Edited Science datasets. When received as a separate file ST DADS shall treat the file as a separate archive class where an Observer Comment dataset shall consist of a single file with the file extension OCX. The file name stem follows the IPPPSSOOT naming convention as shown in Figure A-3. OCX files shall have a filename stem that matches that of a Calibrated Science or Edited Science dataset except that the last character of the OCX file name stem shall always be "C". A single OCX file shall correspond to all versions of the associated Calibrated Science dataset. ST DADS shall catalog these datasets using only dataset name, archive class, and generation date. There are no associated keywords. ST DADS shall archive the ASCII format OCX files in FITS table extension format.

A.3.15 PODPS DATA QUALITY COMMENTS

SOGS shall send PODPS Data Quality Comments to ST DADS in ASCII format. PODPS Data Quality datasets shall consist of a single file with the file extension PDQ. The file name stem follows the IPPPSSOOT naming convention as shown in Figure A-3. PDQ files shall have a filename stem that matches all but the last character of a Calibrated Science dataset. A single PDQ file shall correspond to all versions of the associated Calibrated Science dataset, i.e. only the most recent version of extracted keyword values are maintained in the ST DADS catalog. ST DADS shall extract the values of the keywords QUALITY, QUALCOM1, and QUALCOM2. Keyword values are not validated by ST DADS. ST DADS shall archive the ASCII format PDQ files in FITS table extension format.

A.3.16 PROBLEM SCIENCE DATA

Problem Science datasets are fragmented science files which do not require validation by ST DADS. Problem Science datasets may consist of one or more files that may have any valid VMS extension that does not exceed three characters. The file name stem follows the IPPPSSOOT naming convention as shown in Figure A-3. File extension shall not be required to make the dataset unique from other datasets. ST DADS shall determine the uniqueness of these datasets by the combination of the archive class (PRB), the file name stem, and the date of creation provided in the corresponding Archive Input Request. These datasets shall be cataloged using only dataset name, archive class, and generation date. There are no associated keywords. If ST DADS receives a Problem Science dataset that is not unique using these criteria, the dataset will not be archived by ST DADS. ST DADS shall store these datasets in the binary format in which they are received by ST DADS. All VMS file structures shall be maintained and supplied for distribution.

A.3.17 DAYFILE REPORTS

SOGS shall send Dayfile Reports to ST DADS. Dayfile Report datasets may consist of one or more files having the same file name stem and the extension .RPT. These datasets shall be cataloged using only dataset name, archive class, and generation date. If ST DADS receives a Report dataset that is not unique using these criteria, the dataset will not be archived by ST DADS. There are no associated keywords. ST DADS shall archive the ASCII format RPT files in FITS table extension format.