NAME
ABbuilder - DFO data organization tool

SYNOPSIS
ABbuilder <options>
where <options> are:
   --raw-dir   directory containing raw data
   --cal-dir   directory containing calib data
   --cal-ldir  directory containing symbolic links to calib files
   --cal-list  mcalib list
   --vcal-dir  directory containing virtual products
   --pro-dir   where pipeline products are created
   --ab-dir    where association blocks are created
   --oca-rul   file containing OCA rules
   --instrume  instrument
   --date      YYYY-MM-DD
   --mode      { CALIB | SCIENCE | ALL(default} }
   --help      print this help
   --version   print version


DESCRIPTION
The ABbuilder tool organizes the input data set into Association Blocks
according to the given Classification-Organization-Association rules.
The ABbuilder application performs the following steps:

1- load the OCA rules. If a syntax error is detected during this
   phase, ABbuilder reports a message indicating where the syntax
   error occurred, and the application terminates.

2- read the required keyword values from all of the *.fits *.tfits *.hdr files
   under the raw data directory (including subdirectories).

3- read the required keyword values from all of the *.fits *.tfits *.hdr files
   under the calib data directory (including subdirectories).
   If the virtual product directory is specified (see option --vcal-dir),
   read the relevant keywords from all virtual products under this directory.

4- classify all of the raw files, using the OCA classification rules
   ("if..then" statements)

5- organize the raw files into groups, according to the OCA organization rules
   ("select execute" statements)

6- for each group defined in the previous step, find the associated files
   and recipe using the association rules specified in the relevant
   action

7- for each group, create an association block in the AB directory

8- If the virtual product directory is specified (see option --vcal-dir),
   create virtual products in HDR format under this directory.


SPECIFIC PARAMETERS
ABbuilder expects the following parameters to be defined in the OCA rules.

- PACK.DIR : using the classification rules, the user can define data type
  specific pack dirs. For example:
  if RAW.TYPE=="BIAS" then PACK.DIR="/data/pack/bias"
  The value of PACK.DIR is written in the Association Block, under PACK_DIR.

- PRO.EXT : this parameter should be used in the (virtual) product definition
  to specify the predicted extension of the product filename. For example:
  product mbias { PRO.CATG="MASTER_BIAS"; PRO.EXT="0001.fits"; }

If PACK.DIR or PRO.EXT is not specified, ABbuilder will generate an error message
but the association blocks will still be created.

OPTIONS
The following options are available:

--raw-dir   <pathname>
	    Specify a directory containing raw (input) data.
	    ABbuilder will organize all FITS and HDR files under this directory
	    (including sub-directories).
	    This option is mandatory.

--cal-dir   <pathname>
	    Specify a directory containing calibration data.
	    ABbuilder will build a catalog of all FITS, TFITS and HDR files under this directory
	    (including sub-directories). This catalog is the local calibration database which is
	    used during the association process.
	    Either this option or the options --cal-ldir and --cal-list must be used.

--cal-ldir  <pathname>
	    Specify a directory where ABbuilder creates symbolic links to calibration files
	    listed in the mcalib list (See option --cal-list).
	    Either this option or the option --cal-dir must be given. This option
	    must be used together with --cal-list option.

--cal-list  <filename>
	    This option must be used in connection with option --cal-ldir, to specify
	    a file containing a catalog of calibration data (mcalib list).
	    If this option is used, ABbuilder will create symbolic links to the calibration
	    files listed in the mcalib list under the directory given by --cal-ldir.

--vcal-dir  <pathname>
	    Specify a repository of virtual products, stored as HDR files.
	    Virtual products are the expected products that will be generated
	    as result of the execution of an Association Block.
	    Virtual products can be defined in the OCA association rules (See OCA
	    rules documentation).
	    If this option is used, ABbuilder will load into the local calibration database
	    all virtual products already stored in the given directory, and
	    create new virtual products in the given directory.

--pro-dir   <pathname>
	    Specify the base directory under which pipeline products are created.
	    See also PROD_PATH in Association Block.
	    This option is mandatory.

--ab-dir    <pathname>
	    Specify a directory where Association Blocks are created.
	    This option is mandatory.

--oca-rul   <filename>
	    Specify a file containing OCA rules, to be used for the data organization.
	    This option is mandatory.

--instrume  <instrname>
	    Specify the instrument name to be stored in the Association Block, as value
	    of INSTRUMENT parameter.
	    If this option is not specified, the instrument name will be read from the
	    FITS header (keyword INSTRUME).

--date      <YYYY-MM-DD>
	    Specify the DFO date. In the current version this information is not used.

--mode      { CALIB | SCIENCE | ALL(default} }
	    Specify the category (DPR.CATG) of the raw files to process. The possible
	    values are:
	    CALIB - only calibration ABs are created
	    SCIENCE - only science ABs are created
	    ALL - default value, all raw data under directory given by --raw-dir is processed
	    (organized).

--help	    print help and exit

--version   print version and exit


CONFIGURATION FILES
ABbuilder configuration files are stored under directory "conf"
of ABbuilder installation tree. At present the following configuration files
are available:

- log4j.cf : configuration file for the logging system (log4j)

- ABbuilder.cfg : contains some specific configuration parameters, which
 are not supported by the OCA rules, e.g.PACK_ADD

EXAMPLES
1) calibration data is available under /data/calib.

ABbuilder --raw-dir  /data/raw
	  --cal-dir  /data/calib
	  --pro-dir  /data/product
	  --ab-dir   /data/ab
	  --oca-rul  /data/rul/test.rul
	  --instrume TEST
	  --vcal-dir /data/virtual

2) a catalog of calibration data is provided (mcalib_list). Symbolic
   links are created under /data/calib/links. Instrument is
   not specified, will be read from the fits header.
   Virtual products (headers) will not be stored, since vcal-dir is not specified.

ABbuilder --raw-dir  /data/raw
	  --cal-ldir /data/calib/links
	  --cal-list /data/calib/mcalib_list
	  --pro-dir  /data/product
	  --ab-dir   /data/ab
	  --oca-rul  /data/rul/test.rul

KNOWN ISSUES
- Possible performance issues in connection with very large calibration databases
 (several thousends of files)


FUTURE DEVELOPMENTS
- for large calibration data sets, the performance of association could be improved by
  storing the relevant keywords in a relational database, to avoid
  re-reading the fits files each time the ABbuilder is executed.

SEE ALSO
- OCA rules
- Association Block