# "@(#) $Id: README 2844 2015-11-17 13:46:05Z jpritcha $"
README file for J.Pritchard's ESO-MIDAS FEROS Data Reduction System Environment
This archive (jFEROS-DRS.tgz) contains the J.Pritchard's *personal*
version of the FEROS Data Reduction System MIDAS scripts plus various
helper scripts and c code.
It is derived from version v1.61 of the official FEROS-DRS.tgz available
from:
http://www.ls.eso.org/lasilla/sciops/2p2/E2p2M/FEROS/DRS/index.html
I develope this personal version of the FEROS-DRS (jFEROS-DRS) for my own
benefit. I make it available to the public via my web site:
http://www.hq.eso.org/~jpritcha/jFEROS-DRS/index.html
Of course I give no guarantee of its suitability for your needs or its
scientific efficacy and neither I nor ESO accepts any responsibility for
data loss, hardware loss, or loss of any otherkind resulting from its
use in any way, shape or form.
None the less, if you use my version, I of course expect suitable
recognition in any resulting publication. If you ask for my help and/or
advise and any such help and/or advise is useful, then I expect to be
included as a co-author in any resulting scientific publication.
Table of contents
=================
0.0 Requirements
1.0 Installation
1.1 Shell configuration
1.2 Screen Colour Depth
1.3 Graphic and display windows
2.0 Use
2.1 Prepare the data directory
2.2 Extract the data from the Archive CDs
2.3 Run the DRS
2.4 General Controlling Environmental and/or MIDAS variables
2.4.1 Pack Reduced files
2.4.1.1 Sorting Packed Red files by Target
2.4.1.2 Cleaning Up after packing
2.4.2 Make HTML summary pages
2.4.2.1 Submit convert commands into background
2.4.3 Choice of filenames for FITS versions of DRS products
2.4.4 Creation of RAW image PS/PNG files
2.4.5 AutoInit
2.4.6 Extract Non-Flatfielded spectra
2.4.7 Merge DRS Product FITS into EXTENDED FITS
2.4.8 Use MODIFY/COLUMN instead of @@ colmean
2.4.9 Expected Number of orders in Initialisation
2.4.10 Reduce All Images
2.4.11 Background Subtraction
2.4.12 Input FITS file extensions
2.5 Observatory Specific Controlling Environmental and/or MIDAS variables
2.5.1 Make Quality Control Plots
2.5.2 Inclusion of FFHV TCCD images in DRS tar files
2.5.3 Concatenate tar files
2.5.4 Process DRS Products for QC
2.5.5 Archive DRS tar files in ESO Science Archive
2.5.6 Display QC web pages after init...
2.5.7 Keep Intermediate MT files
2.6 Multiple Extension FITS files
2.7 Timezone
2.8 Cross Correlation Functions
2.9 Running in an esorex environment
Appendix 1: DRS Product filename conventions
Appendix 2: Shell Variables
Appendix 3: Installation on w2p2off
Appendix 4: MIDAS variables
Appendix 5: Dependant software
================================================================================
================================================================================
0.0 Requirements
================
The scripts have been prepared originally under Linux, and more recently
under Mac OS X with a view to portability. They are tested regularly within
these two environments.
Several scripts rely the WCS-Tools package, CFITSIO and PGPlot which are all
incldued in SciSoft.
http://www.eso.org/science/scisoft/
If you do not have and/or do not wish to install SciSoft then they should be
installed independantly (See Appendix 5).
The wget tool is used by in the Makefile in the src directory. wget is
a standard part of most Linux distributions. For Mac OS X it is available as
a standalone Cocoa Applet from:
http://e.cocoawget.nobody.jp/
or as part of Fink or MacPorts/DarwinPorts. Alternatively you can download
the required packages manaully. A complete list of packages to be downloaded
is given in Appendix 5.
Please report any problems you experience with the MIDAS scripts or the
other files in the archive to:
j.pritchard@eso.org
================================================================================
1.0 Installation
================
Normally this archive should be extracted into the user's home directory,
though you may want to extract to a dedicated subdirectory to avoid the
possibility of overwriting already existing files.
In the following it will be assumed the user creates a directory FEROS-DRS in
their Home directory in which to extract the file:
mkdir $HOME/FEROS-DRS
cd $HOME/FEROS-DRS
tar -zxvf
/jFEROS-DRS.tgz
or alternatively...
gzip -dc /jFEROS-DRS.tgz | tar -xvf -
This should produce eight directories and this README file...
README
bin
dat
fmt
html
lib
midwork
saved_session
src
Assuming you have and use SciSoft, then:
Build src code:
*) For tcsh:
setenv FEROS_DRS `pwd`
cd src
setenv PERL5LIB ${PERL5LIB}:${FEROS_DRS}/lib/perl5:${FEROS_DRS}/lib/perl5/site_perl
setenv CFITSIO /scisoft
setenv PERL `which perl`
setenv PGPLOT /scisoft
You may also need to set the folowing vaiables to appropriate values for
your system: MIDASHOME, MIDVERS.
make PMs
make all install
If you wish to use the new PERL script for cross-correlation you will also
need to do:
make XPMs
*) For Bash:
export FEROS_DRS=`pwd`
cd src
export PERL5LIB=${PERL5LIB}:${FEROS_DRS}/lib/perl5:${FEROS_DRS}/lib/perl5/site_perl
export CFITSIO=/scisoft
export PGPLOT=/scisoft
export PERL=`which perl`
You may also need to set the folowing vaiables to appropriate values for
your system: MIDASHOME, MIDVERS.
make PMs
make all install
If you wish to use the new PERL script for cross-correlation you will also
need to do:
make XPMs
1.1 Shell configuration
=======================
Once the software is installed, you will need to set several environment
variables. The following are the basic minimum. Others are detailed in the
sections that follow, and the complete lists are sumarised in Appendix 2.
*) For tcsh, add the following lines to .cshrc (or .tcshrc):
setenv FEROS_DRS "${HOME}/FEROS-DRS"
setenv FEROS_MIDWORK "${FEROS_DRS}/midwork"
alias inmidas.feros "inmidas -m ${FEROS_MIDWORK}"
setenv PATH ${PATH}:${FEROS_DRS}/bin
if ( $?PERL5LIB ) then
setenv PERL5LIB ${PERL5LIB}:${FEROS_DRS}/lib/perl5:${FEROS_DRS}/lib/perl5/site_perl
else
setenv PERL5LIB ${FEROS_DRS}/lib/perl5:${FEROS_DRS}/lib/perl5/site_perl
endif
*) For Bash, add the following lines to .bashrc
export FEROS_DRS="${HOME}/FEROS-DRS"
export FEROS_MIDWORK="${FEROS_DRS}/midwork"
alias inmidas.feros="inmidas -m ${FEROS_MIDWORK}"
export PATH=${PATH}:${FEROS_DRS}/bin
export PERL5LIB=${PERL5LIB}:${FEROS_DRS}/lib/perl5:${FEROS_DRS}/lib/perl5/site_perl
To get MIDAS running in a nice blue separate xterminal just like at
the telescope use the following for you inmidas.feros alias:
*) For tcsh, add the following lines to .cshrc (or .tcshrc):
alias inmidas.feros "xterm -ls -bg blue -fg white -geometry 80x34+0-6 -fn 8x16 -sb -e inmidas -m ${FEROS_MIDWORK}"
*) For Bash, add the following lines to .bashrc
alias inmidas.feros="xterm -ls -bg blue -fg white -geometry 80x34+0-6 -fn 8x16 -sb -e inmidas -m ${FEROS_MIDWORK:-$HOME/FEROS-DRS/midwork}"
1.2 Screen Colour Depth
-----------------------
In the file midwork/login.prg it is assumed the X11 Screen Colour Depth will
be 24-bit. If you do not have this you will need to comment out the line:
init/displ p5=RGBQ
1.3 Graphic and display windows
-------------------------------
Several of the integral MIDAS prg files now require FIVE MIDAS graphic
windows (with IDs 0, 1, 2, 3, 4) to exist before running. By default these
are created in a rather simple fashion by the login.prg file, however
several other more sophisticated examples are provided in the files
midwork/creaDandG.prg.alt*. To use one of these examples you can simply make
a softlink pointing at the relevant one, e.g.:
cd $HOME/FEROS-DRS/midwork
ln -s creaDandG.prg.alt0 creaDandG.prg
If you want to customise one of the alternatives, then (for example):
cd $HOME/FEROS-DRS/midwork
cp creaDandG.prg.alt0 creaDandG.prg
$EDITOR creaDandG.prg
or if you use the same midwork directory from different computers with
different Video capabilities you may want to create a HOST specific
creaDandG.prg:
cd $HOME/FEROS-DRS/midwork
cp creaDandG.prg.alt0 creaDandG.prg.${HOST}
$EDITOR creaDandG.prg.${HOST}
In either of the two above cases the files creaDandG.prg,
creaDandG.prg.${HOST} will survive an installation to a new version of the
jFEROS-DRS.tgz package. If you make a link but then edit the original file,
these modifications will be overwritten when a new version of the
jFEROS-DRS.tgz package is installed.
================================================================================
2.0 Use
=======
As an example, assume we want to reduce the data from the night of
2003-01-21...
2.1 Prepare the data directory
------------------------------
cd ~/FEROS-DRS
ferosPrepDateDir 2003-01-21
This will create a directory 2003-01-21 and then make softlinks pointing to
the initial guess files. With no -g option to ferosPrepDateDir, the
default initial guess (9001) will be used. If the initial guess files do not
already exist they will be created in the separate directory guess9001.
2.2 Extract the data from the Archive CDs
-----------------------------------------
ferosPrepRawData 2003-01-21 -r /2003-01-21
This will uncompress the files on the FEROS Archive CD and rename them with
their original name (i.e. in fero.mt format for BIAS images or
FEROS_ech_... for FIERA images). The script will automatically detect if the
image was created with FIERA and if so then attempt to make softlinks of the
form fero.mt using the ferosMkferoNNNNLink script.
You may need to do the above command several times if the data from one
night is on more than one CD.
In addition to the Image files, the LOG files will be copied to. If there
are images you are not interested in they can be easily removed using the
log file and the ferosCleanByLog script. Simple edit the log file and remove
the entries for images you do NOT want to reduce and then run the ferosCleanByLog
script:
cd 2003-01-21
FEROS.2003-01-21.log
ferosCleanByLog FEROS.2003-01-21.log
2.3 Run the DRS
---------------
inmidas.feros -P
And then follow the instructions in ~/FEROS-DRS/midwork/README...
2.4 General Controlling Environmental and/or MIDAS variables
------------------------------------------------------------
A number of features have been implemented which are controlled by the
existance and/or value of Operating System Environment variables
and/or MIDAS keywords. They are detailed below.
2.4.1 Pack Reduced files
------------------------
In order to package the reduced files for each reduction as well as the
Guess and Session archives, you need to set the FEROSPackRedFiles
environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSPackRedFiles "Y"
*) For Bash, add the following line to .bashrc
export FEROSPackRedFiles="Y"
To include bias straightened images (e.g. back_str_C1.bdf etc) and
pre-flatfield extracted images (e.g. bfext1) in the
tarballs then set the FEROSPackRedFilesIncAll environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSPackRedFilesIncAll "Y"
*) For Bash, add the following line to .bashrc
export FEROSPackRedFilesIncAll="Y"
2.4.1.1 Sorting Packed Red files by Target
------------------------------------------
In order to create softlinks pointing to the PackRedFiles sorted into
directories on a target by target basis, you need set the
FEROSPackRedFilesTargetsDir environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSPackRedFilesTargetsDir "$FEROS_DRS/targets"
*) For Bash, add the following line to .bashrc
export FEROSPackRedFilesTargetsDir="$FEROS_DRS/targets"
Above $FEROS_DRS/targets is given as an example, but it can be anything.
2.4.1.2 Cleaning Up after packing
---------------------------------
To cleanup the reduction directory after packing set the
FEROSPackRedFilesCleanUp environment variables. The BDF, FITS and PS ones
allow to turn off cleanup of these specific items.
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSPackRedFilesCleanUp "Y"
*) For Bash, add the following line to .bashrc
export FEROSPackRedFilesCleanUp="Y"
If FEROSPackRedFilesCleanUp=Y then the following can be set to turn off
cleanup of these specific items:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSPackRedFilesCleanUpBDF "N"
setenv FEROSPackRedFilesCleanUpFITS "N"
setenv FEROSPackRedFilesCleanUpPS "N"
*) For Bash, add the following line to .bashrc
export FEROSPackRedFilesCleanUpBDF="N"
export FEROSPackRedFilesCleanUpFITS="N"
export FEROSPackRedFilesCleanUpPS="N"
2.4.2 Make HTML summary pages
-----------------------------
In order create HTML summary pages, you need to set the FEROSmkHTML
environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSmkHTML "Y"
*) For Bash, add the following line to .bashrc
export FEROSmkHTML="Y"
or at the MIDAS prompt:
write/keyword symMkHTML/c/1/1 "Y"
You can also turn on debugging in the ferosMkHTML.pl script using the
FEROSmkHTMLdebug environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSmkHTMLdebug "-d"
*) For Bash, add the following line to .bashrc
export FEROSmkHTMLdebug="-d"
or at the MIDAS prompt:
write/keyword symMkHTMLdebug/c/1/2 "-d"
2.4.2.1 Run convert commands into background
------------------------------------------------
To run the commands to convert PS files into PNGs in the background, you
need to set the FEROSmkHTMLbgConv environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSmkHTMLbgConv "Y"
*) For Bash, add the following line to .bashrc
export FEROSmkHTMLbgConv="Y"
2.4.3 Choice of filenames for FITS versions of DRS products
-----------------------------------------------------------
To have VLT compliant name format for output DRS files, i.e.
r..nnnn.fits (See Appendix 1 below for naming conventions), you
need to set the FEROSUseAFDRSFITS environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSUseAFDRSFITS "Y"
*) For Bash, add the following line to .bashrc
export FEROSUseAFDRSFITS="Y"
or at the MIDAS prompt:
write/keyword UseAFDRSFITS/c/1/1 "Y"
2.4.4 Creation of RAW image PS/PNG files
----------------------------------------
To create PS and/or PNG files (PNGs will then be included in the html pages set
the following environment variables:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSDRSDS9mkPS "Y"
setenv FEROSDRSDS9mkPNG "Y"
*) For Bash, add the following line to .bashrc
export FEROSDRSDS9mkPS="Y"
export FEROSDRSDS9mkPNG="Y"
You must have the ds9 software installed (preferably the latest version, 4.0b7
at this writing) and there must exist a softlink ferosDRSds9 which points to
the ds9 binary (though the software will try to create this if it does not
already exist if the ds9 binary can be found using the which command, see
ferosDRSds9cmd)
PNG files are created from the PS files so PNG creation implies PS creation
also. If these variables do not exist nothing happens.
Furthermore the name of the output PS/PNG files can be controlled using the
FEROSDRSDS9useVLTFN environment variable: If set as follows:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSDRSDS9useVLTFN "Y"
*) For Bash, add the following line to .bashrc
export FEROSDRSDS9useVLTFN="Y"
output files will have the filesname FEROS.yyyy-mm-ddThh:mm:ss.sss.ps/png,
otherwise they will be fero.ps/png.
The resulting PNG files will be included in the html files whichever filename
format is used.
2.4.5 AutoInit
--------------
At the telescope the the daily health check allows initialisation of both
1x1 and 2x2 binning modes. In order to use the correct initialisation during
the night it is necessary to enable AutoInit detection. To do so set the
following environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSAutoInit "Y"
*) For Bash, add the following line to .bashrc
export FEROSAutoInit="Y"
This will override any manual setting of the FEROS configuration, e.g. to
make a Optimum extraction. Therefore to be able to control the reduction in
Visitor mode, in the MIDAS DRS session set the following MIDAS variable as
follows:
write/key symAutoInit/c/1/1 "N"
The the extraction mode (for example) can be set to OPTIMUM as follows:
set/feros EXT_MODE=O
This functionality is implemented in autoreduce.prg.
2.4.6 Extract Non-Flatfielded spectra
-------------------------------------
To extract Non-Flatfielded spectra AS WELL AS flatfielded spectra set
following environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSExtractNonFFSpec "Y"
*) For Bash, add the following line to .bashrc
export FEROSExtractNonFFSpec="Y"
or at the MIDAS prompt:
write/keyword ExtrNonFFSpec/c/1/1 "Y"
Non-Flatfielded spectra are useful when there is VERY little flux in
the flatfield, as was the case for the first few bluest orders prior
to 2004-Sep-09. For these spectra what signal there was in the SCIENCE
spectra were completely dominated by the noise in the zero signal
flatfields.
2.4.7 Merge DRS Product FITS into EXTENDED FITS
-----------------------------------------------
By default the DRS produces one BDF for the merged spectrum and one
BDF image for each unmerged individual order, i.e. 39 files for each
fibre. Since by default the pipeline generates flatfielded and
NON-flatfielded reductions in the end this is 2 x (1+39) x 2 = 120
files. If one also does all this for Optimum as well as Standard
extraction you get 240 files!
By default each these is converted to one FITS file. This option
allows to merge these individual FITS files into 4 FITS files each
with 40 Header Data Units (HDUs) thus keeping things a little
tidier. As of MIDAS 06FEB it will be possible to create the 40 HDU
FITS files directly from MIDAS with outdisk/sfits but MIDAS versions
before 06FEB can not cope with so many input files. Therefore this in
the meantime this is achieved with a simple C program
(ferosMkMEFs) based on the FITSIO library.
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSMkMEFs "Y"
*) For Bash, add the following line to .bashrc
export FEROSMkMEFs="Y"
or at the MIDAS prompt:
write/keyword symMkExtdFITS/c/1/1 "Y"
In the resulting Multi-HDU FITS or Multi Extension FITS (MEF) files
the merged spectrum is in the zero-index HDU and the non-merged orders
are in index 1-39 HDUs.
2.4.8 Use MODIFY/COLUMN instead of @@ colmean
---------------------------------------------
To use the MIDAS generic MODIFY/COLUMN command instead of the FEROS
DRS @@ colmean command to deal with bad columns set following
environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSUseModColumn "Y"
*) For Bash, add the following line to .bashrc
export FEROSUseModColumn="Y"
or at the MIDAS prompt:
write/keyword symUseModColumn/c/1/1 "Y"
At least one of the "bad columns" (at column 1350 in raw frames,
column 1300 in prered processed frames) actually affects two
consequetive columns. The MIDAS MODIFY/COLUMN command which can deal
with consequective columns therefore does a better job than @@ colmean
which can only deal with one column at a time.
2.4.9 Expected Number of orders in Initialisation
-------------------------------------------------
By default the FEROS DRS currently expects to locate 39 orders when
performing the order location step in the DRS initialisation. This can
be controlled by the environment variable FEROSInitExpectNOrders:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSInitExpectNOrders ""
*) For Bash, add the following line to .bashrc
export FEROSInitExpectNOrders=""
or at the MIDAS prompt:
write/keyword FEROSInitExpNo/I/1/1
where is an integer, e.g. 38 or 40...
2.4.10 Reduce All Images
------------------------
By default the FEROS DRS only reduces spectra with EXPTYPE = SCIENCE. To
force a reduction of all EXPTYPES set the MIDAS variable RedAllIms:
at the MIDAS prompt:
write/keyword RedAllIms/C/1/1 "Y"
to disable:
delete/keyword RedAllIms
or
write/keyword RedAllIms/C/1/1 "N"
2.4.11 Background Subtraction
-----------------------------
By default the FEROS DRS the 1st step of the autoreduce procedure is
background subtration (BACKGROUND/FEROS). To disable this set the MIDAS
variable ARBkgSub:
at the MIDAS prompt:
write/keyword ARBkgSub/C/1/1 "N"
to disable:
delete/keyword ARBkgSub
or
write/keyword ARBkgSub/C/1/1 "Y"
2.4.12 Input FITS file extensions
---------------------------------
By default, due to historical reasons, input files are expected with
filenames like fero.mt (.mt is a MIDAS and/or ESO file extension for
"Magnetic Tape", but in this case also implies fits files).
Alternatively .fits can be used (i.e. input files with filenames like
fero.fits). This can be controlled by the environment variable
FEROSferoNNNNsuf:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSferoNNNNsuf "fits"
*) For Bash, add the following line to .bashrc
export FEROSferoNNNNsuf="fits"
2.5 Observatory Specific Controlling Environmental and/or MIDAS variables
-------------------------------------------------------------------------
A number of futher features have been implemented which are controlled
by the existance and/or value of Operating System Environment variables
and/or MIDAS keywords but are generally only relevant to operations at
the telescope. They are detailed below.
2.5.1 Make Quality Control Plots
--------------------------------
In order to create Quality Control plots, you need to set the FEROSQCPlots
environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSQCPlots "Y"
*) For Bash, add the following line to .bashrc
export FEROSQCPlots="Y"
This is done automatically at the telescope by detection of the hostname
being equal to w2p2off.
2.5.2 Inclusion of FFHV TCCD images in DRS tar files
----------------------------------------------------
To include FFHV TCCD images in the DRS product tar files, you need to set
the FEROSFFHVImFitsFile environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSFFHVImFitsFile ""
*) For Bash, add the following line to .bashrc
export FEROSFFHVImFitsFile=""
2.5.3 Concatenate tar files
---------------------------
To concatenate files to already existing DRS product tar files set the
FEROSConcatTarFiles environment variable as follows:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSConcatTarFiles "Y"
*) For Bash, add the following line to .bashrc
export FEROSConcatTarFiles="Y"
2.5.4 Process DRS Products for QC
---------------------------------
To Process the DRS Products for QC trending, you need to set the
FEROSProcImage environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSProcImage "Y"
*) For Bash, add the following line to .bashrc
export FEROSProcImage="Y"
2.5.5 Archive DRS tar files in ESO Science Archive
--------------------------------------------------
To copy the the DRS product tar files (including TCCD images) into the
ESO Science Archive via the DFS, you need to set the FEROSDRStars2DFS
environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSDRStars2DFS "Y"
*) For Bash, add the following line to .bashrc
export FEROSDRStars2DFS="Y"
You will probably also want to force the inclusion of README files for
the Archive tarballs:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSPackRedFilesSpecialIncludes " [file2...]"
*) For Bash, add the following line to .bashrc
export FEROSPackRedFilesSpecialIncludes=" [file2...]"
2.5.6 Display QC web pages after init...
----------------------------------------
To display QC web pages after an initialisation set the following
environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSQCURLroot ""
*) For Bash, add the following line to .bashrc
export FEROSQCURLroot=""
or at the MIDAS prompt:
write/keyword FEROSQCURLroot/c/1/100 ""
2.5.7 Keep Intermediate MT files
---------------------------------
By default when MEFs and/or VLT filenames are used for output files the
intermediate f1.mt, f0001.mt etc files are deleted. This can be
switched off by setting the following environment variable:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROSKeepIntMTs "Y"
*) For Bash, add the following line to .bashrc
export FEROSKeepIntMTs="Y"
or at the MIDAS prompt:
write/keyword symKeepIntMTs/c/1/1 "Y"
2.6 Multiple Extension FITS files
---------------------------------
Creation of Multiple Extension FITS (MEF) files for FEROS will be supported
by MIDAS natively as of version 06SEP. To obtain MEFs under pre 06SEP
versions of MIDAS do:
cd $HOME/FEROS-DRS/src
make all install
to create ferosMkMEFs, a simple tool based on cfitsio.
2.7 Timezone
------------
Since files are sorted in the web pages according to date and time of OB
start and grouped together according to local midday before the night till
local midday the day after the night it can be useful to set the timezone to
"Chile/Continental" or the equivalent on your system for use by
ferosMJD2DSofN.pl, the script used to compute the date of the start of the
night i.e.:
*) For tcsh, add the following line to .cshrc (or .tcshrc):
setenv FEROS_TZ "Chile/Continental"
*) For Bash, add the following line to .bashrc
export FEROS_TZ="Chile/Continental"
2.8 Cross Correlation Functions
-------------------------------
The MIDAS xcorall.prg script allows crosscorrelation between two FEROS
spectra as well as allowing to make a correction based on simultaneous WLC
spectra taken. This is useful to (for example) build the radial velocity
curve for an object for which you have multiple spectra observed in
Simultaneous Calibration (OBJCAL) mode.
A new perl script, ferosXcor.pl, has been implemented and allows the
crosscorrelation of FEROS spectra both with other FEROS spectra but also
with (almost) any other kind of spectrum, e.g. a theoretical template.
If the -p or -P command line options are given (see ferosXcor.pl -h) plots
will be made to the /XWIN and /CPS devices by default. These two options can
be overridden by the Environment variables PGPLOT_DEV and PGPLOT_PSPNG. The
-G command line option will also force plotting to the /PNG device rather
than the /CPS device.
2.9 Running in an esorex environment
------------------------------------
export FEROSesorex="Y"
export FEROSferoNNNNsuf="fits"
export FEROSusePROCATGnames="Y"
export FEROSnoDisplay="Y"
export FEROSUseAFDRSFITS="N"
export FEROSMkMEFs="Y"
export FEROSmkHTML="N"
export FEROSPackRedFiles="N"
Note: in all of the below, optional handling of the MASTER_BIAS is not yet
implemented.
2.9.1 I/O for DRS_INIT
======================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+BIAS
DPR.CATG+DPR.TYPE=CALIB+FLAT
DPR.CATG+DPR.TYPE=CALIB+WAVE
CALIB
...from calibDB:
PRO.CATG=BLAZE_TBL
PRO.CATG=LINE_REF_TBL
PRO.CATG=DRS_SETUP_TBL
PRO.CATG=LINE_TBL_1
PRO.CATG=LINE_TBL_2
PRO.CATG=ORDER_GUESS_TBL
PRO.CATG=ORDER_TBL
PRO.CATG=PROF_TEMPLATE
PRO.CATG=COEFF_TBL_1
PRO.CATG=COEFF_TBL_2
Output
PRO.CATG=COEF_COP_1 coef_cop_1.fits
PRO.CATG=COEF_COP_2 coef_cop_2.fits
PRO.CATG=COEFF_TBL_1 coeff_tbl_1.fits
PRO.CATG=COEFF_TBL_2 coeff_tbl_2.fits
PRO.CATG=DRS_SETUP_TBL drs_setup_tbl.fits
PRO.CATG=EXTR_FF_1 extr_ff1.fits
PRO.CATG=EXTR_FF_2 extr_ff2.fits
PRO.CATG=EXTR_WLC_1 extr_wlc1.fits
PRO.CATG=EXTR_WLC_2 extr_wlc2.fits
PRO.CATG=LINE_REF_TBL line_ref_tbl.fits
PRO.CATG=LINE_TBL_1 line_tbl_1.fits
PRO.CATG=LINE_TBL_2 line_tbl_2.fits
PRO.CATG=MASTER_BIAS master_bias.fits
PRO.CATG=MASTER_FLAT master_flat.fits
PRO.CATG=MASTER_WAVE master_wave.fits
PRO.CATG=ORDER_GUESS_TBL order_guess_tbl.fits
PRO.CATG=ORDER_TBL order_tbl.fits
PRO.CATG=PROF_TEMPLATE prof_template.fits
PRO.CATG=PROF_TEMPLATET prof_templatet.fits
2.9.2 I/O for BIAS
==================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+BIAS
CALIB:
None required
Output
PRO.CATG=MASTER_BIAS master_bias.fits
master-bias-0001.paf
2.9.3 I/O for DARK
==================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+DARK
CALIB:
...from localDB
PRO.CATG=MASTER_BIAS master_bias.fits
Output
PRO.CATG=MASTER_DARK master_dark.fits
master_dark-0001.paf
2.9.4 I/O for FLAT
==================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+FLAT
CALIB:
...from calibDB
PRO.CATG=BLAZE_TBL
PRO.CATG=LINE_REF_TBL
PRO.CATG=DRS_SETUP_TBL
PRO.CATG=LINE_TBL_1
PRO.CATG=LINE_TBL_2
PRO.CATG=ORDER_GUESS_TBL
PRO.CATG=ORDER_TBL
PRO.CATG=PROF_TEMPLATE
PRO.CATG=COEFF_TBL_1
PRO.CATG=COEFF_TBL_2
...from localDB
PRO.CATG=MASTER_BIAS master_bias.fits
Output
PRO.CATG=COEF_COP_1 coef_cop_1.fits
PRO.CATG=COEF_COP_2 coef_cop_2.fits
PRO.CATG=EXTR_FF_1 extr_ff_1.fits
PRO.CATG=EXTR_FF_2 extr_ff_2.fits
PRO.CATG=MASTER_FLAT master_flat.fits
PRO.CATG=ORDER_GUESS_TBL order_guess_tbl.fits
PRO.CATG=ORDER_TBL order_tbl.fits
PRO.CATG=PROF_TEMPLATET prof_templatet.fits
master_flat-0001.paf
2.9.5 I/O for WAVE
==================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+WAVE
CALIB:
...from calibDB
PRO.CATG=BLAZE_TBL
PRO.CATG=LINE_REF_TBL
PRO.CATG=DRS_SETUP_TBL
PRO.CATG=LINE_TBL_1
PRO.CATG=LINE_TBL_2
PRO.CATG=PROF_TEMPLATE
PRO.CATG=COEFF_TBL_1
PRO.CATG=COEFF_TBL_2
...from localDB
PRO.CATG=COEF_COP_1 coef_cop_1.fits
PRO.CATG=COEF_COP_2 coef_cop_2.fits
PRO.CATG=EXTR_FF_1 extr_ff_1.fits
PRO.CATG=EXTR_FF_2 extr_ff_2.fits
PRO.CATG=MASTER_BIAS master_bias.fits
PRO.CATG=MASTER_FLAT master_flat.fits
PRO.CATG=ORDER_GUESS_TBL order_guess_tbl.fits
PRO.CATG=ORDER_TBL order_tbl.fits
PRO.CATG=PROF_TEMPLATET prof_templatet.fits
Output
PRO.CATG=COEFF_TBL_1 coeff_tbl_1.fits
PRO.CATG=COEFF_TBL_2 coeff_tbl_2.fits
PRO.CATG=DRS_SETUP_TBL drs_setup_tbl.fits
PRO.CATG=EXTR_WLC_1 extr_wlc_1.fits
PRO.CATG=EXTR_WLC_2 extr_wlc_2.fits
PRO.CATG=LINE_TBL_1 line_tbl_1.fits
PRO.CATG=LINE_TBL_2 line_tbl_2.fits
PRO.CATG=MASTER_WAVE master_wave.fits
master_wave-0001.paf
2.9.6 I/O for SCIENCE
=====================
Input:
RAW:
DPR.CATG=SCIENCE
CALIB:
...from calibDB
PRO.CATG=BLAZE_TBL
PRO.CATG=LINE_REF_TBL
...from localDB
PRO.CATG=COEF_COP_1 coef_cop_1.fits
PRO.CATG=COEF_COP_2 coef_cop_2.fits
PRO.CATG=COEFF_TBL_1 coeff_tbl_1.fits
PRO.CATG=COEFF_TBL_2 coeff_tbl_2.fits
PRO.CATG=DRS_SETUP_TBL drs_setup_tbl.fits
PRO.CATG=EXTR_FF_1 extr_ff1.fits
PRO.CATG=EXTR_FF_2 extr_ff2.fits
PRO.CATG=EXTR_WLC_1 extr_wlc1.fits
PRO.CATG=EXTR_WLC_2 extr_wlc2.fits
PRO.CATG=LINE_REF_TBL line_ref_tbl.fits
PRO.CATG=LINE_TBL_1 line_tbl_1.fits
PRO.CATG=LINE_TBL_2 line_tbl_2.fits
PRO.CATG=MASTER_BIAS master_bias.fits
PRO.CATG=MASTER_FLAT master_flat.fits
PRO.CATG=MASTER_WAVE master_wave.fits
PRO.CATG=ORDER_GUESS_TBL order_guess_tbl.fits
PRO.CATG=ORDER_TBL order_tbl.fits
PRO.CATG=PROF_TEMPLATE prof_template.fits
PRO.CATG=PROF_TEMPLATET prof_templatet.fits
Output
PRO.CATG=EXTR_SCI_1 extr_sci_1.fits
PRO.CATG=EXTR_SCI_2 extr_sci_2.fits
PRO.CATG=RED_SCI_1 red_sci_1.fits
PRO.CATG=RED_SCI_2 red_sci_2.fits
2.9.7 I/O for SKY,SKY
=====================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+SKY,SKY
CALIB:
...from calibDB
PRO.CATG=BLAZE_TBL
PRO.CATG=LINE_REF_TBL
...from localDB
PRO.CATG=COEF_COP_1 coef_cop_1.fits
PRO.CATG=COEF_COP_2 coef_cop_2.fits
PRO.CATG=COEFF_TBL_1 coeff_tbl_1.fits
PRO.CATG=COEFF_TBL_2 coeff_tbl_2.fits
PRO.CATG=DRS_SETUP_TBL drs_setup_tbl.fits
PRO.CATG=EXTR_FF_1 extr_ff1.fits
PRO.CATG=EXTR_FF_2 extr_ff2.fits
PRO.CATG=EXTR_WLC_1 extr_wlc1.fits
PRO.CATG=EXTR_WLC_2 extr_wlc2.fits
PRO.CATG=LINE_REF_TBL line_ref_tbl.fits
PRO.CATG=LINE_TBL_1 line_tbl_1.fits
PRO.CATG=LINE_TBL_2 line_tbl_2.fits
PRO.CATG=MASTER_BIAS master_bias.fits
PRO.CATG=MASTER_FLAT master_flat.fits
PRO.CATG=MASTER_WAVE master_wave.fits
PRO.CATG=ORDER_GUESS_TBL order_guess_tbl.fits
PRO.CATG=ORDER_TBL order_tbl.fits
PRO.CATG=PROF_TEMPLATE prof_template.fits
PRO.CATG=PROF_TEMPLATET prof_templatet.fits
Output
PRO.CATG=EXTR_SOL_1 extr_sol_1.fits
PRO.CATG=EXTR_SOL_2 extr_sol_2.fits
PRO.CATG=RED_SOL_1 red_sol_1.fits
PRO.CATG=RED_SOL_2 red_sol_2.fits
PRO.CATG=FIBRATIO_TBL fibratio_tbl.fits
fibratio_tbl-0001.paf
2.9.8 I/O for Standard stars
============================
Input:
RAW:
DPR.CATG+DPR.TYPE=CALIB+...
CALIB:
...from calibDB
PRO.CATG=BLAZE_TBL
PRO.CATG=LINE_REF_TBL
...from localDB
PRO.CATG=COEF_COP_1 coef_cop_1.fits
PRO.CATG=COEF_COP_2 coef_cop_2.fits
PRO.CATG=COEFF_TBL_1 coeff_tbl_1.fits
PRO.CATG=COEFF_TBL_2 coeff_tbl_2.fits
PRO.CATG=DRS_SETUP_TBL drs_setup_tbl.fits
PRO.CATG=EXTR_FF_1 extr_ff1.fits
PRO.CATG=EXTR_FF_2 extr_ff2.fits
PRO.CATG=EXTR_WLC_1 extr_wlc1.fits
PRO.CATG=EXTR_WLC_2 extr_wlc2.fits
PRO.CATG=LINE_REF_TBL line_ref_tbl.fits
PRO.CATG=LINE_TBL_1 line_tbl_1.fits
PRO.CATG=LINE_TBL_2 line_tbl_2.fits
PRO.CATG=MASTER_BIAS master_bias.fits
PRO.CATG=MASTER_FLAT master_flat.fits
PRO.CATG=MASTER_WAVE master_wave.fits
PRO.CATG=ORDER_GUESS_TBL order_guess_tbl.fits
PRO.CATG=ORDER_TBL order_tbl.fits
PRO.CATG=PROF_TEMPLATE prof_template.fits
PRO.CATG=PROF_TEMPLATET prof_templatet.fits
PRO.CATG=FLUX_STD_TABLE .fits
PRO.CATG=EXTCOEFF_TABLE atmoextn.fits
Output
PRO.CATG=EXTR_STD_1 extr_std_1.fits
PRO.CATG=EXTR_STD_2 extr_std_2.fits
PRO.CATG=RED_STD_1 red_std_1.fits
PRO.CATG=RED_STD_2 red_std_2.fits
PRO.CATG=EFFICIENCY_TBL efficiency_tbl.fits
PRO.CATG=RESPONSE_TBL response_tbl.fits
PRO.CATG=RESPONSE_IMG response_img.fits
efficiency_tbl-0001.paf
2.10 Using the Herman Hensberge algorithm
=========================================
Prof. Herman Hensberge and his group have provided updates to the
ferso software, including both bug-fixes and an alternative approach
to the ordertracing algorithm.
By default the alternative approach is DISABLED. So to use the
alternative apporach, set the following environment variable:
*) For tcsh:
setenv FEROSUseHH Y
*) For Bash:
export FEROSUseHH=Y
Details concerning the alternative approcah are given in <>.
John Pritchard
2007-03-18
================================================================================
============================================================================
Appendix 1: DRS Product filename conventions
============================================
If FEROSUseAFDRSFITS="Y" then DRS products converted to FITS files according
to the following convention:
mDPID = modified DPID = FEROS.YYYY-MM-DDTHH:MM:ss.sss
i.e. without the .fits extension
The one dimensional extracted frames are:
MEF FITS files (if FEROSMkMEFs="Y")
-----------------------------------
If both Merged and Non-Merged products are produced:
Standard extraction...
r..1081.fits <==> f1.bdf,f10001.bdf,...,f10039.bdf
r..1082.fits <==> f2.bdf,f20001.bdf,...,f20039.bdf
Optimum extraction...
r..2081.fits <==> o1.bdf,o10001.bdf,...,o10039.bdf
r..2082.fits <==> o2.bdf,o20001.bdf,...,o20039.bdf
If only Non-Merged products are produced:
Standard extraction...
r..1091.fits <==> f10001.bdf,...,f10039.bdf
r..1092.fits <==> f20001.bdf,...,f20039.bdf
Optimum extraction...
r..2091.fits <==> o10001.bdf,...,o10039.bdf
r..2092.fits <==> o20001.bdf,...,o20039.bdf
If only Merged products are produced:
Standard extraction...
r..1101.fits <==> f1.bdf
r..1102.fits <==> f2.bdf
Optimum extraction...
r..2101.fits <==> o1.bdf
r..2102.fits <==> o2.bdf
i.e. the same as in the case of Single HDU FITS files...
Single HDU FITS files (if FEROSMkMEFs=/="Y")
--------------------------------------------
Standard extraction...
Merged...
r..1101.fits <==> f1.bdf
r..1102.fits <==> f2.bdf
Non-Merged, order-by-order...
r..1251.fits <==> f10039.bdf
r..1252.fits <==> f20039.bdf
..
r..1631.fits <==> f10001.bdf
r..1632.fits <==> f20001.bdf
Optimum extraction...
Merged...
r..2101.fits <==> o1.bdf
r..2102.fits <==> o2.bdf
Non-Merged, order-by-order...
r..2251.fits <==> o10039.bdf
r..2252.fits <==> o20039.bdf
..
r..2631.fits <==> o10001.bdf
r..2632.fits <==> o20001.bdf
The complete list:
Pre-Extraction products...
==========================
MEF files (if FEROSMkMEFs="Y")
------------------------------
r..0001.fits <==> back_str_C1.bdf, back_str_D1.bdf, back_str_X1.bdf
r..0002.fits <==> back_str_C2.bdf, back_str_D2.bdf, back_str_X2.bdf
Single HDU FITS files (if FEROSMkMEFs=/="Y")
--------------------------------------------
r..0011.fits <==> back_str_C1.bdf
r..0021.fits <==> back_str_D1.bdf
r..0031.fits <==> back_str_X1.bdf
r..0011.fits <==> back_str_C2.bdf
r..0022.fits <==> back_str_D2.bdf
r..0032.fits <==> back_str_X2.bdf
Standard extraction:
====================
r..1041.fits <==> bfext1.bdf
r..1042.fits <==> bfext2.bdf
r..1051.fits <==> fext1.bdf
r..1052.fits <==> fext2.bdf
r..1061.fits <==> rebinned1.bdf
r..1062.fits <==> rebinned2.bdf
r..1063.fits or r..1071.fits <==> rebinnedNoFF1.bdf
r..1064.fits or r..1072.fits <==> rebinnedNoFF2.bdf
MEF files (if FEROSMkMEFs="Y")
------------------------------
r..1081.fits <==> f1.bdf,f10001.bdf,...,f10039.bdf
r..1082.fits <==> f2.bdf,f20001.bdf,...,f20039.bdf
r..1083.fits <==> noFFf1.bdf,noFFf10001.bdf,...,noFFf10039.bdf
r..1084.fits <==> noFFf2.bdf,noFFf20001.bdf,...,noFFf20039.bdf
r..1091.fits <==> f10001.bdf,...,f10039.bdf
r..1092.fits <==> f20001.bdf,...,f20039.bdf
r..1093.fits <==> noFFf10001.bdf,...,noFFf10039.bdf
r..1094.fits <==> noFFf20001.bdf,...,noFFf20039.bdf
If FEROSMkMEFs="Y" but MERGE_MTD="S" or "A" then the products will be
as for Single HDU FITS file (see below).
r..1111.tfits <==> efficiencyf1.tbl
r..1112.tfits <==> efficiencyf2.tbl
r..1121.tfits <==> responsef1.bdf,responsef1.tbl
r..1122.tfits <==> responsef2.bdf,responsef2.tbl
r..1130.tfits <==> fiberratiof.tbl
Single HDU FITS files (if FEROSMkMEFs=/="Y")
--------------------------------------------
Merged:
r..1101.fits <==> f1.bdf
r..1102.fits <==> f2.bdf
r..1103.fits <==> noFFf1.bdf
r..1104.fits <==> noFFf2.bdf
Non-Merged, order-by-order...
r..1251.fits <==> f10039.bdf
r..1252.fits <==> f20039.bdf
r..1253.fits <==> noFFf10039.bdf
r..1254.fits <==> noFFf20039.bdf
..
r..1631.fits <==> f10001.bdf
r..1632.fits <==> f20001.bdf
r..1633.fits <==> noFFf10001.bdf
r..1634.fits <==> noFFf20001.bdf
Optimum extraction:
===================
r..2041.fits <==> boext1.bdf
r..2042.fits <==> boext2.bdf
r..2051.fits <==> oext1.bdf
r..2052.fits <==> oext2.bdf
r..2061.fits <==> rebinned1.bdf
r..2062.fits <==> rebinned2.bdf
r..2063.fits or r..2071.fits <==> rebinnedNoFF1.bdf
r..2064.fits or r..2072.fits <==> rebinnedNoFF2.bdf
MEF files (if FEROSMkMEFs="Y")
------------------------------
r..2081.fits <==> o1.bdf,o10001.bdf,...,o10039.bdf
r..2082.fits <==> o2.bdf,o20001.bdf,...,o20039.bdf
r..2083.fits <==> noFFo1.bdf,noFFo10001.bdf,...,noFFo10039.bdf
r..2084.fits <==> noFFo2.bdf,noFFo20001.bdf,...,noFFo20039.bdf
r..2091.fits <==> o10001.bdf,...,o10039.bdf
r..2092.fits <==> o20001.bdf,...,o20039.bdf
r..2093.fits <==> noFFo10001.bdf,...,noFFo10039.bdf
r..2094.fits <==> noFFo20001.bdf,...,noFFo20039.bdf
If FEROSMkMEFs="Y" but MERGE_MTD="S" or "A" then the products will be
as for Single HDU FITS file (see below).
r..2111.tfits <==> efficiencyo1.tbl
r..2112.tfits <==> efficiencyo2.tbl
r..2121.tfits <==> responseo1.bdf,responseo1.tbl
r..2122.tfits <==> responseo2.bdf,responseo2.tbl
Single HDU FITS files (if FEROSMkMEFs=/="Y")
--------------------------------------------
Merged:
r..2101.fits <==> o1.bdf
r..2102.fits <==> o2.bdf
r..2103.fits <==> noFFo1.bdf
r..2104.fits <==> noFFo2.bdf
Non-Merged, order-by-order...
r..2251.fits <==> o10039.bdf
r..2252.fits <==> o20039.bdf
r..2253.fits <==> noFFo10039.bdf
r..2254.fits <==> noFFo20039.bdf
..
r..2631.fits <==> o10001.bdf
r..2632.fits <==> o20001.bdf
r..2633.fits <==> noFFo10001.bdf
r.