make printable	new:	see also: - how to ingest general calibration products
	- v1.8: PAR scheme decommissioned (obsolete)	databases none dfos tools ingestProducts; cdbIngest, cdbDelete; fitsreport; called by moveProducts output executable file rn_files upload/download none

renameProducts

Description

This tool renames FITS product files to a different naming scheme. Presently supported are:

CALIBNAME (mode: CALIB or GEN) | USERSCINAME (mode: SCIENCE)

CALIBNAME is the standard QC name for calibration products, to be inserted into the mcalib archive and to be delivered to the packages.

USERSCINAME is the naming scheme for science products. It is used within the phoenix environment.

This tool uses fitsreport to deliver the set of fits key values which is then further massaged to create the file names.

Plugins. The optional plugin $SPECIAL_PLUGIN can be configured and, if provided under $DFO_BIN_DIR, is called after final creation of rn_file but before execution. You could use this plugin e.g. to check for filename length or other things. The $MODE variable is exported by renameProducts and is available within the pgi.

The optional plugin $PRE_PLUGIN can also be configured. It is called after calling fitsreport and creating $TMP_DIR/results. You can use this plugin to modify the content of that file, before it is used by renameProducts to create the new file names. This may be useful if the key content is strange (e.g. DET.ID has two components in FORS1 separated by a blank).

GEN mode. This mode is very similar to CALIB but renames files under $GEN_CALDIR (as defined in config.createAB) and is used for the static calibration products which are not necessarily pipeline products.

Output

The output comes as a text file called rn_files, under the same directory as the FITS files (the renaming file). It is an executable shell script which can move the files from one scheme to the other scheme. Execution of the renaming file is under responsability of the user. The tool does not rename by itself.

How to use

Type renameProducts -h for on-line help, renameProducts -v for the version number.

renameProducts -m CALIB -d 2024-12-12

will produce a renaming file under $DFO_CAL_DIR/2024-12-12. For modes CALIB and SCIENCE, the tool scans the FITS files under $DFO_CAL_DIR/<date> or $DFO_SCI_DIR/<date>.

In mode GEN, all FITS files under $GEN_CALDIR are scanned:

renameProducts -m GEN

Installation

Use dfosExplorer, or type dfosInstall -t renameProducts .

Configuration file

config.renameProducts defines:

NOTE: Since the tool uses fitsreport to create the key content table, the 'SPECIFIC' part of the config file mimics the syntax of fitsreport config files, with additional pieces describing the optional shell code to modify the key content. The advantage of that approach is to have the full functionality of fitsreport available, with e.g. multiple key definitions, and support and delimiter strings (helping to make key content unique).

NOTE: In the phoenix environment, and there only if MCALIBs are created, a non-standard config file can be used via the option -c <config-file>. This mechanism is used by the PHOENIX workflow tool to maintain versioning for the product names, to match the historical evolution of the product names. This is needed for MCALIB names only, since it might be desirable to keep the original naming shceme. The names of the various config files, and their versioning, are maintained in config.phoenix. This feature is not needed for the IDP production with phoenix, since the IDPs always have standardized product names.

Workflow description (for CALIB and GEN)

1. go to $DFO_CAL_DIR/<date> (CALIB); $GEN_CALDIR (e.g. $DFO_CAL_DIR/gen) (GEN)
2. for all product files (${DFO_FILE_NAME}*fits) create new name:

2.1 use ${MCAL_CODE} to have 2letter instrument code (ins_code)
2.2 read pro.catg; map into 4letter mcalib code in config file (mcal_code)
2.3 use ${DFO_OFFSET}, read mjd-obs; construct DATE (anything between [DATE]T12+offset … [DATEplus1]T12+offset is called DATE); NOTE: contrary to all other cases, DATE here has the format yymmdd!
2.4 CALIB: fill placeholder for version (e.g. "VERSION"); GEN: fill "A" for VERSION
2.5 read config file for setup key, support strings, SPECSTRING (pieces of shell code read from the config file), separators
2.6 execute per key: read key, modify with SPECSTRING, add support string, add separator; repeat for all parameters listed in config file to get complete setup_code. The setup_code could be made of 0 or 1…N keys.
2.7 if configured, execute $PRE_PLUGIN

2.8 read extension from original file
2.9 construct new name:
<ins_code>_<mcal_code>_<DATE>_VERSION_<setup_code>.<ext>
Examples:
GI_PFEX_040414VERSION_Medusa1_L437.2nm_o2.fits (CALIB)
GI_GCAT_021230A_line_catalog.fits (GEN)

2.10 check for filename length less than 68 chars; exit if longer name constructed: will lead to problems with database and tools

3. CALIB only: do versioning

3.1 read pro.catg; loop on pro.catg:
3.2 check per pro.catg in config file if versioning is required
3.3 if NO: loop on setup per pro.catg, write for all files 'A' instead of 'VERSION'
3.4 if YES: loop on setup per pro.catg:

3.4.1 sort by mjd-obs
3.4.2 give first one 'A', second one 'B' etc, up to 26 = Z
Example: GI_PFEX_040414B_Medusa1_L437.2nm_o2.fits (CALIB)
The tool has found two cases of files with GI_PFEX_040414VERSION_Medusa1_L437.2nm_o2.fits, this is the second one.

4. renaming

4.1 read CHECK_YN from config file
4.2 write per file "mv OLDNAME NEWNAME" into $DFO_CAL_DIR/<date>/rn_files
4.3 add line to set status flag to 'cal_Renamed'
4.4 if configured, execute $SPECIAL_PLUGIN
4.5 make rn_files executable but don't execute; if CHECK_YN=YES, warn the user if multiple OLDNAMEs end in a single NEWNAME
4.6 finish

Note: The procedure 4.5 is thought as a safety measure. Although in daily operations execute=yes will be the standard case, the authority given to the tool to 'implicitly delete' is potentially dangerous. By setting the additional parameter 'permission', the user keeps this under his control.

Workflow description for SCIENCE

1. go to $DFO_SCI_DIR/<date>

2. for all product files (${DFO_FILE_NAME}*fits) create new name:

2.1 use ${MCAL_CODE} to have 2letter instrument code (ins_code)
2.2 read pro.catg, map into sci_code
2.3 read obs.id
2.4 read PIPEFILE, use date_time part up to HH:MM:SS (datetime)
2.5 read config file for setup key, support strings, SPECSTRING
2.6 execute per key: read key, modify with SPECSTRING, add support string, add separator; repeat for all parameters listed in config file to get complete setup_code. The setup_code could be made of 0 or 1…N keys.
2.7 read extension from original file
2.8 construct new name: <ins_code>_<sci_code>_<obs_id>_<datetime>_<setup_code>.<ext>
Example: GI_SIFU_125427_2004-04-05T23:12:23 _Medusa1_L437.2nm_o2.fits

3. renaming

3.1 write per file "mv OLDNAME NEWNAME" into $DFO_SCI_DIR/<date>/rn_files
3.2 add line to set status flag to 'sci_Renamed'
3.3 if configured, execute $SPECIAL_PLUGIN
3.4 make rn_files executable but don't execute
3.5 finish

NOTE: contrary to CALIB, multiple files with the same name cannot occur in SCIENCE mode (unless the setup string is not properly defined).

Operational aspects

• The output file is called rn_file; check it before you execute. Keep it to go back if necessary.
• If configured, the tool is polite and warns you if files would be overwritten if rn_file would be executed (remember: the tool never does it).
• VERSIONing can be configured per pro.catg. You can e.g. decide to not version MBIA files (so always the latest one per directory survives), and to version all others.
• A configuration string can be defined either for all products in the same way, or for each pro.catg differently, or anything in between. The string is made from configurable header keywords, which can be massaged (e.g. shortened or translated) and decorated (e.g. adding BIN, underscores etc.). This information is configured by using pieces of shell code. The syntax conventions are explained in the config file.
• Note: it's good practice to keep the setup part (the one which is under your control) as short as possible since (a) these names show up in listings, and (b) certain tools like e.g. fitsort have an upper limit on the supported file length. There is no requirement to always have support strings, and there is no need to have more than the minimum number of instrument keys in the setup part.
• Reserved letters: first of the four letters in the code is reserved;
  P or M: calib products
  S: science products
  G: general (static) calib products
• In GEN mode, the setup string may not be defined at all (e.g., a line catalogue is generally valid). Then, you may use the setup string for a human readable tag, like e.g. GI_GCAT_021230A_line_catalog.fits.