# Documentation for CalFUSE, the FUSE Calibration Pipeline # # Introduction to CalFUSE v3.2 # # Created 02/03/2004 # Van Dixon # # History: # 03/05/2004 Update for v3.0.5. # Change name of POTHOLE column to QUALITY. # 04/01/2004 Update for v3.0.6. # Pipeline now runs on Linux and works on HIST data. # Default output binning reduced to 0.013 A. # 05/05/2004 Update for v3.0.7. # Fixed bug in initial spectral-centroiding routine. # Extraction routine now uses the same algorithm to # compute the spectral, weights, and bkgd arrays. # Modified boxcar extraction to improve background # estimate. # 08/20/2004 Update for v3.0.7 release. # Pipeline now runs on Mac OS X 10.2 and higher. # Screening now preceeds aperture centroiding. # Raised minimum pulse height to 2 for all segments. # 01/21/2005 Add a note about reading the new extracted # spectral files into IDL. # 04/19/2005 Update for v3.1.0 release. # 06/06/2005 Update for v3.1.1 release. # 06/10/2005 Clarify discussion of spectral file format. # 06/15/2005 Update for v3.1.2 release. # 06/24/2005 Add instructions for generating new jitter files. # 07/12/2005 Modify instructions for generating new jitter files. # 07/21/2005 Add information about post-processing tools. # 09/07/2005 Update for v3.1.3 release. # 12/02/2005 Update for v3.1.5 release. # 12/03/2005 New changes to cf_mirror_motion and cf_grating_motion # 01/24/2006 Update for v3.1.6 release. # 02/02/2006 Update for v3.1.7. # 05/24/2006 Update for v3.1.8. # 11/08/2006 Update for v3.2.0. # 01/15/2007 Move FIFO-overflow correction to cf_convert_to_farf. # 08/24/2007 Update for v3.2.1. # 02/15/2008 Update for v3.2.2. # 12/05/2008 Update for v3.2.3. # I. Introduction CalFUSE v3.2.3 is the latest version of the FUSE data-reduction pipeline. This package includes a new set of calibration files and support for Linux 64-bit and Solaris 10 machines. For complete details about the program, see the paper, "CalFUSE v3: A Data-Reduction Pipeline for the Far Ultraviolet Spectroscopic Explorer" by Dixon et al. (2007) (http://adsabs.harvard.edu/abs/2007PASP..119..527D). It describes the CalFUSE pipeline, the instrument calibrations upon which it is based, and the format of the resulting calibrated data files. II. Recent Improvements New in CalFUSE v3.1.5 Bad-Pixel Correction: A new routine, cf_screen_bad_pixels, flags photon events falling in known bad-pixel regions. Histogram Mode: For data obtained in HIST mode, cf_count_rate_y_distort must use a single count-rate value for the entire exposure. To better estimate it, we replace the time-dependent FEC-rate array with its weighted mean value. Rather than using different HIST_PAD values for each aperture, cf_extraction_limits now pads apertures by 8 pixels (top and bottom). Jitter Correction: To improve the jitter-correction algorithm, we removed the data-screening functions from cf_satellite_jitter and moved them to a new routine, cf_screen_jitter. New in CalFUSE v3.1.6 Burst Correction: Because cf_screen_bursts attempts to exclude airglow photons from the calculated background, these photons must be flagged before the routine is called. To this end, cf_screen_airglow is now the first of the screening routines. Background Models: The model background consists of a spatially-uniform component and a scattered-light image. As a result of changes introduced in CalFUSE v3.1.3, the scattered-light component was over-estimated, because the data were summed over a larger region of the detector than the scattered-light image. Now, the same limits are applied to both data and models. New in CalFUSE v3.1.7 Spectral Centroids: We tightened the limits over which cf_find_spectra computes the spectral centroid to prevent a bright star in a nearby aperture or the enhanced background near the detector edge from pulling the extraction window off center. In both cf_compute_y_centroid and cf_find_spectra, we returned the parameter-file keywords EMAX_LIF and EMAX_SIC to their original behavior: if the centroid differs from the default value by more than this limit, the default value is used. New in CalFUSE v3.1.8 Astigmatism and Dispersion: The routine cf_astigmatism_and_dispersion was divided into two programs, cf_astigmatism and cf_dispersion. Fewer Warnings: The pipeline no longer issues WARNING messages when a raw or intermediate data file is empty, when the jitter file is missing, when the detector voltage is low, or when it cannot perform optimal extraction. Fewer Bad Spectra: The program cf_extract_spectra no longer discards (that is, sets to zero the flux and error arrays of) spectra with non-zero values of EXP_STAT. New in CalFUSE v3.2.0 Time-Dependent X and Y Distortions: It turns out that the X and Y coordinates computed by the detector electronics for a particular spectral feature drift slowly with time. The new routine cf_time_xy_distort corrects for this effect. Grating Motion: The grating-motion correction has been revised to better correct the zero point of the wavelength scale. Jitter Files: Over the years, FUSE engineers have repeatedly modified the spacecraft-control software, but the routine that generates jitter files has not been kept up to date. As a result, the program was not using all available information to determine the quality and value of pointing errors. We have modified the code to make use of this information. As a result, the meaning of the jitter-file TRKFLG array has changed: values between 1 and 5 reflect increasing levels of confidence in the quality of reported pointing errors, a value of 0 indicates that pointing telemetry is missing, and a value of -1 says that the pointing is unknown but likely to be bad. Jitter files generated with the new routine receive a version number >= 3.0. The use of old jitter files with the new software is likely to yield unsatisfactory results. If you have the housekeeping files, you can generate jitter files that are consistent with the new pipeline. To do it, delete (or hide) the old jitter files, then type prompt> cf_jitter P9990101001hskpf.fit P9990101001jitrf.fit for each exposure. Run the pipeline as usual. Jitter Correction: The jitter routines cf_screen_jitter and cf_satellite_jitter have been updated to take advantage of the information present in the new jitter files, and three new keywords have been added to the parameter file. Times for which the X or Y pointing errors are greater than DX_MAX or DY_MAX (default is 30") are flagged as bad. Only pointing errors with associated tracking flag values >= TRKFLG (default is 3) are considered reliable. FIFO Overflow: Before, the screening routines flagged times when the detector count rate went to zero because of FIFO overflows. This technique underestimated the actual target flux. Now, we modify the IDS dead-time correction to account for photon losses due to FIFO overflows. New in CalFUSE v3.2.1 Jitter Correction: We have modified the program cf_jitter, which constructs jitter files, to be more robust when presented with weird pointing data. SAA Contours: The size of the South Atlantic Anomaly varies on timescales of weeks. If you find that the default contours (in the file saac004.fit) are too small, causing times with elevated background rates to be flagged as good, you can switch to the smaller set of contours in the file saac005.fit by modifying the file master_calib_file.dat. New in CalFUSE v3.2.2 New Platforms: CalFUSE now runs on 64-bit machines running Linux and Sun workstations running Solaris v10. See the file INSTALLING_CalFUSEv3.2.2 for installation details. We have used data from the last few months of the mission to produce final versions of the PHAH_CAL, STIM_CAL, and PHAX_CAL files. New in CalFUSE v3.2.3 The final version of the pipeline includes updates needed for the final processing of the FUSE archive. First, airglow exposures get special treatment: data taken close to the earth limb are flagged but not rejected. Airglow exposures are ignored by cf_combine and idf_combine unless the -a flag is set. Second, the usual limits on the spectral centroids are not applied to HIST data, because spectra from other apertures are not present to confuse the centroiding algorithm. III. Overview of Pipeline The following modules are called by the shell script calfuse.csh: cf_ttag_init, cf_hist_init Converts raw-data file into an IDF containing photon-event list, GTI's, and timeline table. Histogram data are converted to a pseudo-time-tag format. cf_convert_to_farf Corrects for detector deadtime. Transforms photon coordinates into the FARF (flight alignment reference frame). cf_screen_photons Checks data quality. Assigns status flag to each photon. cf_remove_motions Corrects for mirror, grating, FPA, and spacecraft motions. cf_assign_wavelength Applies astigmatism and Doppler corrections. Assigns wavelength to each photon. cf_flux_calibrate Converts WEIGHT to ERG/CM2 for each photon. cf_bad_pixels Applies image-motion corrections to bad-pixel map. cf_extract_spectra Extracts LiF and SiC spectra only for target aperture. Wavelength array is user defined; default spacing is 0.013 A. IV. Installing and Running the Pipeline CalFUSE is now available for Macs running OS X 10.2 and higher, 64-bit Linux machines, and Sun workstations running Solaris 10. Please see the document INSTALLING_CalFUSEv3.2.2 for instructions. For both TTAG and HIST data, the command is prompt> calfuse P99901010011attagfraw.fit prompt> calfuse P99901010011ahistfraw.fit The pipeline expects to find all data files (*raw.fit, *jitrf.fit, *hskpf.fit) in the current directory. It operates on one detector segment at a time. CalFUSE v3 produces the following output files: an intermediate data file (IDF), discussed below; a bad-pixel map (BPM), with a format similar to the IDF; two extracted spectral files, one for each of the LiF and SiC channels; and (if IDL is installed on your machine) a pair of GIF or JPEG files, one an image of the detector and the other a count-rate plot. V. The Intermediate Data File (IDF) The photon-list files consist of three FITS binary tables. The first contains the photon events themselves. The pipeline does not discard any events, but uses a series of flags to indicate whether particular photons violate pulse-height limits, limb-angle constraints, etc. The IDL tool cf_edit (discussed below) allows users to modify these flags and combine IDF files from multiple exposures. The second extension to the data files is a list of good-time intervals (GTI's). They are not used by the extraction routine, but may prove helpful to pipeline users. The third extension is called a timeline table. For each second during the exposure, it lists a dozen parameters, including count rate, day-night status, and detector voltage. The pipeline uses this list to set the status flags for each photon. Note: because of the new file format employed by IDF files, extensions 1 and 3 must be read using the /fscale keyword. idl> a=mrdfits('P99901010011attagfidf.fit',1,/fscale) idl> help,a,/str Elements of individual arrays must be addressed using the syntax idl> print,a.time[3:30] -- not a[3:30].time For details about the format of the IDF, please see the document [calfuse_directory]/v3.1/src/README_FILES/IDF_Format.txt VI. Manipulating the IDF with cf_edit CF_EDIT is an Interactive Data Language (IDL) visualization tool for the examination and modification of FUSE IDF files containing flux- and wavelength-calibrated photon-event lists. Users without IDL licenses may use the IDL "Virtual Machine" version of the tool for free. Complete instructions for installing and using cf_edit are available at this URL: http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html More information on the IDL Virtual Machine is available here: http://www.rsinc.com/idlvm/index.asp VII. The Output Spectral File The format of the output spectral files is somewhat changed from earlier versions of CalFUSE. float WAVE Wavelength (Angstroms) float FLUX Flux (erg/cm2/s/A) float ERROR Gaussian errors (erg/cm2/s/A) int COUNTS Raw counts in extraction window float WEIGHTS Raw counts corrected for deadtime, etc. float BKGD Estimated background in extraction window short QUALITY Percentage of window used for extraction Note that the QUALITY array no longer lists the number of Y pixels that contribute to each output X pixel. Instead, it is the fraction of the extraction window considered to contain valid data, expressed as a percentage (0 is all bad; 100 is all good). Do-it-yourself-ers can use the following recipe to generate a flux-calibrated spectrum: TARGET_COUNTS = WEIGHTS - BKGD TARGET_FLUX = TARGET_COUNTS * HC / LAMBDA / AEFF / EXPTIME / WPC where AEFF = effective area in cm^2 and WPC = size of output pixel in Angstroms. VIII. Additional Files Available from MAST CalFUSE produces eight extracted spectral (*fcal.fit) files for each exposure. We combine them into a set of three observation-level files for submission to the MAST archive. Depending on the target and the questions that you are trying to answer, you may find that these files are of sufficient fidelity for scientific investigation. Here's a brief description of their contents: ALL: For each channel (LiF 1A, SiC 1A, etc.), we combine data from all exposures in the observation into a single spectrum. If the individual spectra are bright enough, we cross correlate and shift before combining them. If the spectra are too faint, we combine the individual IDF files and extract a single spectrum to optimize the background model. The combined spectrum for each channel is stored in a separate extension. Note: The cataloging software used by MAST requires the presence of an ALL file for each exposure, not just for the entire observation. We now generate such a file, but it contains no data, only a FITS file header. The observation-level ALL files discussed above can be identified by the string "00000all" in their names. ANO (all, night-only): Same format as the ALL files, but using only data obtained during the night-time portion of each exposure. These files are generated only for TTAG data, and only if EXPNIGHT > 0. NVO (National Virtual Observatory): Contains a single spectrum spanning the entire FUSE wavelength range. The spectrum is assembled by cutting and pasting segments from the most sensitive channel at each wavelength. Segments are shifted to match LiF 1A between 1045 and 1070 E. Columns are WAVE, FLUX, and ERROR and are stored in a single binary table extension. IX. Data-Analysis Tools A variety of IDL routines to display and manipulate FUSE data are available from the FUSE IDL Tools Reference Page: http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html Data-analysis tools distributed with the CalFUSE pipeline are reviewed in the document FUSE Tools in C: http://fuse.pha.jhu.edu/analysis/fuse_toolbox.html A subset of these programs, designed specifically for the manipulation of IDF files, is described in the IDF Cookbook: http://fuse.pha.jhu.edu/analysis/idfcook/index.html