From d54fe7c1f704a63824c5bfa0ece65245572e9b27 Mon Sep 17 00:00:00 2001 From: Joseph Hunkeler Date: Wed, 4 Mar 2015 21:21:30 -0500 Subject: Initial commit --- docs/INTRO_TO_CalFUSEv3.2.3 | 385 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 385 insertions(+) create mode 100644 docs/INTRO_TO_CalFUSEv3.2.3 (limited to 'docs/INTRO_TO_CalFUSEv3.2.3') diff --git a/docs/INTRO_TO_CalFUSEv3.2.3 b/docs/INTRO_TO_CalFUSEv3.2.3 new file mode 100644 index 0000000..0408121 --- /dev/null +++ b/docs/INTRO_TO_CalFUSEv3.2.3 @@ -0,0 +1,385 @@ +# Documentation for CalFUSE, the FUSE Calibration Pipeline +# +# Introduction to CalFUSE v3.2 +# +# Created 02/03/2004 +# Van Dixon +# +# History: +# 03/05/2004 Update for v3.0.5. +# Change name of POTHOLE column to QUALITY. +# 04/01/2004 Update for v3.0.6. +# Pipeline now runs on Linux and works on HIST data. +# Default output binning reduced to 0.013 A. +# 05/05/2004 Update for v3.0.7. +# Fixed bug in initial spectral-centroiding routine. +# Extraction routine now uses the same algorithm to +# compute the spectral, weights, and bkgd arrays. +# Modified boxcar extraction to improve background +# estimate. +# 08/20/2004 Update for v3.0.7 release. +# Pipeline now runs on Mac OS X 10.2 and higher. +# Screening now preceeds aperture centroiding. +# Raised minimum pulse height to 2 for all segments. +# 01/21/2005 Add a note about reading the new extracted +# spectral files into IDL. +# 04/19/2005 Update for v3.1.0 release. +# 06/06/2005 Update for v3.1.1 release. +# 06/10/2005 Clarify discussion of spectral file format. +# 06/15/2005 Update for v3.1.2 release. +# 06/24/2005 Add instructions for generating new jitter files. +# 07/12/2005 Modify instructions for generating new jitter files. +# 07/21/2005 Add information about post-processing tools. +# 09/07/2005 Update for v3.1.3 release. +# 12/02/2005 Update for v3.1.5 release. +# 12/03/2005 New changes to cf_mirror_motion and cf_grating_motion +# 01/24/2006 Update for v3.1.6 release. +# 02/02/2006 Update for v3.1.7. +# 05/24/2006 Update for v3.1.8. +# 11/08/2006 Update for v3.2.0. +# 01/15/2007 Move FIFO-overflow correction to cf_convert_to_farf. +# 08/24/2007 Update for v3.2.1. +# 02/15/2008 Update for v3.2.2. +# 12/05/2008 Update for v3.2.3. +# + + +I. Introduction + + CalFUSE v3.2.3 is the latest version of the FUSE data-reduction + pipeline. This package includes a new set of calibration files and + support for Linux 64-bit and Solaris 10 machines. + + For complete details about the program, see the paper, + "CalFUSE v3: A Data-Reduction Pipeline for the Far Ultraviolet + Spectroscopic Explorer" by Dixon et al. (2007) + (http://adsabs.harvard.edu/abs/2007PASP..119..527D). It describes + the CalFUSE pipeline, the instrument calibrations upon which it is + based, and the format of the resulting calibrated data files. + + +II. Recent Improvements + + New in CalFUSE v3.1.5 + + Bad-Pixel Correction: A new routine, cf_screen_bad_pixels, flags + photon events falling in known bad-pixel regions. + + Histogram Mode: For data obtained in HIST mode, + cf_count_rate_y_distort must use a single count-rate value for + the entire exposure. To better estimate it, we replace the + time-dependent FEC-rate array with its weighted mean value. + Rather than using different HIST_PAD values for each aperture, + cf_extraction_limits now pads apertures by 8 pixels (top and + bottom). + + Jitter Correction: To improve the jitter-correction algorithm, we + removed the data-screening functions from cf_satellite_jitter and + moved them to a new routine, cf_screen_jitter. + + New in CalFUSE v3.1.6 + + Burst Correction: Because cf_screen_bursts attempts to exclude + airglow photons from the calculated background, these photons must + be flagged before the routine is called. To this end, + cf_screen_airglow is now the first of the screening routines. + + Background Models: The model background consists of a + spatially-uniform component and a scattered-light image. As a + result of changes introduced in CalFUSE v3.1.3, the scattered-light + component was over-estimated, because the data were summed over a + larger region of the detector than the scattered-light image. Now, + the same limits are applied to both data and models. + + New in CalFUSE v3.1.7 + + Spectral Centroids: We tightened the limits over which + cf_find_spectra computes the spectral centroid to prevent a bright + star in a nearby aperture or the enhanced background near the + detector edge from pulling the extraction window off center. In + both cf_compute_y_centroid and cf_find_spectra, we returned the + parameter-file keywords EMAX_LIF and EMAX_SIC to their original + behavior: if the centroid differs from the default value by more + than this limit, the default value is used. + + New in CalFUSE v3.1.8 + + Astigmatism and Dispersion: The routine + cf_astigmatism_and_dispersion was divided into two programs, + cf_astigmatism and cf_dispersion. + + Fewer Warnings: The pipeline no longer issues WARNING messages + when a raw or intermediate data file is empty, when the jitter file + is missing, when the detector voltage is low, or when it cannot + perform optimal extraction. + + Fewer Bad Spectra: The program cf_extract_spectra no longer + discards (that is, sets to zero the flux and error arrays of) + spectra with non-zero values of EXP_STAT. + + New in CalFUSE v3.2.0 + + Time-Dependent X and Y Distortions: It turns out that the X and Y + coordinates computed by the detector electronics for a particular + spectral feature drift slowly with time. The new routine + cf_time_xy_distort corrects for this effect. + + Grating Motion: The grating-motion correction has been revised to + better correct the zero point of the wavelength scale. + + Jitter Files: Over the years, FUSE engineers have repeatedly + modified the spacecraft-control software, but the routine that + generates jitter files has not been kept up to date. As a result, + the program was not using all available information to determine + the quality and value of pointing errors. We have modified the + code to make use of this information. As a result, the meaning of + the jitter-file TRKFLG array has changed: values between 1 and 5 + reflect increasing levels of confidence in the quality of reported + pointing errors, a value of 0 indicates that pointing telemetry is + missing, and a value of -1 says that the pointing is unknown but + likely to be bad. Jitter files generated with the new routine + receive a version number >= 3.0. The use of old jitter files with + the new software is likely to yield unsatisfactory results. If you + have the housekeeping files, you can generate jitter files that are + consistent with the new pipeline. To do it, delete (or hide) the + old jitter files, then type + + prompt> cf_jitter P9990101001hskpf.fit P9990101001jitrf.fit + + for each exposure. Run the pipeline as usual. + + Jitter Correction: The jitter routines cf_screen_jitter and + cf_satellite_jitter have been updated to take advantage of the + information present in the new jitter files, and three new keywords + have been added to the parameter file. Times for which the X or Y + pointing errors are greater than DX_MAX or DY_MAX (default is 30") + are flagged as bad. Only pointing errors with associated tracking + flag values >= TRKFLG (default is 3) are considered reliable. + + FIFO Overflow: Before, the screening routines flagged times when + the detector count rate went to zero because of FIFO overflows. + This technique underestimated the actual target flux. Now, we + modify the IDS dead-time correction to account for photon losses + due to FIFO overflows. + + New in CalFUSE v3.2.1 + + Jitter Correction: We have modified the program cf_jitter, which + constructs jitter files, to be more robust when presented with + weird pointing data. + + SAA Contours: The size of the South Atlantic Anomaly varies on + timescales of weeks. If you find that the default contours (in the + file saac004.fit) are too small, causing times with elevated + background rates to be flagged as good, you can switch to the + smaller set of contours in the file saac005.fit by modifying the + file master_calib_file.dat. + + New in CalFUSE v3.2.2 + + New Platforms: CalFUSE now runs on 64-bit machines running Linux + and Sun workstations running Solaris v10. See the file + INSTALLING_CalFUSEv3.2.2 for installation details. + + We have used data from the last few months of the mission to + produce final versions of the PHAH_CAL, STIM_CAL, and PHAX_CAL + files. + + New in CalFUSE v3.2.3 + + The final version of the pipeline includes updates needed for the + final processing of the FUSE archive. First, airglow exposures get + special treatment: data taken close to the earth limb are flagged + but not rejected. Airglow exposures are ignored by cf_combine and + idf_combine unless the -a flag is set. Second, the usual limits on + the spectral centroids are not applied to HIST data, because + spectra from other apertures are not present to confuse the + centroiding algorithm. + + +III. Overview of Pipeline + + The following modules are called by the shell script calfuse.csh: + + cf_ttag_init, cf_hist_init + Converts raw-data file into an IDF containing photon-event + list, GTI's, and timeline table. Histogram data are converted + to a pseudo-time-tag format. + + cf_convert_to_farf + Corrects for detector deadtime. Transforms photon coordinates + into the FARF (flight alignment reference frame). + + cf_screen_photons + Checks data quality. Assigns status flag to each photon. + + cf_remove_motions + Corrects for mirror, grating, FPA, and spacecraft motions. + + cf_assign_wavelength + Applies astigmatism and Doppler corrections. + Assigns wavelength to each photon. + + cf_flux_calibrate + Converts WEIGHT to ERG/CM2 for each photon. + + cf_bad_pixels + Applies image-motion corrections to bad-pixel map. + + cf_extract_spectra + Extracts LiF and SiC spectra only for target aperture. + Wavelength array is user defined; default spacing is 0.013 A. + + +IV. Installing and Running the Pipeline + + CalFUSE is now available for Macs running OS X 10.2 and higher, + 64-bit Linux machines, and Sun workstations running Solaris 10. + + Please see the document INSTALLING_CalFUSEv3.2.2 for instructions. + + For both TTAG and HIST data, the command is + + prompt> calfuse P99901010011attagfraw.fit + prompt> calfuse P99901010011ahistfraw.fit + + The pipeline expects to find all data files (*raw.fit, *jitrf.fit, + *hskpf.fit) in the current directory. It operates on one detector + segment at a time. + + CalFUSE v3 produces the following output files: an intermediate + data file (IDF), discussed below; a bad-pixel map (BPM), with a + format similar to the IDF; two extracted spectral files, one for + each of the LiF and SiC channels; and (if IDL is installed on your + machine) a pair of GIF or JPEG files, one an image of the detector + and the other a count-rate plot. + + +V. The Intermediate Data File (IDF) + + The photon-list files consist of three FITS binary tables. The + first contains the photon events themselves. The pipeline does not + discard any events, but uses a series of flags to indicate whether + particular photons violate pulse-height limits, limb-angle + constraints, etc. The IDL tool cf_edit (discussed below) allows + users to modify these flags and combine IDF files from multiple + exposures. + + The second extension to the data files is a list of good-time + intervals (GTI's). They are not used by the extraction routine, + but may prove helpful to pipeline users. + + The third extension is called a timeline table. For each second + during the exposure, it lists a dozen parameters, including count + rate, day-night status, and detector voltage. The pipeline uses + this list to set the status flags for each photon. + + Note: because of the new file format employed by IDF files, + extensions 1 and 3 must be read using the /fscale keyword. + + idl> a=mrdfits('P99901010011attagfidf.fit',1,/fscale) + idl> help,a,/str + + Elements of individual arrays must be addressed using the syntax + + idl> print,a.time[3:30] -- not a[3:30].time + + For details about the format of the IDF, please see the document + + [calfuse_directory]/v3.1/src/README_FILES/IDF_Format.txt + + +VI. Manipulating the IDF with cf_edit + + CF_EDIT is an Interactive Data Language (IDL) visualization tool + for the examination and modification of FUSE IDF files containing + flux- and wavelength-calibrated photon-event lists. Users without + IDL licenses may use the IDL "Virtual Machine" version of the + tool for free. Complete instructions for installing and using + cf_edit are available at this URL: + + http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html + + More information on the IDL Virtual Machine is available here: + + http://www.rsinc.com/idlvm/index.asp + + +VII. The Output Spectral File + + The format of the output spectral files is somewhat changed from + earlier versions of CalFUSE. + + float WAVE Wavelength (Angstroms) + float FLUX Flux (erg/cm2/s/A) + float ERROR Gaussian errors (erg/cm2/s/A) + int COUNTS Raw counts in extraction window + float WEIGHTS Raw counts corrected for deadtime, etc. + float BKGD Estimated background in extraction window + short QUALITY Percentage of window used for extraction + + Note that the QUALITY array no longer lists the number of Y pixels + that contribute to each output X pixel. Instead, it is the fraction + of the extraction window considered to contain valid data, expressed + as a percentage (0 is all bad; 100 is all good). + + Do-it-yourself-ers can use the following recipe to generate a + flux-calibrated spectrum: + + TARGET_COUNTS = WEIGHTS - BKGD + TARGET_FLUX = TARGET_COUNTS * HC / LAMBDA / AEFF / EXPTIME / WPC + + where AEFF = effective area in cm^2 and WPC = size of output pixel in + Angstroms. + +VIII. Additional Files Available from MAST + + CalFUSE produces eight extracted spectral (*fcal.fit) files for + each exposure. We combine them into a set of three + observation-level files for submission to the MAST archive. + Depending on the target and the questions that you are trying to + answer, you may find that these files are of sufficient fidelity + for scientific investigation. Here's a brief description of their + contents: + + ALL: For each channel (LiF 1A, SiC 1A, etc.), we combine data from + all exposures in the observation into a single spectrum. If the + individual spectra are bright enough, we cross correlate and shift + before combining them. If the spectra are too faint, we combine the + individual IDF files and extract a single spectrum to optimize the + background model. The combined spectrum for each channel is stored + in a separate extension. + + Note: The cataloging software used by MAST requires the presence of + an ALL file for each exposure, not just for the entire observation. + We now generate such a file, but it contains no data, only a FITS + file header. The observation-level ALL files discussed above can be + identified by the string "00000all" in their names. + + ANO (all, night-only): Same format as the ALL files, but using only + data obtained during the night-time portion of each exposure. These + files are generated only for TTAG data, and only if EXPNIGHT > 0. + + NVO (National Virtual Observatory): Contains a single spectrum + spanning the entire FUSE wavelength range. The spectrum is + assembled by cutting and pasting segments from the most sensitive + channel at each wavelength. Segments are shifted to match LiF 1A + between 1045 and 1070 E. Columns are WAVE, FLUX, and ERROR and are + stored in a single binary table extension. + +IX. Data-Analysis Tools + + A variety of IDL routines to display and manipulate FUSE data are + available from the FUSE IDL Tools Reference Page: + + http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html + + Data-analysis tools distributed with the CalFUSE pipeline are + reviewed in the document FUSE Tools in C: + + http://fuse.pha.jhu.edu/analysis/fuse_toolbox.html + + A subset of these programs, designed specifically for the + manipulation of IDF files, is described in the IDF Cookbook: + + http://fuse.pha.jhu.edu/analysis/idfcook/index.html -- cgit