Massive rework

1 files changed, 385 insertions, 0 deletions

diff --git a/doc/INTRO_TO_CalFUSEv3.2.3 b/doc/INTRO_TO_CalFUSEv3.2.3
new file mode 100644
index 0000000..0408121
--- /dev/null
+++ b/doc/INTRO_TO_CalFUSEv3.2.3
@@ -0,0 +1,385 @@
+# Documentation for CalFUSE, the FUSE Calibration Pipeline
+#
+# Introduction to CalFUSE v3.2
+#
+# Created 02/03/2004
+# Van Dixon
+#
+# History:
+#   03/05/2004	Update for v3.0.5.
+#		Change name of POTHOLE column to QUALITY.
+#   04/01/2004  Update for v3.0.6.
+#		Pipeline now runs on Linux and works on HIST data.
+#		Default output binning reduced to 0.013 A.
+#   05/05/2004  Update for v3.0.7.
+#		Fixed bug in initial spectral-centroiding routine.
+#		Extraction routine now uses the same algorithm to
+#		compute the spectral, weights, and bkgd arrays.
+#		Modified boxcar extraction to improve background
+#		estimate.
+#   08/20/2004  Update for v3.0.7 release.
+#		Pipeline now runs on Mac OS X 10.2 and higher.
+#		Screening now preceeds aperture centroiding.
+#		Raised minimum pulse height to 2 for all segments.
+#   01/21/2005  Add a note about reading the new extracted
+#		spectral files into IDL.
+#   04/19/2005	Update for v3.1.0 release.
+#   06/06/2005	Update for v3.1.1 release.
+#   06/10/2005	Clarify discussion of spectral file format.
+#   06/15/2005	Update for v3.1.2 release.
+#   06/24/2005	Add instructions for generating new jitter files.
+#   07/12/2005	Modify instructions for generating new jitter files.
+#   07/21/2005	Add information about post-processing tools.
+#   09/07/2005	Update for v3.1.3 release.
+#   12/02/2005  Update for v3.1.5 release.
+#   12/03/2005  New changes to cf_mirror_motion and cf_grating_motion
+#   01/24/2006  Update for v3.1.6 release.
+#   02/02/2006  Update for v3.1.7.
+#   05/24/2006  Update for v3.1.8.
+#   11/08/2006  Update for v3.2.0.
+#   01/15/2007	Move FIFO-overflow correction to cf_convert_to_farf.
+#   08/24/2007  Update for v3.2.1.
+#   02/15/2008  Update for v3.2.2.
+#   12/05/2008  Update for v3.2.3.
+#
+
+
+I.  Introduction
+
+    CalFUSE v3.2.3 is the latest version of the FUSE data-reduction
+    pipeline.  This package includes a new set of calibration files and
+    support for Linux 64-bit and Solaris 10 machines.
+
+    For complete details about the program, see the paper, 
+    "CalFUSE v3: A Data-Reduction Pipeline for the Far Ultraviolet
+    Spectroscopic Explorer" by Dixon et al. (2007)
+    (http://adsabs.harvard.edu/abs/2007PASP..119..527D).  It describes
+    the CalFUSE pipeline, the instrument calibrations upon which it is
+    based, and the format of the resulting calibrated data files.
+
+
+II.  Recent Improvements
+
+    New in CalFUSE v3.1.5
+    
+    Bad-Pixel Correction:  A new routine, cf_screen_bad_pixels, flags
+    photon events falling in known bad-pixel regions.
+
+    Histogram Mode: For data obtained in HIST mode,
+    cf_count_rate_y_distort must use a single count-rate value for
+    the entire exposure.  To better estimate it, we replace the
+    time-dependent FEC-rate array with its weighted mean value.
+    Rather than using different HIST_PAD values for each aperture,
+    cf_extraction_limits now pads apertures by 8 pixels (top and
+    bottom).
+
+    Jitter Correction: To improve the jitter-correction algorithm, we
+    removed the data-screening functions from cf_satellite_jitter and
+    moved them to a new routine, cf_screen_jitter.
+
+    New in CalFUSE v3.1.6
+
+    Burst Correction:  Because cf_screen_bursts attempts to exclude
+    airglow photons from the calculated background, these photons must
+    be flagged before the routine is called.  To this end,
+    cf_screen_airglow is now the first of the screening routines.
+
+    Background Models:  The model background consists of a
+    spatially-uniform component and a scattered-light image.  As a
+    result of changes introduced in CalFUSE v3.1.3, the scattered-light
+    component was over-estimated, because the data were summed over a
+    larger region of the detector than the scattered-light image.  Now,
+    the same limits are applied to both data and models.
+
+    New in CalFUSE v3.1.7
+
+    Spectral Centroids:  We tightened the limits over which
+    cf_find_spectra computes the spectral centroid to prevent a bright
+    star in a nearby aperture or the enhanced background near the
+    detector edge from pulling the extraction window off center.  In
+    both cf_compute_y_centroid and cf_find_spectra, we returned the
+    parameter-file keywords EMAX_LIF and EMAX_SIC to their original
+    behavior:  if the centroid differs from the default value by more
+    than this limit, the default value is used.
+
+    New in CalFUSE v3.1.8
+
+    Astigmatism and Dispersion:  The routine
+    cf_astigmatism_and_dispersion was divided into two programs,
+    cf_astigmatism and cf_dispersion.
+
+    Fewer Warnings:  The pipeline no longer issues WARNING messages
+    when a raw or intermediate data file is empty, when the jitter file
+    is missing, when the detector voltage is low, or when it cannot
+    perform optimal extraction.
+
+    Fewer Bad Spectra:  The program cf_extract_spectra no longer
+    discards (that is, sets to zero the flux and error arrays of)
+    spectra with non-zero values of EXP_STAT.
+
+    New in CalFUSE v3.2.0
+
+    Time-Dependent X and Y Distortions:  It turns out that the X and Y
+    coordinates computed by the detector electronics for a particular
+    spectral feature drift slowly with time.  The new routine
+    cf_time_xy_distort corrects for this effect.
+
+    Grating Motion:  The grating-motion correction has been revised to
+    better correct the zero point of the wavelength scale.
+
+    Jitter Files:  Over the years, FUSE engineers have repeatedly
+    modified the spacecraft-control software, but the routine that
+    generates jitter files has not been kept up to date.  As a result,
+    the program was not using all available information to determine
+    the quality and value of pointing errors.  We have modified the
+    code to make use of this information.  As a result, the meaning of
+    the jitter-file TRKFLG array has changed:  values between 1 and 5
+    reflect increasing levels of confidence in the quality of reported
+    pointing errors, a value of 0 indicates that pointing telemetry is
+    missing, and a value of -1 says that the pointing is unknown but
+    likely to be bad.  Jitter files generated with the new routine
+    receive a version number >= 3.0.  The use of old jitter files with
+    the new software is likely to yield unsatisfactory results.  If you
+    have the housekeeping files, you can generate jitter files that are
+    consistent with the new pipeline.  To do it, delete (or hide) the
+    old jitter files, then type
+
+    prompt> cf_jitter P9990101001hskpf.fit P9990101001jitrf.fit
+
+    for each exposure.  Run the pipeline as usual.
+
+    Jitter Correction:  The jitter routines cf_screen_jitter and
+    cf_satellite_jitter have been updated to take advantage of the
+    information present in the new jitter files, and three new keywords
+    have been added to the parameter file.  Times for which the X or Y
+    pointing errors are greater than DX_MAX or DY_MAX (default is 30")
+    are flagged as bad.  Only pointing errors with associated tracking
+    flag values >= TRKFLG (default is 3) are considered reliable.
+
+    FIFO Overflow:  Before, the screening routines flagged times when
+    the detector count rate went to zero because of FIFO overflows.
+    This technique underestimated the actual target flux.  Now, we
+    modify the IDS dead-time correction to account for photon losses
+    due to FIFO overflows.
+
+    New in CalFUSE v3.2.1
+
+    Jitter Correction:  We have modified the program cf_jitter, which
+    constructs jitter files, to be more robust when presented with
+    weird pointing data.
+
+    SAA Contours: The size of the South Atlantic Anomaly varies on
+    timescales of weeks.  If you find that the default contours (in the
+    file saac004.fit) are too small, causing times with elevated
+    background rates to be flagged as good, you can switch to the
+    smaller set of contours in the file saac005.fit by modifying the
+    file master_calib_file.dat.
+
+    New in CalFUSE v3.2.2
+
+    New Platforms:  CalFUSE now runs on 64-bit machines running Linux
+    and Sun workstations running Solaris v10.  See the file
+    INSTALLING_CalFUSEv3.2.2 for installation details.
+
+    We have used data from the last few months of the mission to
+    produce final versions of the PHAH_CAL, STIM_CAL, and PHAX_CAL
+    files.
+
+    New in CalFUSE v3.2.3
+
+    The final version of the pipeline includes updates needed for the
+    final processing of the FUSE archive.  First, airglow exposures get
+    special treatment: data taken close to the earth limb are flagged
+    but not rejected.  Airglow exposures are ignored by cf_combine and
+    idf_combine unless the -a flag is set.  Second, the usual limits on
+    the spectral centroids are not applied to HIST data, because
+    spectra from other apertures are not present to confuse the
+    centroiding algorithm.
+
+
+III. Overview of Pipeline
+
+    The following modules are called by the shell script calfuse.csh:
+
+    cf_ttag_init, cf_hist_init
+	Converts raw-data file into an IDF containing photon-event
+	list, GTI's, and timeline table.  Histogram data are converted
+	to a pseudo-time-tag format.
+
+    cf_convert_to_farf
+	Corrects for detector deadtime.  Transforms photon coordinates
+	into the FARF (flight alignment reference frame).
+
+    cf_screen_photons
+	Checks data quality.  Assigns status flag to each photon.
+
+    cf_remove_motions
+	Corrects for mirror, grating, FPA, and spacecraft motions.
+
+    cf_assign_wavelength
+	Applies astigmatism and Doppler corrections.
+	Assigns wavelength to each photon.
+
+    cf_flux_calibrate
+	Converts WEIGHT to ERG/CM2 for each photon.
+
+    cf_bad_pixels
+	Applies image-motion corrections to bad-pixel map.
+
+    cf_extract_spectra
+	Extracts LiF and SiC spectra only for target aperture.
+	Wavelength array is user defined; default spacing is 0.013 A.
+
+
+IV.  Installing and Running the Pipeline
+
+    CalFUSE is now available for Macs running OS X 10.2 and higher,
+    64-bit Linux machines, and Sun workstations running Solaris 10.
+
+    Please see the document INSTALLING_CalFUSEv3.2.2 for instructions.
+
+    For both TTAG and HIST data, the command is 
+
+	prompt> calfuse P99901010011attagfraw.fit
+	prompt> calfuse P99901010011ahistfraw.fit
+
+    The pipeline expects to find all data files (*raw.fit, *jitrf.fit,
+    *hskpf.fit) in the current directory.  It operates on one detector
+    segment at a time.
+
+    CalFUSE v3 produces the following output files: an intermediate
+    data file (IDF), discussed below; a bad-pixel map (BPM), with a
+    format similar to the IDF; two extracted spectral files, one for
+    each of the LiF and SiC channels; and (if IDL is installed on your
+    machine) a pair of GIF or JPEG files, one an image of the detector
+    and the other a count-rate plot.
+
+
+V.  The Intermediate Data File (IDF)
+
+    The photon-list files consist of three FITS binary tables.  The
+    first contains the photon events themselves.  The pipeline does not
+    discard any events, but uses a series of flags to indicate whether
+    particular photons violate pulse-height limits, limb-angle
+    constraints, etc.  The IDL tool cf_edit (discussed below) allows
+    users to modify these flags and combine IDF files from multiple
+    exposures.
+
+    The second extension to the data files is a list of good-time
+    intervals (GTI's).  They are not used by the extraction routine,
+    but may prove helpful to pipeline users.
+
+    The third extension is called a timeline table.  For each second
+    during the exposure, it lists a dozen parameters, including count
+    rate, day-night status, and detector voltage.  The pipeline uses
+    this list to set the status flags for each photon.
+
+    Note: because of the new file format employed by IDF files, 
+    extensions 1 and 3 must be read using the /fscale keyword.
+
+    idl> a=mrdfits('P99901010011attagfidf.fit',1,/fscale)
+    idl> help,a,/str
+
+    Elements of individual arrays must be addressed using the syntax
+
+    idl> print,a.time[3:30]	-- not a[3:30].time
+
+    For details about the format of the IDF, please see the document
+
+    [calfuse_directory]/v3.1/src/README_FILES/IDF_Format.txt 
+
+
+VI. Manipulating the IDF with cf_edit
+
+    CF_EDIT is an Interactive Data Language (IDL) visualization tool
+    for the examination and modification of FUSE IDF files containing
+    flux- and wavelength-calibrated photon-event lists.  Users without
+    IDL licenses may use the IDL "Virtual Machine" version of the
+    tool for free.  Complete instructions for installing and using
+    cf_edit are available at this URL:
+
+    http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html
+
+    More information on the IDL Virtual Machine is available here:
+
+    http://www.rsinc.com/idlvm/index.asp
+
+
+VII. The Output Spectral File
+
+    The format of the output spectral files is somewhat changed from
+    earlier versions of CalFUSE.
+
+    float     WAVE       Wavelength (Angstroms)
+    float     FLUX       Flux (erg/cm2/s/A)
+    float     ERROR      Gaussian errors (erg/cm2/s/A)
+    int       COUNTS     Raw counts in extraction window
+    float     WEIGHTS    Raw counts corrected for deadtime, etc.
+    float     BKGD       Estimated background in extraction window
+    short     QUALITY    Percentage of window used for extraction
+
+    Note that the QUALITY array no longer lists the number of Y pixels
+    that contribute to each output X pixel.  Instead, it is the fraction
+    of the extraction window considered to contain valid data, expressed
+    as a percentage (0 is all bad; 100 is all good).
+
+    Do-it-yourself-ers can use the following recipe to generate a
+    flux-calibrated spectrum:
+
+    TARGET_COUNTS = WEIGHTS - BKGD                                          
+    TARGET_FLUX = TARGET_COUNTS * HC / LAMBDA / AEFF / EXPTIME / WPC
+
+    where AEFF = effective area in cm^2 and WPC = size of output pixel in
+    Angstroms.
+
+VIII. Additional Files Available from MAST
+
+    CalFUSE produces eight extracted spectral (*fcal.fit) files for
+    each exposure. We combine them into a set of three
+    observation-level files for submission to the MAST archive.
+    Depending on the target and the questions that you are trying to
+    answer, you may find that these files are of sufficient fidelity
+    for scientific investigation. Here's a brief description of their
+    contents:
+
+    ALL: For each channel (LiF 1A, SiC 1A, etc.), we combine data from
+    all exposures in the observation into a single spectrum. If the
+    individual spectra are bright enough, we cross correlate and shift
+    before combining them. If the spectra are too faint, we combine the
+    individual IDF files and extract a single spectrum to optimize the
+    background model. The combined spectrum for each channel is stored
+    in a separate extension.
+
+    Note: The cataloging software used by MAST requires the presence of
+    an ALL file for each exposure, not just for the entire observation.
+    We now generate such a file, but it contains no data, only a FITS
+    file header. The observation-level ALL files discussed above can be
+    identified by the string "00000all" in their names.
+
+    ANO (all, night-only): Same format as the ALL files, but using only
+    data obtained during the night-time portion of each exposure. These
+    files are generated only for TTAG data, and only if EXPNIGHT > 0.
+
+    NVO (National Virtual Observatory): Contains a single spectrum
+    spanning the entire FUSE wavelength range. The spectrum is
+    assembled by cutting and pasting segments from the most sensitive
+    channel at each wavelength. Segments are shifted to match LiF 1A
+    between 1045 and 1070 E. Columns are WAVE, FLUX, and ERROR and are
+    stored in a single binary table extension.
+
+IX. Data-Analysis Tools
+
+    A variety of IDL routines to display and manipulate FUSE data are
+    available from the FUSE IDL Tools Reference Page:
+
+    http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html
+
+    Data-analysis tools distributed with the CalFUSE pipeline are
+    reviewed in the document FUSE Tools in C:
+
+    http://fuse.pha.jhu.edu/analysis/fuse_toolbox.html
+
+    A subset of these programs, designed specifically for the
+    manipulation of IDF files, is described in the IDF Cookbook:
+
+    http://fuse.pha.jhu.edu/analysis/idfcook/index.html