path: root/docs/INTRO_TO_CalFUSEv3.2.3

# Documentation for CalFUSE, the FUSE Calibration Pipeline
#
# Introduction to CalFUSE v3.2
#
# Created 02/03/2004
# Van Dixon
#
# History:
#   03/05/2004	Update for v3.0.5.
#		Change name of POTHOLE column to QUALITY.
#   04/01/2004  Update for v3.0.6.
#		Pipeline now runs on Linux and works on HIST data.
#		Default output binning reduced to 0.013 A.
#   05/05/2004  Update for v3.0.7.
#		Fixed bug in initial spectral-centroiding routine.
#		Extraction routine now uses the same algorithm to
#		compute the spectral, weights, and bkgd arrays.
#		Modified boxcar extraction to improve background
#		estimate.
#   08/20/2004  Update for v3.0.7 release.
#		Pipeline now runs on Mac OS X 10.2 and higher.
#		Screening now preceeds aperture centroiding.
#		Raised minimum pulse height to 2 for all segments.
#   01/21/2005  Add a note about reading the new extracted
#		spectral files into IDL.
#   04/19/2005	Update for v3.1.0 release.
#   06/06/2005	Update for v3.1.1 release.
#   06/10/2005	Clarify discussion of spectral file format.
#   06/15/2005	Update for v3.1.2 release.
#   06/24/2005	Add instructions for generating new jitter files.
#   07/12/2005	Modify instructions for generating new jitter files.
#   07/21/2005	Add information about post-processing tools.
#   09/07/2005	Update for v3.1.3 release.
#   12/02/2005  Update for v3.1.5 release.
#   12/03/2005  New changes to cf_mirror_motion and cf_grating_motion
#   01/24/2006  Update for v3.1.6 release.
#   02/02/2006  Update for v3.1.7.
#   05/24/2006  Update for v3.1.8.
#   11/08/2006  Update for v3.2.0.
#   01/15/2007	Move FIFO-overflow correction to cf_convert_to_farf.
#   08/24/2007  Update for v3.2.1.
#   02/15/2008  Update for v3.2.2.
#   12/05/2008  Update for v3.2.3.
#


I.  Introduction

    CalFUSE v3.2.3 is the latest version of the FUSE data-reduction
    pipeline.  This package includes a new set of calibration files and
    support for Linux 64-bit and Solaris 10 machines.

    For complete details about the program, see the paper, 
    "CalFUSE v3: A Data-Reduction Pipeline for the Far Ultraviolet
    Spectroscopic Explorer" by Dixon et al. (2007)
    (http://adsabs.harvard.edu/abs/2007PASP..119..527D).  It describes
    the CalFUSE pipeline, the instrument calibrations upon which it is
    based, and the format of the resulting calibrated data files.


II.  Recent Improvements

    New in CalFUSE v3.1.5
    
    Bad-Pixel Correction:  A new routine, cf_screen_bad_pixels, flags
    photon events falling in known bad-pixel regions.

    Histogram Mode: For data obtained in HIST mode,
    cf_count_rate_y_distort must use a single count-rate value for
    the entire exposure.  To better estimate it, we replace the
    time-dependent FEC-rate array with its weighted mean value.
    Rather than using different HIST_PAD values for each aperture,
    cf_extraction_limits now pads apertures by 8 pixels (top and
    bottom).

    Jitter Correction: To improve the jitter-correction algorithm, we
    removed the data-screening functions from cf_satellite_jitter and
    moved them to a new routine, cf_screen_jitter.

    New in CalFUSE v3.1.6

    Burst Correction:  Because cf_screen_bursts attempts to exclude
    airglow photons from the calculated background, these photons must
    be flagged before the routine is called.  To this end,
    cf_screen_airglow is now the first of the screening routines.

    Background Models:  The model background consists of a
    spatially-uniform component and a scattered-light image.  As a
    result of changes introduced in CalFUSE v3.1.3, the scattered-light
    component was over-estimated, because the data were summed over a
    larger region of the detector than the scattered-light image.  Now,
    the same limits are applied to both data and models.

    New in CalFUSE v3.1.7

    Spectral Centroids:  We tightened the limits over which
    cf_find_spectra computes the spectral centroid to prevent a bright
    star in a nearby aperture or the enhanced background near the
    detector edge from pulling the extraction window off center.  In
    both cf_compute_y_centroid and cf_find_spectra, we returned the
    parameter-file keywords EMAX_LIF and EMAX_SIC to their original
    behavior:  if the centroid differs from the default value by more
    than this limit, the default value is used.

    New in CalFUSE v3.1.8

    Astigmatism and Dispersion:  The routine
    cf_astigmatism_and_dispersion was divided into two programs,
    cf_astigmatism and cf_dispersion.

    Fewer Warnings:  The pipeline no longer issues WARNING messages
    when a raw or intermediate data file is empty, when the jitter file
    is missing, when the detector voltage is low, or when it cannot
    perform optimal extraction.

    Fewer Bad Spectra:  The program cf_extract_spectra no longer
    discards (that is, sets to zero the flux and error arrays of)
    spectra with non-zero values of EXP_STAT.

    New in CalFUSE v3.2.0

    Time-Dependent X and Y Distortions:  It turns out that the X and Y
    coordinates computed by the detector electronics for a particular
    spectral feature drift slowly with time.  The new routine
    cf_time_xy_distort corrects for this effect.

    Grating Motion:  The grating-motion correction has been revised to
    better correct the zero point of the wavelength scale.

    Jitter Files:  Over the years, FUSE engineers have repeatedly
    modified the spacecraft-control software, but the routine that
    generates jitter files has not been kept up to date.  As a result,
    the program was not using all available information to determine
    the quality and value of pointing errors.  We have modified the
    code to make use of this information.  As a result, the meaning of
    the jitter-file TRKFLG array has changed:  values between 1 and 5
    reflect increasing levels of confidence in the quality of reported
    pointing errors, a value of 0 indicates that pointing telemetry is
    missing, and a value of -1 says that the pointing is unknown but
    likely to be bad.  Jitter files generated with the new routine
    receive a version number >= 3.0.  The use of old jitter files with
    the new software is likely to yield unsatisfactory results.  If you
    have the housekeeping files, you can generate jitter files that are
    consistent with the new pipeline.  To do it, delete (or hide) the
    old jitter files, then type

    prompt> cf_jitter P9990101001hskpf.fit P9990101001jitrf.fit

    for each exposure.  Run the pipeline as usual.

    Jitter Correction:  The jitter routines cf_screen_jitter and
    cf_satellite_jitter have been updated to take advantage of the
    information present in the new jitter files, and three new keywords
    have been added to the parameter file.  Times for which the X or Y
    pointing errors are greater than DX_MAX or DY_MAX (default is 30")
    are flagged as bad.  Only pointing errors with associated tracking
    flag values >= TRKFLG (default is 3) are considered reliable.

    FIFO Overflow:  Before, the screening routines flagged times when
    the detector count rate went to zero because of FIFO overflows.
    This technique underestimated the actual target flux.  Now, we
    modify the IDS dead-time correction to account for photon losses
    due to FIFO overflows.

    New in CalFUSE v3.2.1

    Jitter Correction:  We have modified the program cf_jitter, which
    constructs jitter files, to be more robust when presented with
    weird pointing data.

    SAA Contours: The size of the South Atlantic Anomaly varies on
    timescales of weeks.  If you find that the default contours (in the
    file saac004.fit) are too small, causing times with elevated
    background rates to be flagged as good, you can switch to the
    smaller set of contours in the file saac005.fit by modifying the
    file master_calib_file.dat.

    New in CalFUSE v3.2.2

    New Platforms:  CalFUSE now runs on 64-bit machines running Linux
    and Sun workstations running Solaris v10.  See the file
    INSTALLING_CalFUSEv3.2.2 for installation details.

    We have used data from the last few months of the mission to
    produce final versions of the PHAH_CAL, STIM_CAL, and PHAX_CAL
    files.

    New in CalFUSE v3.2.3

    The final version of the pipeline includes updates needed for the
    final processing of the FUSE archive.  First, airglow exposures get
    special treatment: data taken close to the earth limb are flagged
    but not rejected.  Airglow exposures are ignored by cf_combine and
    idf_combine unless the -a flag is set.  Second, the usual limits on
    the spectral centroids are not applied to HIST data, because
    spectra from other apertures are not present to confuse the
    centroiding algorithm.


III. Overview of Pipeline

    The following modules are called by the shell script calfuse.csh:

    cf_ttag_init, cf_hist_init
	Converts raw-data file into an IDF containing photon-event
	list, GTI's, and timeline table.  Histogram data are converted
	to a pseudo-time-tag format.

    cf_convert_to_farf
	Corrects for detector deadtime.  Transforms photon coordinates
	into the FARF (flight alignment reference frame).

    cf_screen_photons
	Checks data quality.  Assigns status flag to each photon.

    cf_remove_motions
	Corrects for mirror, grating, FPA, and spacecraft motions.

    cf_assign_wavelength
	Applies astigmatism and Doppler corrections.
	Assigns wavelength to each photon.

    cf_flux_calibrate
	Converts WEIGHT to ERG/CM2 for each photon.

    cf_bad_pixels
	Applies image-motion corrections to bad-pixel map.

    cf_extract_spectra
	Extracts LiF and SiC spectra only for target aperture.
	Wavelength array is user defined; default spacing is 0.013 A.


IV.  Installing and Running the Pipeline

    CalFUSE is now available for Macs running OS X 10.2 and higher,
    64-bit Linux machines, and Sun workstations running Solaris 10.

    Please see the document INSTALLING_CalFUSEv3.2.2 for instructions.

    For both TTAG and HIST data, the command is 

	prompt> calfuse P99901010011attagfraw.fit
	prompt> calfuse P99901010011ahistfraw.fit

    The pipeline expects to find all data files (*raw.fit, *jitrf.fit,
    *hskpf.fit) in the current directory.  It operates on one detector
    segment at a time.

    CalFUSE v3 produces the following output files: an intermediate
    data file (IDF), discussed below; a bad-pixel map (BPM), with a
    format similar to the IDF; two extracted spectral files, one for
    each of the LiF and SiC channels; and (if IDL is installed on your
    machine) a pair of GIF or JPEG files, one an image of the detector
    and the other a count-rate plot.


V.  The Intermediate Data File (IDF)

    The photon-list files consist of three FITS binary tables.  The
    first contains the photon events themselves.  The pipeline does not
    discard any events, but uses a series of flags to indicate whether
    particular photons violate pulse-height limits, limb-angle
    constraints, etc.  The IDL tool cf_edit (discussed below) allows
    users to modify these flags and combine IDF files from multiple
    exposures.

    The second extension to the data files is a list of good-time
    intervals (GTI's).  They are not used by the extraction routine,
    but may prove helpful to pipeline users.

    The third extension is called a timeline table.  For each second
    during the exposure, it lists a dozen parameters, including count
    rate, day-night status, and detector voltage.  The pipeline uses
    this list to set the status flags for each photon.

    Note: because of the new file format employed by IDF files, 
    extensions 1 and 3 must be read using the /fscale keyword.

    idl> a=mrdfits('P99901010011attagfidf.fit',1,/fscale)
    idl> help,a,/str

    Elements of individual arrays must be addressed using the syntax

    idl> print,a.time[3:30]	-- not a[3:30].time

    For details about the format of the IDF, please see the document

    [calfuse_directory]/v3.1/src/README_FILES/IDF_Format.txt 


VI. Manipulating the IDF with cf_edit

    CF_EDIT is an Interactive Data Language (IDL) visualization tool
    for the examination and modification of FUSE IDF files containing
    flux- and wavelength-calibrated photon-event lists.  Users without
    IDL licenses may use the IDL "Virtual Machine" version of the
    tool for free.  Complete instructions for installing and using
    cf_edit are available at this URL:

    http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html

    More information on the IDL Virtual Machine is available here:

    http://www.rsinc.com/idlvm/index.asp


VII. The Output Spectral File

    The format of the output spectral files is somewhat changed from
    earlier versions of CalFUSE.

    float     WAVE       Wavelength (Angstroms)
    float     FLUX       Flux (erg/cm2/s/A)
    float     ERROR      Gaussian errors (erg/cm2/s/A)
    int       COUNTS     Raw counts in extraction window
    float     WEIGHTS    Raw counts corrected for deadtime, etc.
    float     BKGD       Estimated background in extraction window
    short     QUALITY    Percentage of window used for extraction

    Note that the QUALITY array no longer lists the number of Y pixels
    that contribute to each output X pixel.  Instead, it is the fraction
    of the extraction window considered to contain valid data, expressed
    as a percentage (0 is all bad; 100 is all good).

    Do-it-yourself-ers can use the following recipe to generate a
    flux-calibrated spectrum:

    TARGET_COUNTS = WEIGHTS - BKGD                                          
    TARGET_FLUX = TARGET_COUNTS * HC / LAMBDA / AEFF / EXPTIME / WPC

    where AEFF = effective area in cm^2 and WPC = size of output pixel in
    Angstroms.

VIII. Additional Files Available from MAST

    CalFUSE produces eight extracted spectral (*fcal.fit) files for
    each exposure. We combine them into a set of three
    observation-level files for submission to the MAST archive.
    Depending on the target and the questions that you are trying to
    answer, you may find that these files are of sufficient fidelity
    for scientific investigation. Here's a brief description of their
    contents:

    ALL: For each channel (LiF 1A, SiC 1A, etc.), we combine data from
    all exposures in the observation into a single spectrum. If the
    individual spectra are bright enough, we cross correlate and shift
    before combining them. If the spectra are too faint, we combine the
    individual IDF files and extract a single spectrum to optimize the
    background model. The combined spectrum for each channel is stored
    in a separate extension.

    Note: The cataloging software used by MAST requires the presence of
    an ALL file for each exposure, not just for the entire observation.
    We now generate such a file, but it contains no data, only a FITS
    file header. The observation-level ALL files discussed above can be
    identified by the string "00000all" in their names.

    ANO (all, night-only): Same format as the ALL files, but using only
    data obtained during the night-time portion of each exposure. These
    files are generated only for TTAG data, and only if EXPNIGHT > 0.

    NVO (National Virtual Observatory): Contains a single spectrum
    spanning the entire FUSE wavelength range. The spectrum is
    assembled by cutting and pasting segments from the most sensitive
    channel at each wavelength. Segments are shifted to match LiF 1A
    between 1045 and 1070 E. Columns are WAVE, FLUX, and ERROR and are
    stored in a single binary table extension.

IX. Data-Analysis Tools

    A variety of IDL routines to display and manipulate FUSE data are
    available from the FUSE IDL Tools Reference Page:

    http://fuse.pha.jhu.edu/analysis/fuse_idl_tools.html

    Data-analysis tools distributed with the CalFUSE pipeline are
    reviewed in the document FUSE Tools in C:

    http://fuse.pha.jhu.edu/analysis/fuse_toolbox.html

    A subset of these programs, designed specifically for the
    manipulation of IDF files, is described in the IDF Cookbook:

    http://fuse.pha.jhu.edu/analysis/idfcook/index.html