.help dofoe Feb93 noao.imred.echelle
.ih
NAME
dofoe -- Fiber Optic Echelle (FOE) data reduction task
.ih
USAGE
dofoe objects
.ih
SUMMARY
The \fBdofoe\fR reduction task is specialized for scattered light
subtraction, extraction, flat fielding, and wavelength calibration of Fiber
Optic Echelle (FOE) spectra.  There may be one fiber or two fibers where
the second fiber is illuminated by an arc calibration during arc and object
exposures and a flat field during flat field exposures.  It is a command
language script which collects and combines the functions and parameters of
many general purpose tasks to provide a single complete data reduction
path.  The task provides a degree of guidance, automation, and record
keeping necessary when dealing with the complexities of reducing this type
of data.
.ih
PARAMETERS
.ls objects
List of object spectra to be processed.  Previously processed spectra are
ignored unless the \fIredo\fR flag is set or the \fIupdate\fR flag is set and
dependent calibration data has changed.  Extracted spectra are ignored.
.le
.ls apref = ""
Aperture reference spectrum.  This spectrum is used to define the basic
extraction apertures and is typically a flat field spectrum.
.le
.ls flat = "" (optional)
Flat field spectrum.  If specified the one dimensional flat field spectrum
is extracted and used to make flat field calibrations.
.le
.ls arcs = "" (at least one if dispersion correcting)
List of arc spectra.  The first arc in the list is used to create a
dispersion solution interactively.  All other arc spectra will be
automatically reidentified.
.le
.ls arctable = "" (optional) (refspectra)
Table defining arc spectra to be assigned to object spectra (see
\fBrefspectra\fR).  If not specified an assignment based on a header
parameter, \fIparams.sort\fR, such as the observation time is made.
.le

.ls readnoise = "0." (apsum)
Read out noise in photons.  This parameter defines the minimum noise
sigma.  It is defined in terms of photons (or electrons) and scales
to the data values through the gain parameter.  A image header keyword
(case insensitive) may be specified to get the value from the image.
.le
.ls gain = "1." (apsum)
Detector gain or conversion factor between photons/electrons and
data values.  It is specified as the number of photons per data value.
A image header keyword (case insensitive) may be specified to get the value
from the image.
.le
.ls datamax = INDEF (apsum.saturation)
The maximum data value which is not a cosmic ray.
When cleaning cosmic rays and/or using variance weighted extraction
very strong cosmic rays (pixel values much larger than the data) can
cause these operations to behave poorly.  If a value other than INDEF
is specified then all data pixels in excess of this value will be
excluded and the algorithms will yield improved results.
This applies only to the object spectra and not the flat field or
arc spectra.  For more
on this see the discussion of the saturation parameter in the
\fBapextract\fR package.
.le
.ls norders = 12 (apfind)
Number of orders to be found.  This number is used during the automatic
definition of the apertures from the aperture reference spectrum.  Note
that when there is a second fiber for simultaneous arcs the specified
number will be automatically doubled for finding both sets of orders.
So in either case specify only the number of orders from a single fiber.
The interactive review of the aperture assignments allows verification
and adjustments to the automatic aperture definitions.
.le
.ls width = 4. (apedit)
Approximate base full width of the fiber profiles.  This parameter is used
for the profile centering algorithm.
.le
.ls arcaps = "2x2"
When there is only a single fiber set this parameter to "".  When there is
a second fiber used to create simultaneous arcs during the object exposures
this parameter specifies a list of aperture numbers for the arc fibers.
Since the object and arc fiber orders are paired the default setting
expects the even number apertures to be the are apertures.  This should be
checked interactively.
.le

.ls fitflat = yes (flat1d)
Fit and divide the extracted flat field orders by a smooth function
in order to normalize the wavelength response?  If not done the flat field
spectral shape (which includes the blaze function) will be divided
out of the object spectra, thus altering the object data values.
If done only the small scale response variations are included in the
flat field and the object spectra will retain their observed flux
levels and blaze function.
.le
.ls background = "none" (apsum, apscatter)
Type of background light subtraction.  The choices are "none" for no
background subtraction, "scattered" for a global scattered light
subtraction, "average" to average the background within background regions,
"median" to use the median in background regions, "minimum" to use the
minimum in background regions, or "fit" to fit across the dispersion using
the background within background regions.  The scattered light option fits
and subtracts a smooth global background and modifies the input images.
This is a slow operation and so is NOT performed in quicklook mode.  The
other background options are local to each aperture at each point along the
dispersion.  The "fit" option uses additional fitting parameters from
\fBparams\fR and the "scattered" option uses parameters from \fBapscat1\fR
and \fBapscat2\fR.
.le
.ls clean = yes (apsum)
Detect and correct for bad pixels during extraction?  This is the same
as the clean option in the \fBapextract\fR package.  If yes this also
implies variance weighted extraction and requires reasonably good values
for the readout noise and gain.  In addition the datamax parameters
can be useful.
.le
.ls dispcor = yes
Dispersion correct spectra?  Depending on the \fIparams.linearize\fR
parameter this may either resample the spectra or insert a dispersion
function in the image header.
.le
.ls redo = no
Redo operations previously done?  If no then previously processed spectra
in the objects list will not be processed (unless they need to be updated).
.le
.ls update = no
Update processing of previously processed spectra if aperture, flat
field, or dispersion reference definitions are changed?
.le
.ls batch = no
Process spectra as a background or batch job.
.le
.ls listonly = no
List processing steps but don't process?
.le

.ls params = "" (pset)
Name of parameter set containing additional processing parameters.  The
default is parameter set \fBparams\fR.  The parameter set may be examined
and modified in the usual ways (typically with "epar params" or ":e params"
from the parameter editor).  Note that using a different parameter file
is not allowed.  The parameters are described below.
.le

.ce
-- PACKAGE PARAMETERS

Package parameters are those which generally apply to all task in the
package.  This is also true of \fBdofoe\fR.
.ls observatory = "observatory"
Observatory at which the spectra were obtained if not specified in the
image header by the keyword OBSERVAT.  For FOE data the image headers
identify the observatory as "kpno" so this parameter is not used.
For data from other observatories this parameter may be used
as describe in \fBobservatory\fR.
.le
.ls interp = "poly5" (nearest|linear|poly3|poly5|spline3|sinc)
Spectrum interpolation type used when spectra are resampled.  The choices are:

.nf
	nearest - nearest neighbor
	 linear - linear
	  poly3 - 3rd order polynomial
	  poly5 - 5th order polynomial
	spline3 - cubic spline
	   sinc - sinc function
.fi
.le
.ls dispaxis = 2
Default dispersion axis.  The dispersion axis is 1 for dispersion
running along image lines and 2 for dispersion running along image
columns.  If the image header parameter DISPAXIS is defined it has
precedence over this parameter.
.le
.ls database = "database"
Database (directory) used for storing aperture and dispersion information.
.le
.ls verbose = no
Print verbose information available with various tasks.
.le
.ls logfile = "logfile", plotfile = ""
Text and plot log files.  If a filename is not specified then no log is
kept.  The plot file contains IRAF graphics metacode which may be examined
in various ways such as with \fBgkimosaic\fR.
.le
.ls records = ""
Dummy parameter to be ignored.
.le
.ls version = "ECHELLE: ..."
Version of the package.
.le

.ce
PARAMS PARAMETERS

The following parameters are part of the \fBparams\fR parameter set and
define various algorithm parameters for \fBdofoe\fR.

.ce
--  GENERAL PARAMETERS --
.ls line = INDEF, nsum = 10
The dispersion line (line or column perpendicular to the dispersion
axis) and number of adjacent lines (half before and half after unless
at the end of the image) used in finding, recentering, resizing,
editing, and tracing operations.  A line of INDEF selects the middle of the
image along the dispersion axis.
.le
.ls extras = no (apsum)
Include extra information in the output spectra?  When cleaning or using
variance weighting the cleaned and weighted spectra are recorded in the
first 2D plane of a 3D image, the raw, simple sum spectra are recorded in
the second plane, and the estimated sigmas are recorded in the third plane.
.le

.ce
-- DEFAULT APERTURE LIMITS --
.ls lower = -3., upper = 3. (apdefault)
Default lower and upper aperture limits relative to the aperture center.
These limits are used when the apertures are first found and may be
resized automatically or interactively.
.le

.ce
-- AUTOMATIC APERTURE RESIZING PARAMETERS --
.ls ylevel = 0.05 (apresize)
Data level at which to set aperture limits during automatic resizing.
It is a fraction of the peak relative to a local background.
.le

.ce
-- TRACE PARAMETERS --
.ls t_step = 10 (aptrace)
Step along the dispersion axis between determination of the spectrum
positions.  Note the \fInsum\fR parameter is also used to enhance the
signal-to-noise at each step.
.le
.ls t_function = "spline3", t_order = 2 (aptrace)
Default trace fitting function and order.  The fitting function types are
"chebyshev" polynomial, "legendre" polynomial, "spline1" linear spline, and
"spline3" cubic spline.  The order refers to the number of
terms in the polynomial functions or the number of spline pieces in the spline
functions.
.le
.ls t_niterate = 1, t_low = 3., t_high = 3. (aptrace)
Default number of rejection iterations and rejection sigma thresholds.
.le

.ce
-- DEFAULT BACKGROUND PARAMETERS --
.ls buffer = 1. (apscatter)
Buffer distance from the edge of any aperture for data to be included
in the scattered light determination.  This parameter may be modified
interactively.
.le
.ls apscat1 = "", apscat2 = "" (apscatter)
Parameter sets for the fitting functions across and along the dispersion.
These parameters are those used by \fBicfit\fR.  These parameters are
usually set interactively.
.le
.ls b_function = "legendre", b_order = 1 (apsum)
Default background fitting function and order.  The fitting function types are
"chebyshev" polynomial, "legendre" polynomial, "spline1" linear spline, and
"spline3" cubic spline.  The order refers to the number of
terms in the polynomial functions or the number of spline pieces in the spline
functions.
.le
.ls b_naverage = -100 (apsum)
Default number of points to average or median.  Positive numbers
average that number of sequential points to form a fitting point.
Negative numbers median that number, in absolute value, of sequential
points.  A value of 1 does no averaging and each data point is used in the
fit.
.le
.ls b_niterate = 0 (apsum)
Default number of rejection iterations.  If greater than zero the fit is
used to detect deviant fitting points and reject them before repeating the
fit.  The number of iterations of this process is given by this parameter.
.le
.ls b_low_reject = 3., b_high_reject = 3. (apsum)
Default background lower and upper rejection sigmas.  If greater than zero
points deviating from the fit below and above the fit by more than this
number of times the sigma of the residuals are rejected before refitting.
.le
.ls b_smooth = 10 (apsum)
Box car smoothing length for background when using background
subtraction.  Since the background noise is often the limiting factor
for good extraction one may box car smooth the background to improve the
statistics.
.le


.ce
-- APERTURE EXTRACTION PARAMETERS --
.ls weights = "none" (apsum)
Type of extraction weighting.  Note that if the \fIclean\fR parameter is
set then the weights used are "variance" regardless of the weights
specified by this parameter.  The choices are:
.ls "none"
The pixels are summed without weights except for partial pixels at the
ends.
.le
.ls "variance"
The extraction is weighted by the variance based on the data values
and a poisson/ccd model using the \fIgain\fR and \fIreadnoise\fR
parameters.
.le
.le
.ls pfit = "fit1d" (apsum) (fit1d|fit2d)
Profile fitting algorithm for cleaning and variance weighted extractions.
The default is generally appropriate for FOE data but users
may try the other algorithm.  See \fBapprofiles\fR for further information.
.le
.ls lsigma = 3., usigma = 3. (apsum)
Lower and upper rejection thresholds, given as a number of times the
estimated sigma of a pixel, for cleaning.
.le

.ce
-- FLAT FIELD FUNCTION FITTING PARAMETERS --
.ls f_interactive = no (fit1d)
Fit the one dimensional flat field order spectra interactively?
This is used if \fIfitflat\fR is set and a two dimensional flat field
spectrum is specified.
.le
.ls f_function = "spline3", f_order = 20 (fit1d)
Function and order used to fit the composite one dimensional flat field
spectrum.  The functions are "legendre", "chebyshev", "spline1", and
"spline3".  The spline functions are linear and cubic splines with the
order specifying the number of pieces.
.le

.ce
-- ARC DISPERSION FUNCTION PARAMETERS --
.ls threshold = 10. (identify/reidentify)
In order for a feature center to be determined the range of pixel intensities
around the feature must exceed this threshold.
.le
.ls coordlist = "linelist$thar.dat" (ecidentify)
Arc line list consisting of an ordered list of wavelengths.
Some standard line lists are available in the directory "linelist$".
.le
.ls match = 1. (ecidentify)
The maximum difference for a match between the dispersion function computed
value and a wavelength in the coordinate list.
.le
.ls fwidth = 4. (ecidentify)
Approximate full base width (in pixels) of arc lines.
.le
.ls cradius = 4. (reidentify)
Radius from previous position to reidentify arc line.
.le
.ls i_function = "chebyshev", i_xorder = 3, i_yorder = 3 (ecidentify)
The default function, function order for the pixel position dependence, and
function order for the aperture number dependence to be fit to the arc
wavelengths.  The functions choices are "chebyshev" or "legendre".
.le
.ls i_niterate = 3, i_low = 3.0, i_high = 3.0 (ecidentify)
Number of rejection iterations and sigma thresholds for rejecting arc
lines from the dispersion function fits.
.le
.ls refit = yes (ecreidentify)
Refit the dispersion function?  If yes and there is more than 1 line
and a dispersion function was defined in the arc reference then a new
dispersion function of the same type as in the reference image is fit
using the new pixel positions.  Otherwise only a zero point shift is
determined for the revised fitted coordinates without changing the
form of the dispersion function.
.le

.ce
-- AUTOMATIC ARC ASSIGNMENT PARAMETERS --
.ls select = "interp" (refspectra)
Selection method for assigning wavelength calibration spectra.
Note that an arc assignment table may be used to override the selection
method and explicitly assign arc spectra to object spectra.
The automatic selection methods are:
.ls average
Average two reference spectra without regard to any sort parameter.
If only one reference spectrum is specified then it is assigned with a
warning.  If more than two reference spectra are specified then only the
first two are used and a warning is given.
This option is used to assign two reference spectra, with equal weights,
independent of any sorting parameter.
.le
.ls following
Select the nearest following spectrum in the reference list based on the
sorting parameter.  If there is no following spectrum use the nearest preceding
spectrum.
.le
.ls interp
Interpolate between the preceding and following spectra in the reference
list based on the sorting parameter.  If there is no preceding and following
spectrum use the nearest spectrum.  The interpolation is weighted by the
relative distances of the sorting parameter.
.le
.ls match
Match each input spectrum with the reference spectrum list in order.
This overrides the reference aperture check.
.le
.ls nearest
Select the nearest spectrum in the reference list based on the sorting
parameter.
.le
.ls preceding
Select the nearest preceding spectrum in the reference list based on the
sorting parameter.  If there is no preceding spectrum use the nearest following
spectrum.
.le
.le
.ls sort = "jd", group = "ljd" (refspectra)
Image header keywords to be used as the sorting parameter for selection
based on order and to group spectra.
A null string, "", or the word "none" may be use to disable the sorting
or grouping parameters.
The sorting parameter
must be numeric but otherwise may be anything.  The grouping parameter
may be a string or number and must simply be the same for all spectra within
the same group (say a single night).
Common sorting parameters are times or positions.
In \fBdofoe\fR the Julian date (JD) and the local Julian day number (LJD)
at the middle of the exposure are automatically computed from the universal
time at the beginning of the exposure and the exposure time.  Also the
parameter UTMIDDLE is computed.
.le
.ls time = no, timewrap = 17. (refspectra)
Is the sorting parameter a 24 hour time?  If so then the time origin
for the sorting is specified by the timewrap parameter.  This time
should precede the first observation and follow the last observation
in a 24 hour cycle.
.le

.ce
-- DISPERSION  CORRECTION PARAMETERS --
.ls linearize = yes (dispcor)
Interpolate the spectra to a linear dispersion sampling?  If yes the
spectra will be interpolated to a linear or log linear sampling
If no the nonlinear dispersion function(s) from the dispersion function
database are assigned to the input image world coordinate system
and the spectral data are not interpolated.
.le
.ls log = no (dispcor)
Use linear logarithmic wavelength coordinates?  Linear logarithmic
wavelength coordinates have wavelength intervals which are constant
in the logarithm of the wavelength.
.le
.ls flux = yes (dispcor)
Conserve the total flux during interpolation?  If \fIno\fR the output
spectrum is interpolated from the input spectrum at each output
wavelength coordinate.  If \fIyes\fR the input spectrum is integrated
over the extent of each output pixel.  This is slower than
simple interpolation.
.le
.ih
ENVIRONMENT PARAMETERS
The environment parameter \fIimtype\fR is used to determine the extension
of the images to be processed and created.  This allows use with any
supported image extension.  For STF images the extension has to be exact;
for example "d1h".
.ih
DESCRIPTION
The \fBdofoe\fR reduction task is specialized for scattered light
subtraction, extraction, flat fielding, and wavelength calibration of Fiber
Optic Echelle (FOE) spectra.  There may be one fiber or two fibers where
the second fiber is illuminated by an arc calibration during arc and object
exposures and a flat field during flat field exposures.  When there is
just one fiber the parameter \fIarcaps\fR is set to "" and when there are
two fibers the parameter is used to select which of the defined
apertures are the orders from the simultaneous arc fiber.

This task is a command language script which collects and combines the
functions and parameters of many general purpose tasks to provide a single
complete data reduction path.  The task provides a degree of guidance,
automation, and record keeping necessary when dealing with the complexities
of reducing this type of data.

The general organization of the task is to do the interactive setup steps
first using representative calibration data and then perform the majority
of the reductions automatically, possibly as a background process, with
reference to the setup data.  In addition, the task determines which setup
and processing operations have been completed in previous executions of the
task and, contingent on the \fIredo\fR and \fIupdate\fR options, skip or
repeat some or all the steps.

The description is divided into a quick usage outline followed by details
of the parameters and algorithms.  The usage outline is provided as a
checklist and a refresher for those familiar with this task and the
component tasks.  It presents only the default or recommended usage.  Since
\fBdofoe\fR combines many separate, general purpose tasks the description
given here refers to these tasks and leaves some of the details to their
help documentation.

\fBUsage Outline\fR

.ls 6 [1]
The images must first be processed with \fBccdproc\fR for overscan,
bias, and dark corrections.
.le
.ls [2]
Set the \fBdofoe\fR parameters with \fBeparam\fR.  Specify the object
images to be processed, the flat field image as the aperture reference and
the flat field, and one or more arc images.  If there are many
object or arc spectra per setup you might want to prepare "@ files".
Verify and set the format parameters, particularly the number of orders to be
extracted and processed.  The processing parameters are set
for simple extraction and dispersion correction but dispersion correction
can be turned off for quicklook or background subtraction and cleaning
may be added.
.le
.ls [3]
Run the task.  This may be repeated multiple times with different
observations and the task will generally only do the setup steps
once and only process new images.  Queries presented during the
execution for various interactive operations may be answered with
"yes", "no", "YES", or "NO".  The lower case responses apply just
to that query while the upper case responses apply to all further
such queries during the execution and no further queries of that
type will be made.
.le
.ls [4]
The apertures are defined using the specified aperture reference image
which is usually a flat field in which both the object and arc fibers are
illuminated.  The specified number of orders are found automatically and
sequential apertures assigned.  The resize option sets the aperture size to
the widths of the profiles at a fixed fraction of the peak height.
.le
.ls [5]
The automatic order identification and aperture assignment is based on peak
height and may be incorrect.  The interactive aperture editor is entered
with a plot of the apertures.  When there is a second simultaneous arc
fiber it is essential that the object and arc
fiber orders are properly paired with the arc fibers having even aperture
numbers and the object fibers having odd aperture numbers.  It is also
required that no orders be skipped in the region of interest.  Missing
orders are added with the 'm' key.  Once all orders have been marked the
aperture numbers are resequenced with 'o'.  If local background subtraction
is selected the background regions should be checked with the 'b' key.
Preceding this with the 'a' key allows any changes to the background
regions to be applied to all orders.  To exit type 'q'.
.le
.ls [6]
The order positions at a series of points along the dispersion are measured
and a function is fit to these positions.  This may be done interactively to
adjust the fitting parameters.  Not all orders need be examined and the "NO"
response will quit the interactive fitting.  To exit the interactive
fitting type 'q'.
.le
.ls [7]
If flat fielding is to be done the flat field spectra are extracted.  A
smooth function is fit to each flat field spectrum to remove the large
scale spectral signature.  The final response spectra are normalized to a
unit mean over all fibers.
.le
.ls [8]
If scattered light subtraction is selected the scattered light parameters
are set using the aperture reference image and the task \fBapscatter\fR.
The purpose of this is to interactively define the aperture buffer distance
for the scattered light and the cross and parallel dispersion fitting
parameters.  The fitting parameters are taken from and recorded in the
parameter sets \fBapscat1\fR and \fBapscat2\fR.  All other scattered light
subtractions are done noninteractively with these parameters.  Note that
the scattered light correction modifies the input images.
.le
.ls [9]
If dispersion correction is selected the first arc in the arc list is
extracted.  One fiber is used to identify the arc lines and define the
dispersion function using the task \fBecidentify\fR.  Identify a few arc
lines in a few orders with 'm' and 'k' or 'o', use the 'l' line list
identification command to automatically add additional lines and fit the
dispersion function.  Check the quality of the dispersion function fit
with 'f'.  When satisfied exit with 'q'.
.le
.ls [10]
If there is a second fiber the dispersion function is automatically
determined using the task \fBecreidentify\fR.
.le
.ls [11]
The arc reference spectrum is dispersion corrected.
If the spectra are resampled to a linear dispersion system
(which will be the same for all spectra) the dispersion parameters
determined from the dispersion solution are printed.
.le
.ls [12]
The object spectra are now automatically background subtracted (an
alternative to scattered light subtraction), extracted, flat fielded,
and dispersion corrected.  Any new dispersion function reference arcs
assigned to the object images are automatically extracted and
dispersion functions determined.  A zero point wavelength correction
is computed from the simultaneous arc fiber spectrum and applied to
the object spectrum if orders from the second fiber have been identified
with the \fIarcaps\fR parameter.
.le
.ls [13]
The final spectra will have the same name as the original 2D images
with a ".ec" extension added.
.le

\fBSpectra and Data Files\fR

The basic input consists of single or dual fiber FOE object and calibration
spectra stored as IRAF images.  The \fIarcaps\fR parameter is used to
discriminate between the two cases.  The type of image format is defined by
the environment parameter \fIimtype\fR.  Only images with that extension
will be processed and created.  The raw CCD images must be processed to
remove overscan, bias, and dark count effects.  This is generally done
using the \fBccdred\fR package.  Flat fielding is generally not done at
this stage but as part of \fBdofoe\fR.  The calibration spectra are flat
field observations in all fibers, comparison arc lamp spectra in all
fibers, and, for dual fiber model, arc spectra in one fiber while the
second fiber observes the object.  If for some reason the flat field or
calibration arc spectra have separate exposures for the two fibers the
separate exposures may simply be added.

The assignment of arc calibration exposures to object exposures is
generally done by selecting the nearest in time and interpolating.
However, the optional \fIarc assignment table\fR may be used to explicitly
assign arc images to specific objects.  The format of this file is
described in the task \fBrefspectra\fR.

The final reduced spectra are recorded in two or three dimensional IRAF
images.  The images have the same name as the original images with an added
".ec" extension.  Each line in the reduced image is a one dimensional
spectrum (an echelle order) with associated aperture and wavelength
information.  When the \fIextras\fR parameter is set the lines in the
third dimension contain additional information (see
\fBapsum\fR for further details).  These spectral formats are accepted by the
one dimensional spectroscopy tasks such as the plotting tasks \fBsplot\fR
and \fBspecplot\fR.  The special task \fBscopy\fR may be used to extract
specific apertures or to change format to individual one dimensional
images.  The task \fBscombine\fR is used to combine or merge orders into
a single spectrum.

\fBPackage Parameters\fR

The \fBechelle\fR package parameters set parameters affecting all the tasks
in the package.  Some of the parameters are not applicable to the
\fBdofoe\fR task.  The observatory parameter is only required for data
without an OBSERVAT header parameter (currently included in NOAO data).
The spectrum interpolation type might be changed to "sinc" but with the
cautions given in \fBonedspec.package\fR.  The dispersion axis parameter is
only needed if a DISPAXIS image header parameter is not defined.  The other
parameters define the standard I/O functions.  The verbose parameter
selects whether to print everything which goes into the log file on the
terminal.  It is useful for monitoring what the \fBdofoe\fR task does.  The
log and plot files are useful for keeping a record of the processing.  A
log file is highly recommended.  A plot file provides a record of
apertures, traces, and extracted spectra but can become quite large.
The plotfile is most conveniently viewed and printed with \fBgkimosaic\fR.

\fBProcessing Parameters\fR

The input images are specified by image lists.  The lists may be
a list of explicit, comma separate image names, @ files, or image
templates using pattern matching against file names in the directory.
The aperture reference spectrum is used to find the orders and trace
them.  Thus, this requires an image with good signal in both fibers
which usually means a flat field spectrum.  It is recommended that
flat field correction be done using one dimensional extracted spectra
rather than as two dimensional images.  This is done if a flat field
spectrum is specified.  The arc assignment table is used to specifically
assign arc spectra to particular object spectra and the format
of the file is described in \fBrefspectra\fR.

The detector read out noise and gain are used for cleaning and variance
(optimal) extraction.  The dispersion axis defines the wavelength direction
of spectra in the image if not defined in the image header by the keyword
DISPAXIS.  The width parameter (in pixels) is used for the profile
centering algorithm (\fBcenter1d\fR).

The number of orders selects the number of orders for a single
fiber and "pairs" of object and arc
fiber profiles for dual fibers.   The number specified will be
automatically found based on the strongest peaks.
In the  dual fiber case it is important that both elements of a pair be found,
so no orders be skipped, and the aperture numbers must be sequential with
arc profiles having even aperture numbers and object profiles having
odd numbers in the region of interest, the automatic identification is  
just a starting point for the interactive review.  The even/odd
relationship between object and arc profiles is set by the \fIarcaps\fR
parameter and so may be reversed if desired.

The next set of parameters select the processing steps and options.  The
flat fitting option allows fitting and removing the overall shape of the
flat field spectra while preserving the pixel-to-pixel response
corrections.  This is useful for maintaining the approximate object count
levels, including the blaze function, and not introducing the reciprocal of
the flat field spectrum into the object spectra.  If not selected the flat
field will remove the blaze function from the observations and introduce
some wavelength dependence from the flat field lamp spectrum.

The \fIbackground\fR option selects the type of correction for background or
scattered light.  If the type is "scattered" a global scattered light is
fit to the data between the apertures  and subtracted from the images.
\fINote that the input images are modified by this operation\fR.  This
option is slow.  Alternatively, a local background may be subtracted using
background regions defined for each aperture.  The data in the regions may
be averaged, medianed, or the minimum value used.  Another choice is to fit
the data in the background regions by a function and interpolate to the
object aperture.

The \fIclean\fR option invokes a profile fitting and deviant point rejection
algorithm as well as a variance weighting of points in the aperture.  These
options require knowing the effective (i.e. accounting for any image
combining) read out noise and gain.  For a discussion of cleaning and
variance weighted extraction see \fBapvariance\fR and \fBapprofiles\fR.

The dispersion correction option selects whether to extract arc spectra,
determine a dispersion function, assign them to the object spectra, and,
possibly, resample the spectra to a linear (or log-linear) wavelength
scale.

Generally once a spectrum has been processed it will not be reprocessed if
specified as an input spectrum.  However, changes to the underlying
calibration data can cause such spectra to be reprocessed if the
\fIupdate\fR flag is set.  The changes which will cause an update are a new
reference image, new flat field, adding the scattered light option, and a
new arc reference image.  If all input spectra are to be processed
regardless of previous processing the \fIredo\fR flag may be used.  Note
that reprocessing clobbers the previously processed output spectra.

The \fIbatch\fR processing option allows object spectra to be processed as
a background or batch job.  The \fIlistonly\fR option prints a summary of
the processing steps which will be performed on the input spectra without
actually doing anything.  This is useful for verifying which spectra will
be affected if the input list contains previously processed spectra.  The
listing does not include any arc spectra which may be extracted to
dispersion calibrate an object spectrum.

The last parameter (excluding the task mode parameter) points to another
parameter set for the algorithm parameters.  The way \fBdofoe\fR works
this may not have any value and the parameter set \fBparams\fR is always
used.  The algorithm parameters are discussed further in the next section.

\fBAlgorithms and Algorithm Parameters\fR

This section summarizes the various algorithms used by the \fBdofoe\fR
task and the parameters which control and modify the algorithms.  The
algorithm parameters available to the user are collected in the parameter
set \fBparams\fR.  These parameters are taken from the various general
purpose tasks used by the \fBdofoe\fR processing task.  Additional
information about these parameters and algorithms may be found in the help
for the actual task executed.  These tasks are identified in the parameter
section listing in parenthesis.  The aim of this parameter set organization
is to collect all the algorithm parameters in one place separate from the
processing parameters and include only those which are relevant for
FOE data.  The parameter values can be changed from the
defaults by using the parameter editor,
.nf

	cl> epar params

.fi
or simple typing \fIparams\fR.  The parameter editor can also be
entered when editing the \fBdofoe\fR parameters by typing \fI:e
params\fR or simply \fI:e\fR if positioned at the \fIparams\fR
parameter.

\fBAperture Definitions\fR

The first operation is to define the extraction apertures, which include the
aperture width, background regions, and position dependence with
wavelength, for the object and arc orders of interest.  This is done
on a reference spectrum which is usually a flat field taken through
all fibers.  Other spectra will inherit the reference apertures and
apply a correction for any shift of the orders across the dispersion.
The reference apertures are defined only once unless the \fIredo\fR
option is set.

The selected number of orders are found automatically by selecting the
highest peaks in a cut across the dispersion.  Note that the specified
number of orders is multiplied by two in defining the apertures when
there is a second fiber.  Apertures
are assigned with a limits set by the \fIlower\fR and
\fIupper\fR parameter and numbered sequentially.  A query is then
given allowing the aperture limits to be "resized" based on the profile
itself (see \fBapresize\fR).

A cut across the orders is then shown with the apertures marked and
an interactive aperture editing mode is entered (see \fBapedit\fR).
For \fBdofoe\fR the aperture identifications and numbering is particularly
critical.  When there is a single fiber the aperture numbers must
be sequential with the order numbers.  If an order is skipped then the
aperture number must also be skipped.

For dual fibers all "pairs" of object and arc orders in the region of
interest must be defined without skipping any orders.  The orders must
also be numbered sequentially (though the direction does not matter)
so that the arc apertures are either all even or all odd as defined
by the \fIarcaps\fR parameter (the default is even numbers for the
arc apertures).  The 'o' key will provide the necessary reordering.

If local background subtraction is used the background regions should
also be checked with the 'b' key.  Typically one adjusts all
the background regions at the same time by selecting all apertures with
the 'a' key first.  To exit the background and aperture editing steps type
'q'.

Next the positions of the orders at various points along the dispersion are
measured and "trace functions" are fit.  The user is asked whether to fit
each trace function interactively.  This is selected to adjust the fitting
parameters such as function type and order.  When interactively fitting a
query is given for each aperture.  After the first aperture one may skip
reviewing the other traces by responding with "NO".  Queries made by
\fBdofoe\fR generally may be answered with either lower case "yes" or "no"
or with upper case "YES" or "NO".  The upper case responses apply to all
further queries and so are used to eliminate further queries of that kind.

The above steps are all performed using tasks from the \fBapextract\fR
package and parameters from the \fBparams\fR parameters.  As a quick
summary, the dispersion direction of the spectra are determined from the
package \fBdispaxis\fR parameter if not defined in the image header.  The
default line or column for finding the orders and the number of image lines
or columns to sum are set by the \fIline\fR and \fInsum\fR parameters.  A
line of INDEF (the default) selects the middle of the image.  The automatic
finding algorithm is described for the task \fBapfind\fR and basically
finds the strongest peaks.  The resizing is described in the task
\fBapresize\fR and the parameters used are also described there and
identified in the PARAMETERS section.  The tracing is done as described in
\fBaptrace\fR and consists of stepping along the image using the specified
\fIt_step\fR parameter.  The function fitting uses the \fBicfit\fR commands
with the other parameters from the tracing section.

\fBBackground or Scattered Light Subtraction\fR

In addition to not subtracting any background scattered light there are two
approaches to subtracting this light.  The first is to determine a smooth
global scattered light component.  The second is to subtract a locally
determined background at each point along the dispersion and for each
aperture.  Note that background subtraction is only done for object images
and not for arc images.

The global scattered light fitting and subtraction is done with the task
\fBapscatter\fR.  The function fitting parameters are set interactively
using the aperture reference spectrum.  All other subtractions are done
noninteractively with the same set of parameters.  The scattered light is
subtracted from the input images, thus modifying them, and one might wish
to first make backups of the original images.

The scattered light is measured between the apertures using a specified
buffer distance from the aperture edges.  The scattered light pixels are
fit by a series of one dimensional functions across the dispersion.  The
independent fits are then smoothed along the dispersion by again fitting
low order functions.  These fits then define the smooth scattered light
surface to be subtracted from the image.  The fitting parameters are
defined and recorded in the two parameter sets \fIapscat1\fR and
\fIapscat2\fR.  The scattered light algorithm is described more fully in
\fBapscatter\fR.  This algorithm is relatively slow.

Local background subtraction is done during extraction based on background
regions and parameters defined by the default background parameters or
changed during interactive review of the apertures.  The background
subtraction options are to subtract the average, median, or minimum of the
pixels in the background regions, or to fit a function and subtract the
function from under the extracted object pixels.  The background regions
are specified in pixels from the aperture center and follow changes in
center of the spectrum along the dispersion.  The syntax is colon separated
ranges with multiple ranges separated by a comma or space.  The background
fitting uses the \fBicfit\fR routines which include medians, iterative
rejection of deviant points, and a choice of function types and orders.
Note that it is important to use a method which rejects cosmic rays such as
using either medians over all the background regions (\fIbackground\fR =
"median") or median samples during fitting (\fIb_naverage\fR < -1).
The background smoothing parameter \fIb_smooth\fR is may be used
to provide some additional local smoothing of the background light.
The background subtraction algorithm and options are described in greater
detail in \fBapsum\fR and \fBapbackground\fR.

\fBExtraction\fR

The actual extraction of the spectra is done by summing across the fixed
width apertures at each point along the dispersion.  The default is to
simply sum the pixels using partial pixels at the ends.  There is an
option to weight the sum based on a Poisson noise model using the
\fIreadnoise\fR and \fIgain\fR detector parameters.  Note that if the
\fIclean\fR option is selected the variance weighted extraction is used
regardless of the \fIweights\fR parameter.  The sigma threshold for
cleaning are also set in the \fBparams\fR parameters.

The cleaning and variance weighting options require knowing the effective
(i.e. accounting for any image combining) read out noise and gain.  These
numbers need to be adjusted if the image has been processed such that the
intensity scale has a different origin (such as a scattered light
subtraction) or scaling (such as caused by unnormalized flat fielding).
These options also require using background subtraction if the profile does
not go to zero.  For optimal extraction and cleaning to work it is
recommended that any scattered light be accounted for by local background
subtraction rather than with the scattered light subtraction and the
\fIfitflat\fR option be used.  The \fIb_smooth\fR parameter is also
appropriate in this application and improves the optimal extraction results
by reducing noise in the background signal.  For further discussion of
cleaning and variance weighted extraction see \fBapvariance\fR and
\fBapprofiles\fR as well as  \fBapsum\fR.

\fBFlat Field Correction\fR

Flat field corrections may be made during the basic CCD processing; i.e.
direct division by the two dimensional flat field observation.  In that
case do not specify a flat field spectrum; use the null string "".  The
\fBdofoe\fR task provides an alternative flat field response correction
based on division of the extracted object spectra by the extracted flat field
spectra.  A discussion of the theory and merits of flat fielding directly
verses using the extracted spectra will not be made here.  The
\fBdofoe\fR flat fielding algorithm is the \fIrecommended\fR method for
flat fielding since it works well and is not subject to the many problems
involved in two dimensional flat fielding.

The first step is extraction of the flat field spectrum, if one is specified,
using the reference apertures.  Only one flat field is allowed so if
multiple flat fields are required the data must be reduced in groups.  When
the \fIfitflat\fR option is selected (the default) the extracted flat field
spectra are fit by smooth functions and the ratio of the flat field spectra
to the smooth functions define the response spectra.  The default fitting
function and order are given by the parameters \fIf_function\fR and
\fIf_order\fR.  If the parameter \fIf_interactive\fR is "yes" then the
fitting is done interactively using the \fBfit1d\fR task which uses the
\fBicfit\fR interactive fitting commands.

If the \fIfitflat\fR option is not selected the extracted and globally
normalized flat field spectra are directly divided in the object spectra.
This removes the blaze function, thus altering the data counts, and
introduces the reciprocal of the flat field spectrum in the object
spectra.

The final step is to normalize the flat field spectra by the mean counts over
all the fibers.  This normalization step is simply to preserve the average
counts of the extracted object and arc spectra after division by the
response spectra.

\fBDispersion Correction\fR

If dispersion correction is not selected, \fIdispcor\fR=no, then the object
spectra are simply extracted.  If it is selected the arc spectra are used
to dispersion calibrate the object spectra.  There are three steps involved;
determining the dispersion functions relating pixel position to wavelength,
assigning the appropriate dispersion function to a particular observation,
and either storing the nonlinear
dispersion function in the image headers or resampling the spectra to
evenly spaced pixels in wavelength.  When there are two fibers there is
also a step of applying a zero point correction to the object fiber based
on the arc fiber.

The first arc spectrum in the arc list is used to define the reference
dispersion solution.  It is extracted using the reference aperture
definitions.  Note extractions of arc spectra are not background or
scattered light subtracted.  The interactive task \fBecidentify\fR is used
to define the dispersion function in one fiber.  The idea is to mark some
lines in a few orders whose wavelengths are known (with the line list used
to supply additional lines after the first few identifications define the
approximate wavelengths) and to fit a function giving the wavelength from
the aperture number and pixel position.  The dispersion function for the
second fiber, if one is present, is then determined automatically by
reference to the first fiber using the task \fBecreidentify\fR.

The arc dispersion function parameters are for \fBecidentify\fR and it's
related partner \fBecreidentify\fR.  The parameters define a line list for
use in automatically assigning wavelengths to arc lines, a centering width
(which should match the line widths at the base of the lines), the
dispersion function type and orders, parameters to exclude bad lines from
function fits, and defining whether to refit the dispersion function as
opposed to simply determining a zero point shift.  The defaults should
generally be adequate and the dispersion function fitting parameters may be
altered interactively.  One should consult the help for the two tasks for
additional details of these parameters and the interactive operation of
\fBecidentify\fR.

Once the reference dispersion functions are defined other arc spectra are
extracted as they are assign to the object spectra.  The assignment of
arcs is done either explicitly with an arc assignment table (parameter
\fIarctable\fR) or based on a header parameter such as a time.
The assignments are made by the task \fBrefspectra\fR.  When two arcs are
assigned to an object spectrum an interpolation is done between the two
dispersion functions.  This makes an approximate correction for steady
drifts in the dispersion.

When a second arc fiber monitors any zero point shifts in the dispersion
functions it is probably only necessary to have one or two arc spectra, one
at the beginning and/or one at the end of the night.

The tasks \fBsetjd\fR and \fBsetairmass\fR are automatically run on all
spectra.  This computes and adds the header parameters for the Julian date
(JD), the local Julian day number (LJD), the universal time (UTMIDDLE), and
the air mass at the middle of the exposure.  The default arc assignment is
to use the Julian date grouped by the local Julian day number.  The
grouping allows multiple nights of data to be correctly assigned at the
same time.

Defining the dispersion function for a new arc extraction is done with
the task \fBecreidentify\fR.  This is done noninteractively with log
information recorded about the line reidentifications and the fit.

When there are two fibers there are two full dispersion function from the
single or pair of arc spectra, one for the object fiber and one for the arc
fiber.  When an object spectrum is extracted so is the simultaneous arc
spectrum.  A zero point shift of the arc spectrum relative to the
dispersion solution of the dual arc observation is computed using
\fBecreidentify\fR (\fIrefit\fR=no).  This zero point shift is assumed to
be the same for the object fiber and it is added to the dispersion function
of the dual arc observation for the object fiber.  Note that this does not
assume that the object and arc fiber dispersion functions are the same or
have the same wavelength origin, but only that the same shift in wavelength
zero point applies to both fibers.  Once the dispersion function correction
is determined from the extracted arc fiber spectrum it is deleted leaving
only the object spectrum.

The last step of dispersion correction is setting the dispersion
of the object spectrum.  There are two choices here.
If the \fIlinearize\fR parameter is not set the nonlinear dispersion
function is stored in the image header.  Other IRAF tasks interpret
this information when dispersion coordinates are needed for plotting
or analysis.  This has the advantage of not requiring the spectra
to be interpolated and the disadvantage that the dispersion
information is only understood by IRAF tasks and cannot be readily
exported to other analysis software.

If the \fIlinearize\fR parameter is set then the spectra are resampled to a
linear dispersion relation either in wavelength or the log of the
wavelength.  For echelle spectra each order is linearized independently so
that the wavelength interval per pixel is different in different orders.
This preserves most of the resolution and avoids over or under sampling of
the highest or lowest dispersion orders.  The wavelength limits are
taken from the limits determined from the arc reference spectrum and
the number of pixels is the same as the original images.  The dispersion
per pixel is then derived from these constraints.

The linearization algorithm  parameters allow selecting the interpolation
function type, whether to conserve flux per pixel by integrating across the
extent of the final pixel, and whether to linearize to equal linear or
logarithmic intervals.  The latter may be appropriate for radial velocity
studies.  The default is to use a fifth order polynomial for interpolation,
to conserve flux, and to not use logarithmic wavelength bins.  These
parameters are described fully in the help for the task \fBdispcor\fR which
performs the correction.
.ih
EXAMPLES
1.  The following example uses artificial data and may be executed
at the terminal (with IRAF V2.10).  This is also the sequence performed
by the test procedure "demos dofoe".  Because the images are small the
dispersion solution is somewhat simplistic.

.nf
ec> demos mkdofoe
Creating image demoobj ...
Creating image demoflat ...
Creating image demoarc ...
ec> echelle.verbose = yes
ec> dofoe demoobj apref=demoflat flat=demoflat arcs=demoarc \
>>> norders=3 width=5.
Set reference apertures for demoflat
Searching aperture database ...
Finding apertures ...
Mar  4  9:39: FIND - 6 apertures found for demoflat
Resize apertures for demoflat?  (yes):
Resizing apertures ...
Mar  4  9:39: RESIZE - 6 apertures resized for demoflat
<Review aperture assignments.  Exit with 'q'>
Fit traced positions for demoflat interactively?  (yes):
Tracing apertures ...
Fit curve to aperture 1 of demoflat interactively  (yes):
<Review trace and fit. Exit with 'q'>
Fit curve to aperture 2 of demoflat interactively  (yes): N
Mar  4  9:39: TRACE - 6 apertures traced in demoflat.
Mar  4  9:39: DATABASE - 6 apertures for demoflat written to database
Create response function demoflatnorm.ec
Extract flat field demoflat
Searching aperture database ...
Mar  4  9:39: DATABASE  - 6 apertures read for demoflat from database
Extracting apertures ...
Mar  4  9:39: EXTRACT - Aperture 1 from demoflat --> demoflat.ec
Mar  4  9:39: EXTRACT - Aperture 2 from demoflat --> demoflat.ec
Mar  4  9:39: EXTRACT - Aperture 3 from demoflat --> demoflat.ec
Mar  4  9:39: EXTRACT - Aperture 4 from demoflat --> demoflat.ec
Mar  4  9:39: EXTRACT - Aperture 5 from demoflat --> demoflat.ec
Mar  4  9:40: EXTRACT - Aperture 6 from demoflat --> demoflat.ec
Fit and ratio flat field demoflat
Create the normalized response demoflatnorm.ec
demoflatnorm.ec -> demoflatnorm.ec  using bzero: 0.  and bscale: 1.
    mean: 1.  median: 0.9990048  mode: 0.9876572
    upper: INDEF  lower: INDEF
Extract arc reference image demoarc
Mar  4  9:40: DATABASE  - 6 apertures read for demoflat from database
Mar  4  9:40: DATABASE - 6 apertures for demoarc written to database
Mar  4  9:40: EXTRACT - Aperture 1 from demoarc --> demoarc.ec
Mar  4  9:40: EXTRACT - Aperture 2 from demoarc --> demoarc.ec
Mar  4  9:40: EXTRACT - Aperture 3 from demoarc --> demoarc.ec
Mar  4  9:40: EXTRACT - Aperture 4 from demoarc --> demoarc.ec
Mar  4  9:40: EXTRACT - Aperture 5 from demoarc --> demoarc.ec
Mar  4  9:40: EXTRACT - Aperture 6 from demoarc --> demoarc.ec
Determine dispersion solution for demoarc
<Mark lines with 'm' and change orders with 'k'
<'m' line at pixel 78 and assign 4965.
<'k' to order 2
<'m' line at pixel 78 and assign 5009
<'m' line at pixel 78 and assign 5020
<'k' to order 3
<'m' line at pixel 78 and assign 5049.8
<'m' line at pixel 78 and assign 5050.8
<'m' line at pixel 78 and assign 5055.3
<'m' line at pixel 78 and assign 5062
<'m' line at pixel 78 and assign 5064.9
<'f' to fit
<'q' to quit fit and 'q' to quit ECIDENTIFY

ECREIDENTIFY: NOAO/IRAF V2.10BETA valdes@puppis Wed 09:54:16 04-Mar-92
  Reference image = demoarc.ec, Refit = yes
   Image    Found     Fit Pix Shift  User Shift  Z Shift      RMS
  d...ec    8/8     8/8        1.48        7.06  2.11E-5  0.00879
d...ec: ap = 1, w1 = 4959.1, w2 = 4978.5, dw = 0.076, nw = 256
d...ec: ap = 2, w1 = 5003.4, w2 = 5022.1, dw = 0.073, nw = 256
d...ec: ap = 3, w1 = 5049.0, w2 = 5067.0, dw = 0.070, nw = 256
Extract object spectrum demoobj
Searching aperture database ...
Mar  4  9:54: DATABASE  - 6 apertures read for demoflat from database
Recentering apertures ...
Mar  4  9:54: RECENTER  - 6 apertures shifted by -0.03 for demoobj.
Mar  4  9:54: DATABASE - 6 apertures for demoobj written to database
Extracting apertures ...
Mar  4  9:54: EXTRACT - Aperture 1 from demoobj --> demoobj.ec
Mar  4  9:54: EXTRACT - Aperture 2 from demoobj --> demoobj.ec
Mar  4  9:54: EXTRACT - Aperture 3 from demoobj --> demoobj.ec
Mar  4  9:54: EXTRACT - Aperture 4 from demoobj --> demoobj.ec
Mar  4  9:54: EXTRACT - Aperture 5 from demoobj --> demoobj.ec
Mar  4  9:54: EXTRACT - Aperture 6 from demoobj --> demoobj.ec
Assign arc spectra for demoobj
[demoobj] refspec1='demoarc'
Reidentify arc fibers in demoobj with respect to demoarc

ECREIDENTIFY: NOAO/IRAF V2.10BETA valdes@puppis Wed 09:54:28 04-Mar-92
  Reference image = demoarcarc.ec, Refit = no
   Image    Found     Fit Pix Shift  User Shift  Z Shift      RMS
  d...ec    8/8     8/8       0.119       0.566  1.69E-6  0.00834
Dispersion correct demoobj
d...ec.imh: ap = 1, w1 = 4959.1, w2 = 4978.5, dw = 0.076, nw = 256
d...ec.imh: ap = 2, w1 = 5003.4, w2 = 5022.1, dw = 0.073, nw = 256
d...ec.imh: ap = 3, w1 = 5049.0, w2 = 5067.0, dw = 0.070, nw = 256
.fi
.ih
REVISIONS
.ls DOFOE V2.10.3
The image format type to be
processed is selected with the \fIimtype\fR environment parameter.  The
dispersion axis parameter is now a package parameter.  Images will only
be processed if the have the CCDPROC keyword.  A \fIdatamax\fR parameter
has been added to help improve cosmic ray rejection.  A scattered
light subtraction processing option has been added.
.le
.ih
SEE ALSO
apedit, apfind, approfiles, aprecenter, apresize, apsum, aptrace, apvariance,
ccdred, center1d, dispcor, fit1d, icfit, ecidentify, observatory,
onedspec.package, refspectra, ecreidentify, setairmass, setjd
.endhelp
