path: root/noao/onedspec/doc/sys/onedproto.ms

.RP
.ND
.TL
Some Notes on the ONEDSPEC Package
.AU
G. Jacoby
.AI
.K2 "" "" "*"
June 1985
.AB
The first phase of the ONEDSPEC prototype package is complete.
Comments and some internal description is presented for each task 
in the package. Also presented are some more global descriptions
of strategies used in the package and considerations for future
improvements.
.AE
.SH
1. Why is ONEDSPEC Different?
.PP
This section describes some of the ways in which the ONEDSPEC
package diverges from other IRAF package strategies.
A few of these should someday be modified to more closely
adhere to IRAF conventions, but in other cases, restrictions
or limitations in the IRAF system are revealed.
.sp 1
.SH
Quantity
.PP
One of the major differences between a two dimensional image processing
package and a one dimensional package is that spectra 
frequently congregate in groups of hundreds to thousands while two-dimensional
images live in groups of tens to hundreds. What this means is that spectral
processing must be somewhat more automated and streamlined - the software cannot
rely on user input to provide assistance and it cannot afford
excessive overhead; otherwise  a large fraction of the processing time will be
spent where it is least useful.
.PP
To process large volumes of spectra in a reasonably automated fashion,
the software must be smart enough to know what to do with a variety
of similar but different spectra. The way adopted here is to key
off header parameters which define the type of spectrum and the
processing required. In fact, most of the ONEDSPEC package will not
work smoothly without some header parameter information.
.PP
It is also important that each task be self-reliant so that the
overhead of task stop and restart is avoided. For many operations,
the actual computation time is a fraction of a second, yet no
operation in the ONEDSPEC package is faster than one second per spectrum
due to task overhead. If task startup and stop were required for each
spectrum, then the overhead would be much worse.
.PP
So the philosophy is one in which each task uses as much information
as it can reasonably expect from the spectral image header.
Usually this is not more than three or four elements.
The strategy of using header information should not be limited to
ONEDSPEC. Many image processing problems can be automated
to a large degree if header information is used. The success of
the KPNO CCD Mountain reduction system emphasizes this point.
It would seem prudent that other IRAF applications make use of
such information when possible.
[See section 3 for a more detailed discussion of headers.]
.sp 1
.SH
Spectral Image Names
.PP
One implication of the quantity problem is that it must be easy for the user to
specify the names of large numbers of spectra.  The approach taken for ONEDSPEC
was to assign a root name to a group of spectra and then
append an index number of 4 or more digits starting with 0000.
So spectra, by default, have the form root.0000, root.0001, ...
To specify the spectra, the user types only the root name and the range
of indices such as "root" and "0-99,112-113,105-108".
The range decoder accesses the spectral indices in the order given
as opposed to access in ascending order, so that the spectrum root.0112
will be processed before root.0105 in the example specification above.
Spectra having more general names may be specified using the
standard IRAF filename expansion methods if the 
the range specification is given as null.
.PP
The specification of large numbers of images is an area where
most IRAF applications are weak. Resorting to odd combinations
of bracket and backslash characters in filename specifications 
is obscure to new users and still fails to
meet the general need. The range specification adopted for ONEDSPEC
comes closer but introduces a fixed image name format.
.sp 1
.SH
Apertures -- A way to group data
.PP
Many spectrographs generate multiple spectra simultaneously by
placing more than one slit or aperture in the focal plane.
Examples include the IIDS, IRS, and Cryogenic Camera in use
at Kitt Peak. The Echelle may be considered a multi-aperture
instrument for purposes of reductions by associating each order
with an "aperture" number. 
.PP
The concept of aperture can be generalized to indicate a set of
spectral data having common group properties such as
wavelength coverage. Most tasks in ONEDSPEC will key off
an aperture number in the image header and treat those
common aperture spectra uniformly.
Defining data groups which are to be processed in this fashion
is a technique not generally exploited by reduction programs.
This is  due in part to the problem of image header usage.
.PP
For programming convenience and to avoid an additional level
of indirectness, in ONEDSPEC the aperture number is used directly as an
index in many static arrays. The current implementation has
a declaration for 50 apertures and due to the IIDS/IRS
notation of apertures 0 and 1, the apertures are zero-indexed, contrary
to standard IRAF nomenclature,
from 0-49. It would certainly be better to map the aperture numbers
to the allowable index range, but the added complexity of another
level of indirectness seemed distracting. Actually the mapping
can still be done by the header reader, "load_ids_hdr", and
unmapped by the header writer, "store_keywords".
.sp 1
.SH
Static versus dynamic arrays
.PP
Although dynamic storage would alleviate some of the memory
requirements in the package, the use of static arrays aids
readability and accounts for only about 10 percent of the total
task memory space. Many of the arrays are arrays of pointers.
For example, in the task BSWITCH, there is an array (called "imnames") 
of pointers for the names of spectral images, several for each aperture.
The actual space for the names is dynamically allocated,
so first we allocate an array of pointers for each
aperture:
.sp 1
.DS
     call salloc (imnames[aperture], nr_names, TY_POINT)
.DE
.sp 1
Then, for each of these pointers, space must be allocated for the
character arrays:
.sp 1
.DS
     do i = 1, nr_names
         call salloc (Memp[imnames[aperture]+i-1], SZ_LINE, TY_CHAR)
.DE
.sp 1
Later to access the character strings, a name is specified as:
.sp 1
.DS
     Memc[Memp[imnames[aperture]+nr_of_this_spectrum-1]]
.DE
.sp 1
If the "imnames" array was also dynamically allocated, the
above access would be even less readable.
If memory requirements become a serious problem, then these ONEDSPEC
tasks should be modified.
.sp 1
.SH
Output image names
.PP
To retain the consistent usage of root names and ranges, output
spectra also have the form root.nnnn. For user convenience,
the current output root name and next suffix are maintained as
package parameters onedspec.output and onedspec.next_rec.
The latter parameter is automatically updated each time a
new spectrum is written. This is done by the individual tasks
which directly access this package parameter.
.PP
There is an interesting side effect when using indirect parameters
(e.g. )onedspec.output) for input. In the local task parameter
file, the mode of the parameter must be declared hidden. So when the user
does an "lpar task", those parameters appear to be unnecessary
(that is, they are enclosed in parenthesis). When run,
prompts appear because the parameter is an automatic mode
parameter in the package parameter file.
If run as a background task, this is more annoying.
Unfortunately, any other choice of parameter modes produces
less desirable actions.
.sp 1
.SH
ONEDUTIL
.PP
As the number of tasks in ONEDSPEC started growing, the
need for a subdivision of the package became clear.
The first cut was made at the utility level, and a number
of task names (not necessarily physical tasks) were
moved out into the ONEDUTIL submenu. In the future,
additional tasks will eventually require another subpackage.
.PP
Actually, many of the tasks in ONEDUTIL may be more at home
in some other package, but a conscious effort was made to
avoid contaminating other IRAF packages with tasks written for
the ONEDSPEC project.  If all the following tasks are relocated, 
then the need for ONEDUTIL is reduced.
.PP
Two of the entries in ONEDUTIL may be considered as more appropriate
to DATAIO - RIDSMTN and WIDSTAPE. In fact RIDSMTN can
replace the version currently in DATAIO. WIDSTAPE may replace the
DATAIO task WIDSOUT if the usage of header parameters does not
present a problem.
.PP
The task MKSPEC may be a candidate for the ARTDATA package.
It should be enhanced to include optional noise generation.
Also, it may be appropriate for SINTERP to replace INTERP
in the UTILITY package.
.PP
I suppose one could argue that SPLOT belongs in the PLOT package.
Certainly, the kludge script BPLOT should be replaced by a more
general batch plot utility in PLOT.
Also, the two task names, IDENTIFY and REIDENTIFY are present
in the ONEDSPEC menu for user convenience, but the task declarations
in ONEDSPEC.CL refer to tasks in the LONGSLIT package.
.PP
Because ONEDUTIL is a logical separation of the tasks, not
a complete physical task breakup, there is no subdirectory
for ONEDUTIL as there is in other packages. This is a bit messy
and it may be best to completely disentangle the tasks in the
subpackage into a true package having all the implications.
.LP
.SH
2. Task Information
.PP
There are currently about 30 tasks in the ONEDSPEC package.
These are summarized in the menu listing below and
a brief description of some less obvious aspects of each follows.
.sp 1
.DS L
   ONEDSPEC

	addsets   - Add subsets of strings of spectra
	batchred  - Batch processing of IIDS/IRS spectra
	bswitch   - Beam-switch strings of spectra to make obj-sky pairs
	calibrate - Apply sensitivity correction to spectra
	coincor   - Correct spectra for photon coincidence
	dispcor   - Dispersion correct spectra
	extinct   - Correct data for atmospheric extinction
	flatfit   - Sum and normalize flat field spectra
	flatdiv   - Divide spectra by flat field
	identify  - Identify features in spectrum for dispersion solution
	iids	  - Set reduction parameters for IIDS
	irs	  - Set reduction parameters for IRS
	onedutil  - Enter ONEDSPEC Utility package
	process   - A task generated by BATCHRED
	reidentify- Automatically identify features in spectra
	sensfunc  - Create sensitivity function
	slist     - List spectral header elements
	splot     - Preliminary spectral plot/analysis
	standard  - Identify standard stars to be used in sensitivity calc
	subsets   - Substract pairs in strings of spectra

   ONEDUTIL

	bplot	  - Batch plots of spectra
	coefs     - Extract mtn reduced ceofficients from henear scans
	combine   - Combine spectra having different wavelength ranges
	lcalib	  - List calibration file data
	mkspec    - Generate an artificial spectrum
	names     - Generate a list of image names from a string
	rebin     - Rebin spectra to new dispersion parameters
	ridsmtn   - Read IIDS/IRS mountain format tapes
	sinterp   - Interpolate a table of x,y pairs to create a spectrum
	widstape  - Write Cyber format IDSOUT tapes
.DE
.sp 1
.SH
ADDSETS
.PP
Spectra for a given object may have been observed through more than
one instrument aperture. For the IIDS and IRS, this is the most common
mode of operation. Both apertures are used to alternately observe
the program objects. 
.PP
Each instrument aperture may be considered an
independent instrument having unique calibration properties, and
the observations may then be processed completely independently
until fully calibrated. At that point the data may be combined to
improve signal-to-noise and reduce systematic errors associated
with the alternating observing technique. Because the data are
obtained in pairs for IIDS and IRS (but may be obtained in groups
of larger sizes from other instruments), ADDSETS provides a way 
to combine the pairs of observations.
.PP
Each pair in the input string is added to produce a single output
spectrum. Although the word "pair" is used here, the parameter
"subset" defines the number of elements in a "pair" (default=2).
The input string is broken down into groups where each group
consists of the pair of spectra defined in order of the input
list of image names.
.PP
"Add" in ADDSETS means:
.RS
.IP 1.
Average the pairs if the data are calibrated to flux (CA_FLAG=0)
optionally weighted by the integration time.
.IP 2.
Add the pairs if uncalibrated (CA_FLAG=-1).
.RE
.sp 1
.SH
BATCHRED
.PP
This is a script task which allows spectra from dual aperture instruments
to be processed completely in a batch mode after the initial wavelength
calibration and correction has been performed. The processes which
may be applied and the tasks referenced are:
.RS
.IP 1.
Declaring observations as standard stars for flux calibration (STANDARD).
.IP 2.
Solving for the sensitivity function based on the standard stars (SENSFUNC).
.IP 3.
Generating object minus sky differences and summing individual
observations if several were made (BSWITCH).
.IP 4.
Correcting for atmospheric extinction (BSWITCH).
.IP 5.
Applying the system sensitivity function to generate flux calibrated
data (CALIBRATE).
.IP 6.
Adding pairs of spectra obtained through the dual apertures (ADDSETS).
.RE
Any or all of these operations may be selected through the task
parameters.
.PP
BATCHRED generates a secondary script task called PROCESS.CL
which is a text file containing  constructed commands to the
ONEDSPEC package. This file may be edited by the user if an
entry to BATCHRED is incorrect. It may also be saved, or appended
by further executions of BATCHRED.
.PP
BATCHRED also generates a log file of the output generated by the
ONEDSPEC tasks it calls.
.sp 1
.SH
BSWITCH
.PP
This task combines multiple observations of a single object
or multiple objects taken through a multiaperture instrument.
Object minus sky differences are generated as pairs of
spectra are accumulated, then optionally corrected for
atmospheric extinction, and the differences added together
with optional weighting using counting statistics.
Each instrument aperture is considered an independent
device.
.PP
Despite the apparently simple goal of this task, it is probably
the most complicated in the ONEDSPEC package due to the
bookkeeping load associated with automated handling of large data sets
having a number of properties associated with each spectrum (e.g
object or sky, aperture number, exposure times).
.PP
There are several modes in which BSWITCH can operate. The mode
appropriate to the IIDS and IRS assumes that the spectra
are input in an order such that after 2N (N=number of
instrument apertures) spectra have been
accumulated, an equal number of object and sky spectra have been
encountered in each aperture. 
When in this mode, a check is made after 2N spectra
have been processed, and the optional extinction correction is
applied to the differences of the object minus sky, and then
(optionally weighted and) added into an accumulator for the aperture.
.PP
If the IIDS mode is switched off, then no guarantee can be
made that sky and object spectra pair off. If extinction
correction is required, it is performed on each spectrum
as it arrives, including sky spectra if any. The spectra are
then added into separate accumulators for object and sky for
each aperture after optional weighting is applied.
.PP
If after all spectra have been processed, there are no sky
spectra, the object spectrum is written out. If there is no
object spectrum, the sky spectrum is written out after
multiplying by -1. (This allows adding an object later on with
addsets, but the -1 multiply is probably a mistake.)
If at least one of each, object and sky spectra were encountered,
then the difference is computed and written out. Since
all accumulations are performed in count rates and later converted
back to counts, the object and sky spectra may have different
exposure times (non IIDS mode only).
.PP
A statistics file is maintained to provide an indication of the
quality of the individual spectra going into the sum. The
statistics information is maintained internally and only
written out after the sums have been generated.
The basic data in the file is the count rate of the spectrum
having the largest count rate, and the ratios of the count rates from
all other spectra to that one.
.PP
If weighting is selected, the weights are taken as proportional to
the count rate (prior to extinction correction) over a wavelength 
delimited region of the spectrum. (Perhaps the weight
should be proportional to counts, not count rate.)
The default wavelength region is the entire spectrum.
If the total count rate is negative, the weight is assigned
a value of 0.0 and will be disregarded in the sum. (The counts
may be negative if the object minus sky difference approaches zero
on a bright and cloudy night.)
.PP
If extinction is selected, an extinction table is read from the
package calibration file. An optional additive term may be applied
as computed by the system sensitivity task SENSFUNC which is placed
in the parameter sensfunc.add_const. A revision to the standard
extinction table (delta extinction as a function of wavelength)
may be read from a text file whose name is specified by the parameter
sensfunc.rev_ext_file. The file format is that of a text file
having pairs of (wavelength, delta extinction) on each line.
[The option to solve for this function in SENSFUNC has not yet been
implemented, but BSWITCH can read the file that would be generated.
Thus, one can experiment with revisions, although this has never been
tested.] BSWITCH will interpolate the values given in the file
so that a course estimate of the revision may be entered, say if the
deltas at U, B, V, R, and I are known.
.PP
BEWARE that the extinction correction is performed assuming the
header parameters used for airmass refer to a "mean" airmass value
for the exposure. In general the header value is wrong! It usually
refers to the beginning, middle, or end of the exposure. I have
never seen a header airmass value which was an equivalent airmass
for the duration of the exposure. This is partly because there is
no way to compute a single effective airmass; it is a function
of wavelength, telescope position as a function of time, and
the extinction function. Fortunately, for most observations
this is not very significant. But anyone taking a one hour exposure near 
3500 Angstroms at airmass values greater than 2, should not complain
when the fluxes look a bit odd.
.sp 1
.SH
CALIBRATE
.PP
Having a system sensitivity function allows the data to be
placed on an absolute flux scale. CALIBRATE performs this
correction using the output sensitivity function from SENSFUNC. Operations are
keyed to the instrument aperture, and a system sensitivity
function is required for each observing aperture, although
this requirement may be overriden.
.PP
A valid exposure time is required (a value of 1.0 should
probably be assumed if not present) to compute the observed
count rate. Input counts are transformed to units of
ergs/cm2/sec/Angstrom (or optionally ergs/cm2/sec/Hz).
CALIBRATE will calibrate two dimensional images as well, applying the
sensitivity function to all image lines.
.PP
The operation is performed on a pixel-by-pixel basis so that
the defined sensitivity function should overlap precisely
with data in terms of wavelength.
.sp 1
.SH
COINCOR
.PP
This task applies a statistical correction to each pixel
to account for undetected photoevents as a result of
coincidental arrival of photons. This is a detector
specific correcton, although the photoelectric detector
model provides a reasonable correction for many detectors
when a judicious value for the deadtime parameter is chosen.
This model assumes that the correction follows the
typical procedures applied to photoelectric photometer data:
.sp 1
.DS L
     Ic = Io * exp [Io * dt / T]
.DE
.sp 1
where Ic is the corrected count rate in a pixel, Io is the
observed count rate in that pixel, dt is the detector deadtime,
and T is the observation integration time.
.PP
In addition to the photoelectric model, a more accurate model
is available for the IIDS and is included in COINCOR. This
model is taken from Goad (1979, SPIE Vol 172, 86.) and the correction
is applied as:
.sp 1
.DS L
     Ic = ln [1 - Io * t] / t
.DE
.sp 1
where t is sweep time between pixel samples (t=1.424 msec).
The IIDS differs from a photomultiplier detector, in that
there is a fixed rate at which each pixel is sampled due to
time required for the dissector to sweep across the image tube
phospher whether a photoevent has occurred in a pixel or not. 
The photomultiplier plus discriminator system
assumes that once a photoevent has been recorded, the detector is
dead until a fixed interval has elapsed.
.sp 1
.SH
DISPCOR
.PP
If a relation is known linking pixel coordinate to user coordinate
(i.e. wavelength as a function of pixel number), then any non-linearities
can be removed by remapping the pixels to a linear wavelength coordinate.
This procedure, dispersion correction, is complicated by the
lack of a wavelength-pixel solution which is derived from data simultaneously
obtained with the object data. Any drifts in the detector then require
an interpolation among solutions for the solution appropriate to
the object observations. Depending on the detector, this interpolation
may be a function of the time of observation, temperature, or some telescope
parameter such as airmass.
When multiple solutions are available, DISPCOR will linearly interpolate
the solution in any available header parameter known to ONEDSPEC (see
section 3).
.PP
Each solution is read from the database file created by the IDENTIFY
task (in TWODSPEC$LONGSLIT), and the image name leading to that solution
is also read from the database file. The image is opened to extract
the header parameter to be used in the above interpolation.
A null name for the interpolation parameter indicates that none
is to be used. In this case, one of the options on the "guide"
parameter should be set to indicate what solution should be used.
The guide may be "precede", "follow", or "nearest" to select
the most appropriate choice for each spectrum.
.PP
If an explicit wavelength solution is to be used, the parameter
"reference" may be used to specify the image name of a comparison
spectrum to be used as the reference for the wavelength solution.
In this case all spectra will be corrected using a single solution - 
no flexure correction will be applied.
.PP
If the parameter to be used for interpolation is a "time-like"
variable, such as RA, UT, ST, then the variable is discontinuous
at 24|0 hours. If UT is the chosen parameter (as has been the
case for IIDS and IRS spectra), the discontinuity occurs at
5 PM local Kitt Peak time. A comparison spectrum taken at 4:59PM
(=23:59h UT, =just before dinner), will be treated as an "end of
the night" observation rather than a beginning of the night
observation. To circumvent this error, the parameter, "time_wrap",
can be specified to a time at which a true zero should be assigned.
For UT at Kitt Peak, a choice like 17h UT (=10AM local, =asleep),
is an unlikely hour for nighttime observations to be made. Then for
a given night's observations, 17h UT becomes the new zero point in time.
.PP
Each solution in the database may be any of the forms legal 
to IDENTIFY: legendre, chebyshev, spline3, or spline1 - the form 
is encoded in the database and will automatically be recalled.
The interpolation in the solution is performed by locating the
pixel location for each required wavelength for the two
solutions bounding each observation and linearly interpolating
for the appropriate pixel location. One cannot simply interpolate
across the coefficients of the solutions to derive a new
single solution because the solutions may have different forms
or orders, so that the coefficients may have quite different
meanings.
.PP
Dispersion correction requires that there be equal intervals
of wavelength between pixels. The wavelength solution
is of a form describing the wavelength for a given pixel location,
not a pixel location for a given wavelength. So the solution
must be inverted.
.PP
The inversion to pixel location for wavelength is done in the
following way: The pixel coordinate in the solution is incremented
until the desired wavelength is bounded. The pixel value for the
desired wavelength is obtained by linearly interpolating across these
two bounding pixel locations. A linear approximation appears to be
very good for typical solutions, providing proper pixel locations to
better than 0.01 pixels. An improvement may be obtained by
increasing the order of the interpolation, but the improvement
is generally not warranted because the wavelength solutions
are rarely known to this accuracy. [Note that the use of real
and not double precision limits the precision of this technique!
For spectra longer than 50,000 pixels, the errors due to
the precision of reals can be serious.]
.PP
Note that a transformation to
a wavelength coordinate which is linear in the logarithm of
wavelength only requires that the inversion occur at wavelengths
selected by equal increments in the logarithm of wavelength.
.PP
During the actual remapping, 5 possible techniques are available.
Actually there are only two techniques: re-interpolation in 4 flavors,
and rebinning by partial pixel summation. The re-interpolation
may be performed with polynomials of order 1 (=linear), 3, or 5,
or by a cubic spline. The 3rd and 5th order polynomials may introduce
some ringing in the wings of strong, sharp, features, but the 5th order
is good at preserving the high frequency component of the data.
The linear and spline interpolators introduce significant smoothing.
The rebinning algorithm offers conservation of flux but also smooths
the data. In fact, rebinning to a course grid offers a good smoothing
algorithm.
.PP
At some future date, it would be a good idea to include a "synch"
function interpolator in the image interpolator package. This would
be a little slower to process, but results in very good frequency
response.
.PP
Other options in DISPCOR include "ids_mode" which forces spectra
from all apertures to a single output mapping (starting wavelength
and pixel-to-pixel increment), and "cols_out" forces the output spectra
to a specified length, zero-filling if necessary.
.PP
DISPCOR will correct two-dimensional data by applying the
remapping to all lines in the image. If the input two-dimensional
spectrum has only one line, the output spectrum will be written as
a one-dimensional spectrum.
.sp 1
.SH
EXTINCT
.PP
Extinction is currently only available as a script file which drives
BSWITCH. This is possible by suppressing all options: weighting,
ids_mode, statistics file, and setting the subset pair size to the
number of instrument apertures.
.sp 1
.SH
FLATDIV
.PP
This task divides the specified spectra by their flat field spectra.
This is not much more than an "en mass" spectrum divider, with the
exceptions that the header elements are used to key on the
aperture number so that the appropriate flat field spectrum is used,
and that the header processing flags are checked to prevent
double divisions and subsequently set after the division. Also,
division by zero is guarded by setting any zeroes in the flat field
spectrum to 1.0 prior to the division.
.sp 1
.SH
FLATFIT
.PP
Pixel-to-pixel variations in the detector response can be removed
by dividing all observations by a flat field spectrum.
Flat field spectra are generally obtained by observing a source
having a continuous energy distribution, such as a tungsten filament
lamp. This is sometimes called a "quartz" lamp when the enclosing
glass bulb is made with quartz rather than silicon. The quartz
enclosure transmits ultraviolet light much better than glass.
.PP
If the color temperature of the source is very low (or very high, though
this is extremely unlikely), then a color term would be introduced
into the data when the flat is divided into the data.
Large scale variations in the system sensitivity also introduce a
color term into the flat - the same variations that are introduced into
any spectrum taken with the system. [Large scale variations are 
evaluated by STANDARD and SENSFUNC, and removed by CALIBRATE.]
This is not of any particular importance except that counting
statistics are destroyed by the division.
.PP
To preserve the statistics, many find it desirable to divide by a flat
field spectrum which has been filtered to remove any large scale variations
but in which the pixel-to-pixel variations have been retained.
A filtered flat can be obtained by fitting a low order polynomial
through the spectrum and dividing the spectrum by the polynomial.
The result is a spectrum normalized to 1.0 and having high frequency
variations only. If one does not care to preserve the statistics,
then this procedure is not required. In fact, for certain instruments
(the IRS), the fitting and normalizing procedure is not recommended
because some intermediate order curvature can be introduced.
.PP
The purpose of FLATFIT is to find the combination of parameters
which produces a well flattened flat with a minimum of wiggles.
The usual curve fitting package is used to fit a function (chebyshev,
legendre, spline3, spline1) to the flats. Pixel rejection is
user selectable by a choice of cutoff sigmas, both above and below
the mean, and an optional growing region [A growing region is the number
of pixels on either side of one rejected which will also be rejected - 
Growing regions are not recommended for most spectral applications].
Any number of iterations may be used to further reject discrepant
pixels. The fitting may be performed interactively and controlled by cursor
keystrokes to select the fitting order, and other fit parameters.
.PP
Prior to the fit, the specified spectra are read, optionally corrected
for coincidence losses, and added to accumulators appropriate to
their instrument apertures. Each aperture is treated independently,
except that, the interactive fitting mode may be selected to operate
on the first aperture only, and then apply the same fitting parameters
to all other aperture accumulations. Or the interactive procedure
may be selected to operate on all apertures or none.
.PP
After the fit has been done, the fit is divided into the accumulation
and written as a new spectrum having a specified root name and a trailing
index indicating the aperture.
.sp 1
.SH
IDENTIFY
.PP
This task (written by Frank Valdes) is used to identify features
in the comparison arcs to be used in the solution for a wavelength calibration.
The solution is performed interactively for at least one spectrum
and then optionally in a batch mode using REIDENTIFY.
IDENTIFY writes to a database file which will contain the solutions
generated from each input comparison spectrum. The database is
later used by DISPCOR to correct spectra according to the solution.
.sp 1
.SH
IIDS
.PP
This script file initializes several hidden parameters in a
variety of tasks to values appropriate to the IIDS instrument.
There is also a script for the IRS. There should probably be
a script for resetting the parameters to a default instrument.
These parameters are:
.RS
.IP 1.
onedspec.calib_file - the package parameter indicating which file
should be used for standard star calibration data and the atmospheric
extinction table (=onedspec$iids.cl.)
.IP 2.
addsets.subset - the number of instrument apertures (=2).
.IP 3.
bswitch.ids_mode - assume and check for data taken in beam-switched
quadruple mode (=yes).
.IP 4.
coincor.ccmode - coincidence correction model (=iids).
.IP 5.
coincor.deadtime - detector deadtime (=1.424e-3 seconds)
.IP 6.
dispcor.flex_par - the name of the parameter to be used as the
guide to removing flexure during the observations (=ut).
.IP 7.
dispcor.time_wrap - the zero point to be adopted for the
flexure parameter if it is a time-like variable having a discontinuity
at 0/24 hours (=17).
.IP 8.
dispcor.idsmode - should data from all instrument apertures be dispersion
corrected to a uniform wavelength scale? (=yes).
.IP 9.
dispcor.cols_out - the number of columns (row length of the spectrum)
to which the output corrected spectrum should be forced during
mapping (=1024).
.IP 10.
extinct.nr_aps - the number of instrument apertures (=2).
.IP 11.
flatfit.order - the order of the fit to be used when fitting to
the flat field spectra (=6).
.IP 12.
flatfit.coincor - apply coincidence correction to the flat field
spectra during accumulations (=yes).
.IP 13.
flatdiv.coincor - apply coincidence correction to all spectra during
the flat field division process (=yes).
.IP 14.
identify.function - the fitting function to be used during the wavelength
solution process (=chebyshev).
.IP 15.
identify.order - the order of the fit to be used during the wavelength
solution process (=6).
.RE
.sp 1
.SH
IRS
.PP
This script file initializes several hidden parameters in a
variety of tasks to values appropriate to the IRS instrument.
These parameters are:
.RS
.IP 1.
onedspec.calib_file - the package parameter indicating which file
should be used for standard star calibration data and the atmospheric
extinction table (=onedspec$irs.cl.)
.IP 2.
addsets.subset - the number of instrument apertures (=2).
.IP 3.
bswitch.ids_mode - assume and check for data taken in beam-switched
quadruple mode (=yes).
.IP 4.
coincor.ccmode - coincidence correction model (=iids).
.IP 5.
coincor.deadtime - detector deadtime (=1.424e-3 seconds)
.IP 6.
dispcor.flex_par - the name of the parameter to be used as the
guide to removing flexure during the observations (=ut).
.IP 7.
dispcor.time_wrap - the zero point to be adopted for the
flexure parameter if it is a time-like variable having a discontinuity
at 0/24 hours (=17).
.IP 8.
dispcor.idsmode - should data from all instrument apertures be dispersion
corrected to a uniform wavelength scale? (=yes).
.IP 9.
dispcor.cols_out - the number of columns (row length of the spectrum)
to which the output corrected spectrum should be forced during
mapping (=1024).
.IP 10.
extinct.nr_aps - the number of instrument apertures (=2).
.IP 11.
flatfit.order - the order of the fit to be used when fitting to
the flat field spectra. IRS users have frequently found that
any curvature in the fit introduces wiggles in the resulting
calibrations and a straight divide by the flat normalized to the
mean works best (=1).
.IP 12.
flatfit.coincor - apply coincidence correction to the flat field
spectra during accumulations (=no).
.IP 13.
flatdiv.coincor - apply coincidence correction to all spectra during
the flat field division process (=no).
.IP 14.
identify.function - the fitting function to be used during the wavelength
solution process (=chebyshev).
.IP 15.
identify.order - the order of the fit to be used during the wavelength
solution process. The IRS has strong deviations from linearity
in the dispersion and a fairly high order is required to correct
the dispersion solution (=8).
.RE
.sp 1
.SH
ONEDUTIL
.PP
This is a group of utility operators for the ONEDSPEC package. They
are documented separately after the ONEDSPEC operators. ONEDUTIL
is a "pseudo-package" - it acts like a package under ONEDSPEC, but
many of its logical tasks are physically a part of ONEDSPEC. This
is done to minimize disk storage requirements, and to logically
separate some of the functions from the main ONEDSPEC menu which
was getting too large to visually handle.
.sp 1
.SH
PROCESS
.PP
This task generally does not exist until the user executes the
script task BATCHRED which creates PROCESS.CL, a secondary script
file containing a CL command stream to batch process spectra.
The task is defined so that the CL is aware of its potential
existence. It is not declared as a hidden task so that the
user is also aware of its existence and may execute PROCESS
in the foreground or background.
.sp 1
.SH
REIDENTIFY
.PP
This task (written b Frank Valdes) is intended to be used after
IDENTIFY has been executed. Once a wavelength solution has been
found for one comparison spectrum, it may be used as a starting point
for subsequent spectra having similar wavelength characteristics.
REIDENTIFY provides a batch-like means of performing wavelength solutions
for many spectra. The output solution is directed to a database text file
used by DISPCOR.
.sp 1
.SH
SENSFUNC
.PP
This task solves for the system sensitivity function across
the wavelength region of the spectra by comparison of observations
of standard stars to their (assumed) known energy distribution.
Each instrument aperture is treated completely independently
with one exception discussed later. SENSFUNC is probably the
largest task in the ONEDSPEC package due to heavy use of
interactive graphics which represents more than half of the
actual coding.
.PP
Input to SENFUNC is the "std" text file produced by STANDARD 
containing the ratio of the count rate adjusted for atmospheric extinction 
to the flux of the star in ergs/cm2/s/Angstrom. Both the count rates and
fluxes are the average values in the pre-defined bandpasses tabulated
in the calibration file (indicated by the parameter onedspec.calib_file).
.PP
Each entry is the "std" file may have an independent set of wavelength sampling
points. After all entries have been loaded, a table containing all sampled
wavelengths is built (a "composite" wavelength table) and all sensitivity
values are reinterpolated onto this sampling grid. This allows the inclusion
of standards in which the observational samples are not uniform.
.PP
When multiple measurements are available, one of two corrections may
be applied to the data to account for either clouds or an additive extinction
term. The effect of clouds is assumed to be grey. Each contributing
observation is compared to the one producing the highest count rate ratio
at each wavelength sample. The deviation averaged over all wavelengths
for a given observation is derived and added back to
each wavelength sample for that observation. This produces a shift
(in magnitudes) which, on the average across the spectrum, accounts
for an extinction due to clouds. This process is called "fudge"
primarily for historical reasons (from the IPPS, R.I.P.) and also
because there is questionable justification to apply this correction.
One reason is so that one can better assess the errors
in the data after a zero-point correction has been made.
Another is that the sensitivity function is that closest to a cloud-free
sky so that claibrations may approach a true flux system if one
standard was observed during relatively clear conditions.
Alsom there are claims that the "color solution" is improved by "fudging", but
I admit that I don't fully understand this argument.
.PP
[Perhaps it goes as follows: 
Although a grey scale correction is applied to each observation,
a color term is introduced in the overall solution. Consider the
case where 5 magnitudes of cloud extinction obscure one standard
relative to another. This star generates a sensitivity curve which
is a factor of 100 smaller. When averaged with the other curve,
any variations are lost, and the net curve will be
very similar to the first curve divided by 2. Now apply a "fudge"
of 5 magnitudes to the second curve. On the average, both curves have
similar amplitudes, so variations in the second now influence the
average. The net curve then has color dependent variations not
in the "un-fudged" net curve. If we assume that the variations in
the individual observations are not systematic, then "fudge" will 
improve the net color solution. Amazing, isn't it?
End of hypothesis.]
.PP
The second form of correction is much more justifiable. In ONEDSPEC
it is referred to as a "grey shift" and accounts for possible
changes in the standard atmospheric extinction model due to
a constant offset. SENSFUNC will optionally solve for this constant
provided the observations sample a range of airmass values.
The constant is computed in terms of magnitudes per airmass, so
if the airmass range is small, then a large error is likely.
To solve for this value, a list of pairs of delta magnitude (from the
observation having the greatest sensitivity) as a function of
delta airmass (relative to the same observation) is generated 
for all observations. The list is fit using a least squares solution 
of the form:
.sp 1
.DS L
     delta_mag = delta_airmass * grey_shift
.DE
.sp 1
Note that this is a restricted least-squares in the sense that there
is no zero-point term. The standard curve fit package in IRAF
does not support this option and the code to perform this is included
in SENSFUNC.
.PP
Because the atmosphere is likely to be the same one for observations
with each instrument aperture, it is not appropriate to limit
the least-squares solution to the individual apertures, but rather
to combine all the data to improve the solution. This would mean
that the user could not view the effects of applying the grey term
until all apertures had been analyzed. So, although each aperture is
solved independently to derive a preliminary value, a final value is
computed at the end when all data have been reviewed. This is the
one exception to the independent aperture equals independent
instrument philosophy.
.PP
When "fudging" is applied, the sensitivity function that is generated
is altered to account for the shifts to the observations. But when
the "grey shift" is computed, it cannot be directly applied to
the sensitivity function because it must be modified by the
observing airmass for each individual object. So the grey shift
constant is written into the image headers of the generated
sensitivity functions (which are IRAF images), and also placed
into the task parameter "add_const" to be used later by BSWITCH.
.PP
SENSFUNC can be run in an interactive mode to allow editing
of the sensitivity data. There are two phases of interaction:
(1) a review of the individual observations in which every data
element can be considered and edited, and (2) a review of the
composite sensitivity table and the calculated fit to the table.
In the interactive mode, both phases are executed for every instrument
aperture.
.PP
At both phases of the interactive modes there will be a plot of the
error in the input values for each wavelength. This is an RMS
error. [The IPPS plotted standard error which is always a smaller number
and represents the error in the mean; the RMS represents the error
in the sample. I'm not sure which is better to use, but RMS is easier
to understand. RMS is the same as the standard deviation.]
During phase one, the rms is computed as the standard deviation of
the sensitivity in magnitudes; but during phase two, it is computed
as the standard deviation in raw numbers
and then converted to a magnitude equivalent. The latter is more
correct but both converge for small errors.
.PP
There is one option in SENSFUNC which has never been tried and it won't
work - the option to enter a predefined table of sensitivities as
a function of wavelength as a simple text file. This option may
be useful a some time and should probably be fixed. I think the
only problem with it is a lack of consistency in the units.
.PP
An additional option has been requested but it is not clear that it
is a high priority item - the ability to compute the extinction
function. There may be instances when the mean extinction table
is not appropriate, or is not known. If sufficient data are
available (many observations of high precision over a range of airmasses
during a photometric night), then the extinction function is
calculable. Presently SENSFUNC can only compute a constant offset to
the extinction function, but the same algorithm used may be applied
at each wavelength for which observations are made to compute a
correction to an adopted extinction function (which may be zero),
and the correction can then be written out to the revised extinction
table file. This file will then be read by BSWITCH during the
extinction correction process.
So at each wavelength, pairs of delta magnitude as a function of
delta airmass are tabulated and fit as above:
.sp 1
.DS L
     delta_mag[lambda] = delta_airmass * delta_extinction[lambda]
.DE
.sp 1
Because the data have been heavily subdivided into wavelength bins,
there are only a few measurements available for solving this
least-squares problem and the uncertainties are large unless many
observations have been taken. Experience has shown that at least
7-8 measurements are needed to come close, and 15 measurements are
about the minimum to get a good solution. Unless the data are of
high quality, the uncertainty in the solution is comparable to
the error in assuming a constant offset to the mean extinction function.
Nevertheless, the option should be installed at some time since
some observers do obtain the necessary data. 
.sp 1
.SH
SLIST
.PP
The spectrum specific header elements are listed in either a short
or long form. See the discussion on headers (section 3) for an explanation
of the terms. Values for airmass are printed if present in the header;
otherwise, the value is given as the string "?????" to indicate no
value present (even if one can be calculated from the telescope
pointing information elsewhere in the header).
.PP
The short form header lists only the image name, whether it is
an object or sky observation, the spectrum length, and the title.
.sp 1
.SH
SPLOT
.PP
This is probably the second largest task in the ONEDSPEC package. It continues
to grow as users provide suggestions for enhancement, although
the growth rate appears to be slowing. SPLOT is an interactive
plot program with spectroscopy in mind, although it can be used
to plot two dimensional images as well.
.PP
SPLOT should still be considered a prototype - many of the algortihms
used in the analysis functions are crude, provided as interim
software to get results from the data until a more elaborate package
is written. It would probably be best to create an analysis specific
package - SPLOT is reasonably general, and to enhance it further
would complicate the keystroke sequences.
.PP
Ideally it should be possible to do anything to a spectrum with
a single keystroke. In reality, several keystrokes are required.
And after 15 or 20 functions have been installed, the keystroke
nomenclature becomes obscure - all the best keys are used up, and
you have to resort to things like '(' which is rather less
mneumonic than a letter. So some of the functionality in SPLOT
has been assigned to the "function" submenu invoked by 'f' and
exited by 'q' keystrokes. These include the arithmetic  operators:
add, multiply by a constant, add, subtract, multiply, divide by
a spectrum, and logarithms, square root, inverse, and absolute
value of a spectrum.
.PP
Some of the analysis functions include: equivalent width, line centers,
flux integration under a line, smoothing, spectrum flattening,
and deblending of lines.
.PP
The deblender  has serious limitations but handles about half the
cases that IIDS/IRS users are interested in. It fits only
Gaussian models to the blends, and only a single width parameter.
The fit is a non-linear least-squares problem, so starting values
present some difficulties. All starting values are initialized to 1.0 -
this includes the width, relative strengths of the lines, and deviation
from intial marked centers. The iterative solution usually converges
for high signal-to-noise data, but may go astray, resulting in
a numerical abort for noisy data. If this occurs, it is often
possible to find a solution by fitting to a single strong line
to force a better approximation to the starting values, and then refit
the blend of interest.
.PP
The non-linear least-squares routine is one obtained from an industrial
source. The code is very poorly written and in FORTRAN. No one should
attempt to understand it. The basic algorithm is an unconstrained simplex
minization search combined with a parabolic linear least-squares approximation
when in the region of a local minimum.
A test was made comparing this to the algorithm in Bevington, and the
Bevington algorithm appeared less likely to converge on noisy data.
Only one test case was used, so this is hardly a fair benchmark.
.PP
The problem with non-convergence is that a floating point error is
almost surely to arise. This is usually a floating point over/under
flow while computing an exponential (as required for a Gaussian).
In UNIX, there is apparently no easy way to discriminate from
FORTRAN which floating point exception has occurred, and so there
is no easy way to execute a fix up and continue. This is most
unfortunate because the nature of these non-linear techniques is
that given a chance, they will often recover from searching
down the wrong alley. A VMS version of the same routines seems to
survive the worst data because the error recovery is handled
somewhat better. [The VMS version also seems to run much faster,
presumably because the floating point library support is better
optimized.]
.PP
The net result of all this, is that a weird undocumented subroutine
is used which provides no error estimate. The Bevington routines
do provide an error estimate which is why I wanted to use them.
[In fact, there is no way to exactly compute the errors in the
fit of a non-linear least-squares fit. One can however apply
an approximation theory which assumes the hypersurface can be
treated locally as a linear function.]
.PP
There are several methods for computing equivalent widths in SPLOT.
The first method for measuring equivalent width is simply to integrate the
flux above/under a user defined continuum level. Partial pixels
are considered at the marked endpoints. A correction for the pixel size, 
in Angstroms, is applied because the units of equivalent width are Angstroms.
You will probably get a different answer when doing equivalent
width measurements in channel mode ('$' keystroke) as compared to
wavelength mode ('p').
.PP
Centering is performed as a weighted first moment of the region:
.sp 1
.DS L
     int1 = integral [ (I-Ic) * sqrt (I-Ic) * w]
     int2 = integral [ (I-Ic) * sqrt (I-Ic)    ]
     xc = int1 / int2
.DE
.sp 1
where I is the intensity at the pixel at wavelength w, and Ic is
the estimated continuum intensity. The square root term provides
the weighting assuming photon statistics [sigma = sqrt(I)], and xc
is the derived center of the region.
.PP
An alternative method for equivalent widths was supplied by Caty
Pilachowski and is described in some detail in the help file for
SPLOT. This method is fast and insensitive to cursor settings, so
the user can really zip through a spectrum quickly.
.PP
Smoothing is performed using a simple boxcar smooth of user specified
size (in pixels). To handle edge effects, the boxcar size is
dynamically reduced as the edge is approached, thereby reducing
the smoothing size in those regions.
.PP
The flattening operator is a preliminary one, written before the
curve fitting package was available in IRAF. This operator
should probably be re-written to include the interactive
style used in FLATFIT. Currently the flattening is done
using classic polynomial least-squares with pixel rejection
chosen to preferentially reject absorption lines and strong
emission lines. The rejection process is repeated through
a number of iterations specifiable as a hidden parameter to SPLOT.
This is poorly done - the order of the fit and the number of
iterations should be controllable while in SPLOT. However,
experimentation has shown that for a given series of spectra,
the combination of rejection criteria, order, and iteration count
which works well on one spectrum will generally work well
on the other spectra. Note that the flatten operator attempts to
find a continuum level and normalize to that continuum, not to the
average value of the spectrum.
.PP
There are also the usual host of support operators - expansion,
overplotting, and so forth. There is also a pixel modifer mode
which connects two cursor positions. This forces a replot of the entire
spectrum after each pair of points has been entered. This should
probably be changed to inhibit auto-replot.
.PP
Some users have requested that all two cursor operators allow
an option to escape from the second setting in case the wrong
key was typed. I think this is a good idea, and might be implemented
using the "esc" key (although I could not seem to get this keystroke
through the GIO interface).
.PP
Another user request is the option to overplot many spectra with
autoscaling operational on the entire range. This is also a good
idea. Yet another improvement could be made by allowing the user
to specify the x and y range of the plot, rather than autoscaling.
.PP
There is one serious problem with respect to plotting spectra
corrected to a logarithmic wavelength scale. It would be nice to
plot these spectra using the logarithmic axis option, but this
option in GIO requires that at least one entire decade of x axis
be plotted. So for optical data, the x axis runs from 1000 Angstroms
to 10,000 Angstroms. Imagine a high dispersion plot having only 100
Angstroms of coverage - the plot will look like a delta function!
The current version of SPLOT uses a linear axis but plots in
the log10 of wavelength. Not very good, is it.
.sp 1
.SH
STANDARD
.PP
This task computes the sensitivity factor of the instrument
at each wavelength for which an a priori measured flux value is known
and within the wavelength range of the observations.
Sensitivity is defined as
[average counts/sec/Angstrom]/[average ergs/cm2/sec/Angstrom]
over the specified bandpass for which the star has been measured.
Both numerator and denominator refer to quantities above the
Earth's atmosphere and so the count rates must be corrected for
extinction.
The wavelengths of known measurements, the bandpasses, the
fluxes (in magnitudes), and the mean extinction table
are read from a calibration file whose name is specified
by the calib_file parameter (see LCALIB for a description of this
file). If a magnitude is exactly 0.0, it is assumed 
that no magnitude is known for this star at the wavelength
having a 0.0 magnitude. This allows entries having incomplete
information.
.PP
As each observation is read, it is added into an accumulator for
its aperture. Or subtracted if it is a sky measurement. After
a pair of object and sky observations have been added, the
difference is corrected for extinction (as in BSWITCH), converted
to counts per second, and integrations performed over the bandpasses
for which flux measures are known. The bandpasses must be completely
contained within the spectrum - partial coverage of a bandpass
disqualifies it from consideration. The integrations are compared
with the known flux values and the ratio is written to a text
file (the "std" file) along with the wavelength of the measurement
and the total counts in the bandpass. The total counts value may
be used by SENSFUNC for weighting the measurements during averaging.
.PP
Many users are surprised by the order of the spectral names
printed out as STANDARD executes since the order is not necessarily
ascending through the spectrum list. This is because the name
printed is the name of the object spectrum most recently associated
with an object-sky pair. So if a sky pair is several spectra down the
list, an intervening object-sky pair taken through a different
instrument aperture may be processed in the meantime.
For example, say spectra 1-8 are taken so that object spectra
numbers 1 and 7 and sky spectra 3 and 5 are taken through aperture 0, 
object spectra 4 and 6  and sky spectra 2 and 8 are taken through
aperture 1. [This is a very common pattern for IIDS/IRS users.]
Then spectrum 1 and 3 will pair up and be processed first (spectrum
name 1 will be printed). Then 4 and 2 (name 4 printed), then 
7 and 5 (name 7 printed), and then 6 and 8 (name 6 printed).
So the order of names printed will be 1,4,7,6.  Simple, isn't it?
.PP
If the input spectra are not taken in a beam-switched mode
then the parameter "beam_switch" should be set to no.
Then no sky subtraction will be attempted.
.PP
The user may enter sensitivity values directly into a file and use
it as the "std" file for a correction.
See the help file for STANDARD for a description of the entries in
the file, and see a typical file.
.PP
STANDARD offers a limited interactive mode. The first sky subtracted
spectrum is displayed and the bandpasses at which sensitivity
measurements are made will be shown as boxes. This provides a means
to see where the measurements are falling on the observational
data and to assess whether a bandpass may be including some
absorption edge which may be affecting the measurement. While it
is true that the wavelengths of the reference measurements should
fall in the same place, the effects of instrument resolution and
inaccuracies in the wavelength calibration may shift the positions
of the apparent bandpasses. The samples may then be biased.
.PP
The second purpose of the interactive mode is to allow the user
to artificially create new bandpasses on the fly. By placing the
cursor to bound a new wavelength region, STANDARD will interpolate
in the magnitude table of the reference star to estimate the magnitude
of the star at the bounded wavelength. The sensitivity will be calculated
at that wavelength just as if the bandpass had come from the calibration
file. This option should be exercised with care. Obviously, points
should not be generated between reference wavelengths falling on
strong absorption lines, or on a line either. This option is most useful
when at a high dispersion and few samples happen to fall in the
limited wavelength region. Sufficient space is allocated for 10
artificial samples to be inserted. Once the artificial bandpasses
have been designated, they are applied to the entire sequence of
spectra for the current invocation of STANDARD. Once STANDARD
completes, the added bandpasses are forgotten. This prevents
an accidental usage of newly created bandpasses on stars of different
spectral types where a bandpass may fall in a region of continuum
for one star, but on an absorption line in another.
.sp 1
.SH
SUBSETS
.PP
This is a simple task to subtract the second spectrum from the
first in a series of spectra. So if spectra 1-10 are input,
5 new spectra will be created from 1 minus 2, 3 minus 4, and so on.
This is a straight subtraction, pixel for pixel, with no
compensation for exposure time differences.
The header from the first spectrum of the pair is applied to the
output spectrum.
.sp 1
.SH
The ONEDUTIL tasks
.PP
These utility tasks are logically separated from the ONEDSPEC
package.
.sp 1
.SH
COEFS
.PP
This task reads the header parameters contained in comparison arc spectra
describing the wavelength solution generated by the mountain reduction
program and re-writes the solution parameters into a database
text file for use by DISPCOR. Otherwise those solutions would be
lost. COEFS assumes that the coefficients represent a Legendre
polynomial which is what the mountain reduction programs use.
.sp 1
.SH 
COMBINE
.PP
When an object has been observed over a wide range of wavelength
coverage by using more than one instrumental setup (such as
a blue and a red setting) or with different instruments (such
as IUE and the IRS), it is often desirable to combine the 
spectra into a single spectrum. COMBINE will rebin a group of
spectra to new spectra having a single dispersion and average the
new spectra to create a single long spectrum.
If there are gaps in the composite spectrum, zeroes are used
as fillers. Ideally those pixels which have no known value
should be considered blank pixels. IRAF does not currently
support blank pixels, so zeroes are used for now. [One
might suggest using INDEF, but then all other routines will
have to check for this value.]
A side effect of choosing 0.0 is that during the averaging
of overlapping spectra, a true 0.0 will be ignored by COMBINE.
The basic rebinning algorithms used in DISPCOR are used in COMBINE
(and also REBIN).
.PP
The averaging can be weighted by exposure time, or by user assigned weights. 
It would be better if each spectrum had an associated vector of
weights (one weight at each wavelength) so that the weighted averaging
could be done on a pixel basis. This is very expensive in terms
of both storage and file access overhead since each spectrum would 
require twice the storage and number of files. 
[Actually weights could be small 4 bit integers and take up very little space.]
.PP
A less ideal alternative would be to place a small number 
(about 16) of weight parameters
in the header file which represent the approximate weights of that many
regions of the spectrum, and then one could interpolate in these parameters
for a weight appropriate to the pixel of interest.
.PP
A third solution (and even less ideal)
is to place a single parameter in the header which
represents an average weight of the entire spectrum. For the latter two cases,
the header weights could be derived from the average counts per 
wavelength region - the region being the entire spectrum in the last case.
The weights must be entered into the header during the BSWITCH operation
since that is the last time that true counts are seen. [An implicit
assumption is that counts are proportional to photons. If data from
two different instruments are to be averaged, then the weights should be
expressed in photons because the ratio of counts to photons is highly
instrument dependent.]
.PP
COMBINE suffers from a partial pixel problem at the end points.
Interpolation at the ends can lead to an underestimate of the flux
in the last pixels because the final pixel is not filled. When averaging
in data from another spectrum or instrument, these pixels show up
as sharp drops in the spectrum. The problem appears due to the
rebinning algorithm and should be corrected someday (also in DISPCOR
and REBIN).
.sp 1
.SH
LCALIB
.PP
This utility provides a means of checking the calibration files
containing the standard star fluxes and extinction table. 
Any of the entries in the file may be listed out - the bandpasses,
extinction, standard star names, standard star fluxes in either
magnitudes, flambda, or fnu. For a description of the calibration
file format, see the help documentation for LCALIB.
.PP
The primary uses for LCALIB are to verify that new entries in
the tables are correct, to generate a list of standard star names
in a calibration file, and to produce a table of fluxes for a given standard
star. The table may then be used to generate a spectrum over a specified
wavelength region using SINTERP and overplotted with observational
data to check the accuracy of the reductions.
.sp 1
.SH
MKSPEC
.PP
MKSPEC provides a way to generate a limited set of artificial
spectra. Noise generation is not available. The current options
are to generate a spectrum which is either a constant, a ramp,
or a black body. The spectrum may be two dimensional, but
all image lines will be the same.
.sp 1
.SH
NAMES
.PP
This is the simplest task in the ONEDSPEC package. It
generates the image file names which are implied by a
root name and record string. The primary use for this
task is to generate a list of image names to be used
as input for some other program such as WFITS.
The output from NAMES can be redirected to file
and that file used with the "@file" notation for image
name input. An optional parameter allows an additional
string to be appended to the generated file name 
to allow a subraster specification.
.sp 1
.SH
REBIN
.PP
Spectra are rebinned to the wavelength parameters specified
by either matching to a reference spectrum or by user input.
The algorithms are those used by DISPCOR and the same options
for the interpolation method are available. REBIN is useful
when data are obtained with different instruments or setups
producing roughly comparable wavelength ranges and possibly
different dispersions, and the data are to be compared.
REBIN may also be used as a shift operator by specifying a
new starting wavelength. Or it may be used as a smoothing operator
by specifying a course dispersion. It may also be used
to convert between the two formats - linear in wavelength and 
linear in the logarithm of wavelength. This latter option has
not been thoroughly exercised - proceed with caution.
.sp 1
.SH
RIDSMTN
.PP
This task was stolen from the DATAIO package to make the following
modification: IIDS and IRS data are both written as 1024 pixel
spectra at the mountain. But the detectors do not produce a full
1024 pixels of acceptable data. In fact the IRS only has 936 pixels.
The data are written this way to conform to the IIDS ideal spectrum
which does have 1024 pixels, but the first few (about 6) are not usable.
To signal the good pixels, the IIDS/IRS header words NP1 and NP2 are
set to the beginning and ending good pixels. Actually NP1 points to
the first good pixel minus one. [Really actually NP1 and NP2 may be reversed,
but one is big and the other small so you can tell them apart.]
.PP
The version of RIDSMTN in ONEDUTIL keys off these parameters and writes
images containing only good pixels which means that the images will be
smaller than 1024 pixels. The user has the option of overriding the
header values with the task parameters "np1" and "np2". These may be
specified as 1 and 1024 to capture the entire set of pixels written to
tape or any other subset. Beware that np1 and np2 as task parameters
refer to the starting pixel and ending pixel respectively. None of this
nonsense about possible role reversals or "first good minus one" is
perpetuated.
.sp 1
.SH
SINTERP
.PP
I think this is a handy little program. It provides a way to make
an IRAF spectral image from a table of values in a text file.
The table is interpolated out to any length and at any sampling
rate. A user can create a table of corrections to be applied to
a set of spectra, for example, use SINTERP to build a spectrum,
and run CALIBRATE to multiply a group of spectra by the correction.
.PP
The original raison d'etre for SINTERP was to create spectra of
standard stars from the listing of fluxes generated by LCALIB.
Using SPLOT the created spectrum can be overplotted with calibrated
observations to compare the true tabulated fluxes with the observed
fluxes.
.PP
SINTERP grew out of the task INTERP in the UTILITIES package
and works pretty much the same way. One major change is that
the table containing the x-y pairs is now stored in a dynamically
allocated array and can be as large as the user requests. The
default size is 1024 pairs, but the parameter tbl_size can
be set to a larger value. This then allows one to create a spectrum
from its tabulated values of wavelength and flux even if the
the table is several thousand elements long.
Note that the option to route the output from INTERP to
STDOUT has been retained if a new table is to be generated rather
than an IRAF image.
.PP
Another major change from INTERP is the use of the IRAF curve fitting
routines as an option. These were not originally available.
The choices now include linear or curvey interpolators, Legendre
or Chebyshev polynomial fits, and cubic or linear splines.
.sp 1
.SH
WIDSTAPE
.PP
This task has vague origins in the DATAIO task WIDSOUT which writes
a tape having the format of the IDSOUT package which ran on the
CYBER (R.I.P.). For convenience to users this format has been
maintained for spectra having lengths up to 1024 pixels.
The version in DATAIO requires that the user enter all the header
parameters as task parameters. For several hundred spectra, this
approach is unwieldy. Because the ONEDSPEC package uses the header
parameters heavily, it is able to read them directly and write the
values to the tape file without user intervention.
.PP
The output tape (or diskfile) may be in either ASCII or EBCDIC format.
Spectra shorter than 1024 are zero filled. Each invocation of
the task write a new tape file followed by a tape mark (EOF).
.LP
.SH
3. Image Header Parameters
.PP
The ONEDSPEC package uses the extended image header to extract
information required to direct processing of spectra. If the
header information were to be ignored, the user would need to
enter observing parameters to the program at the risk of
typographical errors, and with the burden of supplying the
data. For more than a few spectra this is a tedious job,
and the image header information provides the means to eliminate
almost all the effort and streamline the processing.
.PP
However, this requires that the header information be present,
correct, and in a recognizable format. To meet the goal of
providing a functional package in May 1985, the first iteration
of the header format was to simply adopt the IIDS/IRS headers.
This allowed for processing of the data which would be first
used heavily on the system, but would need to be augmented at
a later date. The header elements may be present in any order,
but must be in a FITS-like format and have the following names
and formats for the value fields:
.sp 1
.TS
l c l
l l l.
Parameter	Value Type	Definition

HA	SX	Hour angle (+ for west, - for east)
RA	SX	Right Ascension
DEC	SX	Declination
UT	SX	Universal time
ST	SX	Sidereal time
AIRMASS	R	Observing airmass  (effective)
W0	R	Wavelength at center of pixel 1
WPC	R	Pixel-to-pixel wavelength difference
NP1	I	Index to first pixel containing good data (actually first-1)
NP2	I	Index to last pixel containing good data (last really)
EXPOSURE	I	Exposure time in seconds (ITIME is an accepted alias)
BEAM-NUM	I	Instrument aperture used for this data (0-49)
SMODE	I	Number of apertures in instrument minus one (IIDS only)
OFLAG	I	Object or sky flag (0=sky, 1=object)
DF-FLAG	I	Dispersion fit made on this spectrum (I=nr coefs in fit)
SM-FLAG	I	Smoothing operation performed on this spectrum (I=box size)
QF-FLAG	I	Flat field fit performed on this spectrum (0=yes)
DC-FLAG	I	Spectrum has been dispersion corrected (0=linear, 1=logarithmic)
QD-FLAG	I	Spectrum has been flat fielded (0=yes)
EX-FLAG	I	Spectrum has been extinction corrected (0=yes)
BS-FLAG	I	Spectrum is derived from a beam-switch operation (0=yes)
CA-FLAG	I	Spectrum has been calibrated to a flux scale (0=yes)
CO-FLAG	I	Spectrum has been coincidence corrected (0=yes)
DF1	I	If DF-FLAG is set, then coefficients DF1-DFn (n <= 25) exist
.TE
.PP
The values for the parameters follow the guidelines adopted for
FITS format tapes. All keywords occupy 8 columns and contain
trailing blanks. Column 9 is an "=" followed by a space. The value field
begins in column 11. Comments to the parameter may follow a "/" after
the value field.  The value type code is as follows:
.RS
.IP SX
This is a sexigesimal string of the form '12:34:56   ' where the first
quote appears in column 11 and the last in column 30.
.IP R
This is a floating point ("real") value beginning in column 11 and
extending to column 30 with leading blanks.
.IP I
This is an integer value beginning in column 11 and extending to
column 30 with leading blanks.
.RE
.sp 1
.PP
The parameters having FLAG designations all default to -1 to indicate
that an operation has not been performed.
The ONEDSPEC subroutines "load_ids_hdr" and "store_keywords" follow
these rules when reading and writing spectral header fields.
If not present in a header, load_ids_hdr will assume a value of zero
except that all flags are set to -1, and the object flag parameter
defaults to object.
.PP
When writing an image, only the above parameters are stored by store_keywords.
Other header information is lost. This needs to be improved.
.PP
Not all programs need all the header elements. The following table
indicates who needs what. Tasks not listed generally do not require
any header information. Header elements not listed are not used.
The task SLIST requires all the elements listed above.
The task WIDTAPE requires almost all (except NP1 and NP2).
The headings are abbreviated task names as follows:
.sp 1
.nr PS 8
.ps 8
.TS
center;
l l | l l | l l.
ADD	addsets	COI	coincor	FIT	flatfit
BSW	bswitch	COM	combine	REB	rebin
CAL	calibrate	DIS	dispcor	SPL	splot
COE	coefs	FDV	flatdiv	STA	standard
.TE
.sp 1
.TS
center, tab(/);
l | l | l | l | l | l | l | l | l | l | l | l | l.
Key/ADD/BSW/CAL/COE/COI/COM/DIS/FDV/FIT/REB/SPL/STA
_
HA// X////////// X/
RA// X////////// X/
DEC// X////////// X/
ST// X////////// X/
UT// X////////// X/
AIRMASS// X////////// X/
W0// X/ X/// X//// X/ X/ X/
WPC// X/ X/// X//// X/ X/ X/
NP1/////////// X///
NP2/////////// X///
EXPOSURE/ X/ X/// X/ X///// X///
BEAM-NUM// X/ X//// X/ X/ X// X/ X//
OFLAG// X////////// X/
DF-FLAG//// X
DC-FLAG// X//// X//// X/ X/ X/
QD-FLAG//////// X/
EX-FLAG// X/
BS-FLAG// X/
CA-FLAG/ X// X//////// X/
CO-FLAG///// X//
DFn//// X/
.TE
.nr PS 11
.ps 11
.bp
.SH
Headers From Other Instruments
.PP
The header elements listed above are currently created only when reading
IIDS and IRS data from one of the specific readers: RIDSMTN and RIDSFILE.
The time-like parameters, (RA, DEC, UT, ST, HA), are created in a
compatible fashion by RCAMERA and RFITS (when the FITS tape is written
by the KPNO CCD systems).
.PP
For any other header information, the ONEDSPEC package is at a loss
unless the necessary information is edited into the headers with
an editing task such as HEDIT. This is not an acceptable long term
mode of operation, and the following suggestion is one approach to
the header problem.
.PP
A translation table can be created as a text file which outlines
the mapping of existing header elements to those required by the
ONEDSPEC package. A mapping line is needed for each parameter
and may take the form:
.sp 1
.RS
.DC
1D_param    default    hdr_param    key_start   value_start   type   conversion
.DE
.RE
.sp 1
where the elements of an entry have the following definitions:
.sp 1
.TS
center, tab(	);
l lw(5i).
1D_param	T{
The name of the parameter expected by the ONEDSPEC package,
such as EXPOSURE, OFLAG, BEAM-NUM.  
T}

default	T{
A value to be used if no entry is found for this parameter or if
no mapping exists.
T}

hdr_param	T{
The string actually present in the existing image header to be
associated with the ONEDSPEC parameter. 
T}

key_start	T{
The starting column number at which the string starts 
in the header. 
T}

value_start	T{
The starting column number at which the string describing the
value of the parameter starts in the header. 
T}

type	T{
The format type of the parameter: integer, real, string, boolean,
sexigesimal. 
T}

conversion	T{
If the format type is string, a further conversion may 
optionally be made to one of the formats listed under type. 
The conversion may requires some expression evaluation.
T}
.TE
.sp 1
.PP
Consider the example where the starting wavelength of a
spectrum is contained in a FITS-like comment and the object-
sky flag in a similar fashion:
.sp 1
.DS
     COMMENT = START-WAVE  4102.345   / Starting wavelength
     COMMENT = OBJECT/SKY  'SKY      '/ Object or Sky observation
.DE
.sp 1
The translation file entries for this would be:
.sp 1
.DS
     W0  0.0  START-WAVE  12  24  R
     OFLAG 0  OBJECT/SKY  12  25  S  SKY=0;OBJECT=1
.DE
.sp 1
The first entry is fairly simple. The second requires an expression
evaluation and second conversion.
.PP
A translation file can be built for each instrument and its
special header format, and the file name can be associated with a
ONEDSPEC package parameter. The two subroutines in ONEDSPEC dealing
directly with the headers (load_ids_hdr and store_keywords)
can be modified or replaced to access this file and
translate the header elements.