path: root/noao/twodspec/multispec/doc/MSspecs.hlp

.help multispec Oct83 "Multi-Spectra Extraction Package"
.sp 3
.ce
Detailed Specifications for the Multi-Spectra Extraction Package
.ce
F. Valdes
.ce
December 8, 1983
.sh
1.  Introduction

     The multi-spectra extraction package (MULTISPEC) provides the basic tools
for modeling, cleaning, and extracting spectra from images
containing multiple aperture spectra running roughly parallel.
These tools will generally be combined in reduction script tasks
but may also be used directly for non-standard analysis.

     This design presents the requirements and specifications
for the MULTISPEC package.  Details concerning the 
algorithms are given in a separate document, Algorithms for the
Multi-Spectra Extraction Package.
.sh
2.  Input Data Requirements

    The input data for the MULTISPEC package consists of image
files containing one or more aperture spectra.  The spectra are
required to run roughly parallel to each other and parallel to the
second digitization axis.  The latter requirement may require a
general rotation and interpolation image operator.  The images are
assumed to be corrected to linear relative intensity.  Thus, the
steps of correcting digital detector images for dark current, bias, and
pixel-to-pixel sensitivity variations must be performed before using
the MULTISPEC tasks.

     Because the MULTISPEC package is being developed
concurrently with the IRAF standard image processing
tools this document specifies the requirements for the preliminary
image processing needed to prepare digital detector images for the MULTISPEC
package.
.sh
2.1  Basic Digital Detector Reduction Tasks

     The prelimary reduction of multi-spectra images uses CL scripts
based on general image operators.
Some of the scripts are for specific instruments or specific
reduction applications and some are generally useful image processing
tasks.  The scripts allow the specification of many images for which
the operations will be repetitively applied.

     The following CL scripts are required to reduce multi-spectra images
from digital detectors.


.nf
	debias		multispec_flat		flat_divide
.fi
.ke
.ks
.ls 4 debias
The files in a list of filenames are automatically debiased and trimmed.
This routine will be instrument specific but used by other reduction
tasks beyond MULTISPEC.
.le
.ke
.ks
.ls multispec_flat
The files in a list of quartz multi-spectra filenames are added,
the result is smoothed
along the dispersion dimension, and then the original image is divided
by the smoothed image to produce a flat field image.  The unsmoothed
to smoothed ratio is computed only if the value of the smoothed
pixel is greater than a specified amount.  Otherwise, the ratio is set
to unity.  This routine is not instrument specific but is used only
for MULTISPEC reductions.
.le
.ke
.ks
.ls flat_divide
The files in a list of filenames are each divided by a specified flat
field image.  This routine is not instrument or application specific.
.le
.ke

    The required general image processing programs needed to implement these
scripts are specified below.

.ls (1)
A routine to compute the average value from a specified area of the
image.  Used to determine the average bias value from a bias strip.
.le
.ls (2)
A routine to trim a specified portion of an image.  Used to trim the
bias strip.
.le
.ls (3)
Routines to multiply and subtract images by a constant.  Used to scale
images such as dark exposures and to remove the average bias value as
obtained by (1) above.
.le
.ls (4)
Routines to subtract, add, and divide images.  Used to subtract dark
current and bias exposures, to add several exposures to increase the
signal-to-noise, and to divide by a flat field image.
The divide routine must give the user the option to substitute a constant or
ignore any divisions in which the denominator is less than a specified value.
.le
.ls (5)
A routine to rotate or transpose an image.  Used to align the spectra
along lines or columns.
.le
.ls (6)
A routine to apply a filter to lines of the image.  For multi-spectra images
a smooth quartz is produced by using a running quadratic filter along each
line of the dispersion dimension.  The filter must be able to recognize
bad pixels (specified by a user defined threshold) and remove them from the
filtering operation.
.le
.sh
3.  Requirements for the MULTSPEC Package

     The MULTISPEC package shall satisfy the following requirements.
.ls (1)
The component programs shall be CL callable.
.le
.ls (2)
The programs shall interact only through image files and MULTISPEC data files.
.le
.ls (3)
It shall be possible to extract spectra without modeling.
.le
.ls (4)
The entire image shall be extracted and not limited by failures in the
algorithms.
.le
.ls (5)
It shall be possible to specify specific lines or swaths in the image
on which to operate.
.le
.ls (6)
CL scripts shall be provided for the common data sources.  These scripts
will work automatically.
.le

The follow functions shall be provided:
.ls o
Make an initial rough but automated identification of the spectra
locations.
.le
.ls o
Provide for a user identification list for the spectra locations.
This list shall be of the standard image cursor type to allow generation
of the list with the standard image cursor programs.
.le
.ls o
Determine and correct for a slowly varying background.
.le
.ls o
Reliably and accurately trace spectra in the presence of geometric
distortions (pincushion, s, shear, etc.).
.le

Extract spectra by one of:
.ls a.
Strips of constant width about the located spectra.  The width may be specified
to fractions of a pixel and the extraction will use fractional pixel
interpolation.
l
.le
.ls b.
Strips of width proportional to a Gaussian width parameter.
.le
.ls c.
Modeling to obtain estimates of the total luminosity.  The estimate will
be the integral of the model.
.le
.ls d.
Summation of the data pixel values with fractional contributions of the
pixel value to the spectra based on modeling.
.le
.le
.ls o
An option shall be available to specify whether to ignore blank pixels
or use interpolated values.
.ls o
Programs shall be provided to produce data files which can be accessed
by one dimensional spectroscopic reduction routines.  At a minimum
these formats shall include:
.ls a.
Reduction to an image file consisting of one line per extracted
spectrum
.le
.ls b.
The standard IIDS format available with the CYBER Multi-Aperture Plate
programs
.le
.le
.sh
3.2  Modeling Requirements

     The modeling of multi-spectra images, particularly in the case of
blended spectra, shall:
.ls (1)
Model blended spectra with sufficient reliability and robustness that
a reasonable solution is always obtained, though of possibly limited
usefulness.
.le
.ls (2)
The modeling shall provide estimates for the uncertainties in the fitted
parameters as a function of position along the spectrum.
.le
.ls (3)
Remove cosmic rays and other defective pixels by reference to the model.
.le
.ls (4)
Allow the transfer of a model solution for one image to another image.
.le
.ls (5)
Display numerically and graphically the data, the fitted model, and
the residuals.
.le
.sh
4.  Program Specifications
.sh
4.1  Basic Programs

     The basic programs of the package are general purpose tools which
initialize a MULTISPEC data file and perform a single fundamental operation
on the data in the MULTISPEC data file.  There is one data file associated
with each image.  The data file is hidden from the user and so the user
need not be aware of the data file.
The data files are referenced only the image filename specified in the
program parameters.
The data files contain such information as a processing history, the
spectra positions and extracted luminosities, the model parameters (one
set for each spectra for each modelled image line (or swath), etc.
The programs generally are allowed to specify specific
lines, columns, and/or spectra on which to operate.
The line, column and spectra specifications are given as strings which
contain numbers separated by whitespace, commas, and the range indicator
"-".  The script tasks
of section 4.2 will combine these basic programs to perform a general
multi-spectra extraction.

.ks
.nf
    ap_plate		copy_params	find_spectra	convolve
    fit_bckgrnd		find_bckgrnd	line_list	model_extrac
    model_fit		model_image	model_list	sigma_extract
    strip_extract	to_iids		to_image	to_onedspec
.fi
.ke
.ks
.ls ap_plate
The information from an on-line data file containing descriptions of all
the aperture plates prepared at Kitt Peak is read to find a specified
aperture plate.  The drilled aperture positions are correlated with the
spectra in the image to deduce relative wavelength offsets.  The
identifications for the spectra as well as other auxiliary information
is recorded in the data file.
If no image file is specified then only the aperture
plate information is printed.  This program is used in the MULTIAP_EXTRACT
program.  This program is not essential to the operation of the MULTISPEC
package.

.nf
                Multi-Spectra image  image = 
                     Aperture plate  plate = 
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls The Background
The are two possibilities for dealing with the background.  In the first
case, FIT_BCKGRND, the background will be fitted by polynomials and
the coefficients stored in the MULTISPEC data file.  These coefficients
are then used by the other programs to estimate the background at the
spectra.  The second option, FIND_BCKGRND, generates a background image in which
the spectra and other selected areas are set to blank pixels.  Then a
general image interpolator is used fill in the blank pixels with background
estimates.  The other MULTISPEC programs will then access this background
frame.  The background frame image name will be stored in the MULTISPEC
data file and the image header.

.ls fit_bckgrnd
Fit a background in a MULTISPEC image by a polynomial using pixels
not near the spectra and in the user specified swaths and columns.
The buffer distance is in pixels and refers to a minimum distance from
the center of any spectrum beyond which the background pixels are found.
Blank pixels are ignored in the background fit.  Deviant pixels will be
rejected.

.nf
                Multi-Spectra image  image = 
                Buffer from spectra  buffer = 12
                   Polynomial order  order = 3
                    Lines per swath (lines_per_swath = 32)
                      Swaths to fit (swaths = 1-1000)
                     Columns to fit (columns = 1-1000)
		Rejection threshold (threshold = 5)
	  Print general diagnostics (verbose = no)
                                    (mode = ql)
.fi
.le
.ls find_bckgrnd
The spectra within a buffer distance and specified areas are set to blank
pixels and the remaining pixels copied to a background image file.

.nf
                Multi-Spectra image  image = 
		   Background image  background =
                Buffer from spectra  buffer = 12
                    Lines to ignore (lines = )
                  Columns to ignore (columns = )
                                    (mode = ql)
.fi
.le
.le
.ke
.ks
.ls convolve
A program will be provided to reduce either the extracted spectrum or
the modeled image to a common point-spread function.
.le
.ke
.ks
.ls copy_params
Create a MULTISPEC data file for a new image using
appropriate MULTISPEC parameters from an old image.
The old image must have been processed to find the spectra using FIND_SPECTRA
and possibly model fit.

.nf
            Old Multi-Spectra image  old_image = 
            New Multi-Spectra image  new_image = 
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls find_spectra
Initially locate the spectra in a MULTISPEC image.
The positions of the spectra within the range of columns are determined
for the starting line and then the spectra are tracked within the
range of lines.  The minimum separation
and minimum width would generally be set for a particular instrument.
If the automatic search is not used then a list of cursor positions is
read from the standard input.

.nf
                Multi-Spectra image  image = 
		   Automatic search  auto = yes
		      Starting line  start_line =
		 Minimum separation (min_sep = 1)
		      Minimum width (min_width = 1)
                    Averaging width (average = 32)
                    Lines to search (lines = 1-1000)
                  Columns to search (columns = 1-1000)
	  Print general diagnostics (verbose = no)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls line_list
For the specified lines in the image print the image column
number, data value (possibly as a swath average), the model value at that
point (i.e. the sum of the model contributions from all the spectra),
the background value, and the residual.
Plotting scripts may be written using this routine to
show the quality of a model fit.

.nf
                Multi-Spectra image  image = 
                      Lines to list (lines = 1-1000)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls model_extract
A previously fitted model is used to extract the spectra total luminosity
by apportioning the data values to spectra in the ratio indicated by the
model.  If the clean option is specified then the model is used to detect
pixels which deviate from the model by a specified amount.
The model value replaces the deviant pixel in the extraction and, if specified,
also in the image file.

.nf
                Multi-Spectra image  image = 
                   Lines to extract (lines = 1-1000)
                      Clean spectra (clean = yes)
                 Cleaning threshold (threshold = 5)
                       Modify image (modify = yes)
	  Print general diagnostics (verbose = no)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls model_fit
A specified model is iteratively fitted to the data in each of the specified
lines (or swaths) until the RMS residual fails to decrease.  The models
are selected by a string.  The possible values are

.nf
    (null string) - initialize the model
    i - fit only the intensity scale
    ip - fit the intensity scale and the position
    ips1 - fit the intensity scale, position, and one parameter shape
    ips2 - fit the intensity scale, position, and two parameter shape
    ips3 - fit the intensity scale, position, and three parameter shape
    ips4 - fit the intensity scale, position, and four parameter shape
.fi
These models will be combined in a script to search for the best fit.

The initial shape parameters will generally be set by scripts for a
particular data reduction.

.nf
                Multi-Spectra image  image = 
                         Model type  model = 
                    Lines per swath (lines_per_swath = 32)
                    Swaths to model (swaths = 1-1000)
		     Initial shape1 (shape1 = .1 )
		     Initial shape2 (shape2 = 0 )
		     Initial shape3 (shape3 = 0 )
		     Initial shape4 (shape4 = 5 )
	  Print general diagnostics (verbose = no)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls model_image
An image file of the fitted model is created.  This image may then be displayed
or a residual image may be calculated and displayed.

.nf
                Multi-Spectra image  image = 
		        Model image  model =
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls model_list
For the specified lines and spectra the model is listed.
The listing gives, for each spectra,
the spectrum number, the line number, the fitted position,
the estimated wavelength, the
extracted luminosity, the intensity scale, model width parameters, and
the background polynomial coefficients.  This routine can be used in scripts
to plot the extracted spectra, the trend of width with wavelength, and so
forth.

.nf
                Multi-Spectra image  image = 
                      Lines to list (lines = 1-1000)
                    Spectra to list (spectra = 1-1000)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls sigma_extract
A previously fitted model is used to extract the spectra luminosity
within a specified sigma of the peak.  Because the model is not necessarily
a Gaussian the sigma is used to compute
the intensity ratio of the cutoff to the peak assuming a Gaussian profile
and then the data is extracted to the point the model intensity falls below that
cutoff.  If the clean option is specified then the model is used to detect
pixels which deviate from the model by a specified amount.
The model value replaces the deviant pixel in the extraction and, if specified,
also in the image file.

.nf
                Multi-Spectra image  image = 
             Sigma extraction width  width = 1.
                   Lines to extract (lines = 1-1000)
                      Clean spectra (clean = yes)
                 Cleaning threshold (threshold = 5)
                       Modify image (modify = yes)
	  Print general diagnostics (verbose = no)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls strip_extract
A strip of constant width about the spectra positions is extracted.
If cleanning is desired a smoothed estimate of the profile is obtained
by averaging a number of lines about the line to be cleaned.  After fitting
for the intensity scale pixels are found which deviate from the profile by
a specified amount.
The profile value replaces the deviant pixel in the extraction and,
if specified, also in the image file.  No prior modeling is required
to use this extraction routine.

.nf
                Multi-Spectra image  image = 
             Strip extraction width  width = 1.
                   Lines to extract (lines = 1-1000)
                      Clean spectra (clean = yes)
                 Cleaning threshold (threshold = 5)
          Lines per profile average (averge_lines = 32)
                       Modify image (modify = yes)
	  Print general diagnostics (verbose = no)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls to_iids
For a specified prefix, files of the form prefix.nn, where nn is a specified
spectra number, are created containing the extracted spectra for all
the specified image files.  The format of the files is the IIDS format
developed for the CYBER Multi-Aperture Plate Extractions.

.nf
                Multi-Spectra image  images = 
               IIDS filename prefix  iids_file = 
                  Spectra to format (spectra = 1-1000)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls to_image
An image file containing one line of the extracted luminosities for each
specified spectra in the specified MULTISPEC image.

.nf
                Multi-Spectra image  in_image = 
            Extracted spectra image  out_image = 
                            Spectra (spectra = 1-1000)
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls to_onedspec
The extractions are converted to an as yet to be specified format for
use in the ONEDSPEC reduction package.

.nf
               Multi-Spectra images  images = 
                 ONEDSPEC data file  onedspec_file = 
                            Spectra (spectra = 1-1000)
                                    (mode = ql)
.fi
.le
.ke
.sh
4.2  General MULTISPEC CL Scripts

     The general MULTISPEC CL scripts perform a series of steps needed to
extract the spectra from a specified list of image files.  These steps have
been found to generally perform the desired extraction task fully.


.nf
	multiap_extract		echelle_extract
.fi
.ks
.ls multiap_extract
The specified multi-aperture plate images are extracted.
If no starting solution image, one which has previously been extracted,
is specified then the script performs an automatic search for the
specified number of spectra.
Otherwise the solution from the starting image is used as the initial
model.  The background is then determined.
This is followed by a series of fitting steps on swaths of data.
(For further details on the fitting steps see the Algorithms paper).
A MODEL_EXTRACT and cleaning follows.
Finally, the extraction is correlated with the specified aperture plate
using AP_PLATE.
If there was no starting image then this extraction becomes the
initial solution image.
Subsequent images are extracted starting from the initial solution image.

.nf
              Multi-Aperture images  images = 
             Initial solution image  initial = 
              Aperture plate number  plate = 
                  Number of spectra  nspectra = 
                                    (mode = ql)
.fi
.le
.ke
.ks
.ls echelle_extract
The specified echelle images are extracted.
If no starting solution image, one which has previously been extracted,
is specified then the script performs an automatic search for the
specified number of orders.
Otherwise the solution from the starting image is used as the initial
starting point.  The background is then determined.
Finally a STRIP_EXTRACT and cleaning is performed.
If there was no starting image then this extraction becomes the
initial solution image.
Subsequent images are extracted starting from the initial solution image.

.nf
                     Echelle images  images = 
             Initial solution image  initial = 
                   Number of orders  norders = 
		   Extraction width  width =
                                    (mode = ql)
.fi
.le
.sh
5.  Outline of a MULTISPEC Reduction

     The following outline is for the reduction of a cryogenic camera
multi-aperture plate.  All the programmer supplied default values are
used.

.nf
	(1)  rcamera mtb, "ap165.", "s", "3-9"
	(2)  debias "ap165.*"
	(3)  multispec_flat "ap165.[36]", "ap165.flat"
	(4)  flat_divide "ap165.*", "ap165.flat"
	(5)  multiap_extract "ap165.*", "", 165, 50
	(6)  to_onedspec "ap165.*", oned165
.fi

.ls (1)
The data is read from the observing tape(s) using RCAMERA.
The image files created are ap165.3, ap165.4, ..., ap165.9.  This is
easily accomplished by using the filename prefix "ap165." in the RCAMERA
program.  The raw images may be examined at this point on a display.
.le
.ls (2)
The images are debiased using DEBIAS with all the "ap165." files specified.
The debias program knows about the location of the bias strip for the
cryogenic camera.
.le
.ls (3)
A a flat field is created
using MULTISPEC_FLAT in which the desired quartz frames are specified
and a flat field image filename is defined.  The created flat field
image may be examined on an image display if desired.
.le
.ls (4)
All the debiased images are divided by the flat field using FLAT_DIVIDE.
.le
.ls (5)
The script MULTIAP_EXTRACT is run in which the aperture plate number,
the number of spectra, and the image files to be extracted are specified.
The number of spectra is found by examining an image on an image display
or by plotting a cut across the spectra using a general image profile
program.
.le
.ls (6)
Finally, the extracted spectra are formatted for the ONEDSPEC package
using TO_ONEDSPEC with the extracted images specified.
.le
.endhelp