Initial commit

1 files changed, 512 insertions, 0 deletions

diff --git a/noao/onedspec/doc/sys/Review.hlp b/noao/onedspec/doc/sys/Review.hlp
new file mode 100644
index 00000000..5139f630
--- /dev/null
+++ b/noao/onedspec/doc/sys/Review.hlp
@@ -0,0 +1,512 @@
+.help onedspec Sep84 "Spectral Reductions"
+.ce
+\fBOne Dimensional Spectral Reductions\fR
+.ce
+Analysis and Discussion
+.ce
+September 4, 1984
+.sp 3
+.nh
+Introduction
+
+    The \fBonedspec\fR package is a collection of programs for the reduction
+and analysis of one dimensional spectral data.  The more general problem of
+operations upon one dimensional images or vectors shall be dealt with elsewhere,
+primarily in the \fBimages\fR and \fBplot\fR packages.  The problems of getting
+data in and out of the system are handled by the \fBdataio\fR package, at least
+for the standard data formats such as FITS.
+
+The operators provided in \fBonedspec\fR shall be general purpose and, as far
+as possible, independent of the instrument which produced the data.  Instrument
+dependent reductions tailored for specific instruments will be implemented as
+subpackages of the \fBimred\fR (image reductions) package.  For example,
+the subpackages \fBiids\fR and \fBirs\fR will be provided in \fBimred\fR for
+reducing data from the KPNO instruments of the same name.  The \fBimred\fR
+packages shall call upon the basic operators in \fBonedspec\fR, \fBimages\fR,
+and other packages to reduce the data for a specific instrument.
+
+
+.ks
+.nf
+	iids(etc)
+		imred
+		imredtools
+		onedspec
+			plot
+			tv
+				dataio
+				images
+					dbms
+					lists
+					system
+					language
+
+.fi
+.ce
+Relationship of \fBOnedspec\fR to other IRAF Packages
+.ke
+
+
+The relationship of the \fBonedspec\fR packages to other related packages in
+the IRAF system is shown above.  A program (CL script) in a package at one
+level in the hierarchy may only call programs in packages at lower levels.
+The system will load packages as necessary if not already loaded by the
+user.  The user is expected to be familiar with the standard system packages.
+
+.nh
+Basic Functions Required for One-Dimensional Spectral Reductions
+
+    The following classes of functions have been identified (in the preliminary
+specifications document for \fBonedspec\fR) as necessary to perform basic one
+dimensional spectral reductions.  Only a fraction of the functionality
+required is specific to the reduction of spectral data and is therefore
+provided by the \fBonedspec\fR package itself.
+
+.ls Transport
+Provided by the \fBdataio\fR package, although we do not currently have a
+reader for REDUCER format data tapes.  Readers for all standard format
+tapes are either available or planned.
+.le
+.ls Mathematical
+Standard system functions provided by \fBimages\fR (arithmetic, forward and
+inverse FFT, filtering, etc.).
+.le
+.ls Reduction Operators
+The heart of \fBonedspec\fR.  Operators are required (at a minimum) for
+coincidence correction, dispersion determination and correction, flat
+fielding, sky subtraction, extinction correction, and flux calibration.
+Operators for flat fielding and sky subtraction are already available elsewhere
+in IRAF.  Basic continuum fitting and subtraction is possible with existing
+software but additional algorithms designed for spectral data are desirable.
+.le
+.ls Plotting
+Standard system functions provided by the \fBplot\fR package.
+.le
+.ls Utilities
+Standard system functions provided by the \fBdbms\fR package.
+.le
+.ls Artificial Spectra
+These functions belong in the \fBartdata\fR package, but it is expected that
+prototype operators will be built as part of the initial \fBonedspec\fR
+development.
+.le
+
+.nh
+Data Structures
+
+    Spectra will be stored as one or two dimensional IRAF images embedded in
+database format files.  A free format header is associated with each image.
+Spectra may be grouped together as lines of a two dimensional image provided
+all can share the same header, but more commonly each image will contain a
+single spectrum.  The second image dimension, if used, will contain vectors
+directly associated with the images, such as a signal to noise vector.
+If the image is two dimensional the spectrum must be the first image line.
+The database facilities will allow images to be grouped together in a single
+file if desired.
+
+While most or all \fBonedspec\fR operators will expect a one dimensional
+image as input, image sections may be used to operate on vector subsections
+of higher dimensioned images if desired.  The datatype of an image is
+arbitrary, but all pixel data will be single precision real within
+\fBonedspec\fR.  While the IRAF image format does not impose any restrictions on
+the size of an image or image line, not all spectral operators may be usable
+on very large images.  In general, pointwise and local operations may easily
+be performed on images of any size with modest memory requirements, and
+most of the \fBonedspec\fR operations appear to fall into this class.
+
+.nh 2
+The IRAF Database Faciltities
+
+    An understanding of the IRAF database facilities is necessary to visualize
+how data will be treated by operators in \fBonedspec\fR and other packages.
+The database facilities will be used not just for image storage but also for
+program intercommunication, program output, and the storage of large
+astronomical catalogs (e.g. the SAO catalog).  Access to both small and
+large databases will be quite efficient; achieving this requires little
+innovation since database technology is already highly developed.  We begin by
+defining some important terms.
+
+.ls
+.ls DBIO
+The database i/o package, used by compiled programs to access a database.
+.le
+.ls DBMS
+The database management package, a CL level package used by the user to
+inspect, analyze, and manipulate the contents of a database.
+.le
+.ls database
+A set of one or more "relations" or tables (DBIO is a conventional relational
+database).  A convenient way to think of an IRAF database is as a directory.
+The relations appear as distinct files in the directory.
+.le
+.ls relation
+A relation is a set of \fBrecords\fR.  Each record consists of a set of
+\fBfields\fR, each characterized by a name and a datatype.  All the records
+in a relation have the same set of fields.  Perhaps the easiest way to
+visualize a relation is as a \fBtable\fR.  The rows and columns of the table
+correspond to the records and fields of the relation.
+.le
+.ls field
+A field of a record is characterized by an alphanumeric name, datatype, and
+size.  Fields may be one dimensional arrays of variable size.  Fields may be
+added to a relation dynamically at run time.  When a new field is added to
+a relation it is added to all records in the relation, but the value of the
+field in a particular record is undefined (and consumes no storage) until
+explicitly written into.
+.le
+.ls key
+.br
+A function of the values of one or more fields, used to select a subset of
+rows from a table.  Technically, a valid key will permit selection of any
+single row from a table, but we often use the term is a less strict sense.
+.le
+.le
+
+
+An \fBimage\fR appears in the database as a record.  The record is really
+just the image header; the pixels are stored external to the database in a
+separate file, storing only the name of the pixel storage file in the record
+itself (for very small images we are considering storing the pixels directly
+in the database file).  Note that the record is a simple flat structure;
+this simple structure places restrictions on the complexity of objects which
+can be stored in the database.
+
+The records in a relation form a set, not an array.  Records are referred to
+by a user-defined key.  A simple key might be a single field containing a
+unique number (like an array index), or a unique name.  More complex keys
+might involve pattern matching over one or more fields, selection of records
+with fields within a certain range of values, and so on.
+
+From the viewpoint of \fBonedspec\fR, a relation can be considered a
+\fBdata group\fR, consisting of a set of \fBspectra\fR.
+
+.nh 2
+Image Templates
+
+    The user specifies the set of spectra to be operated upon by means of an
+image template.  Image templates are much like the filename templates commonly
+used in operating systems.  The most simple template is the filename of
+a single data group; this template matches all spectra in the group.  If there
+is only one spectrum in a file, then only one spectrum is operated upon.
+A slightly more complex template is a list of filenames of data groups.
+More complex templates will permit use of expressions referencing the values
+of specific fields to select a subset of the spectra in a group.  The syntax
+of such expressions has not yet been defined (examples are given below
+nonetheless), but the function performed by an image template will be the same
+regardless of the syntax.  In all cases the image template will be a single
+string valued parameter at the CL level.
+
+.nh 2
+Standard Calling Sequence
+
+    The standard calling sequence for a unary image operator is shown below
+The calling sequence for a binary operator would be the same with a second input
+parameter added as the second argument.  In general, any data dependent
+control parameters should be implemented as positional arguments following
+the primary operands, and data independent or optional (rarely used) parameters
+should be implemented as hidden parameters.
+
+
+.ks
+.nf
+	imop (input, output, data_dependent_control_params)
+
+	imop	image operator name
+	input	image template specifying set of input images
+	output	filename of output datagroup
+
+	data_dependent_control_parameters
+	(hidden parameters)
+
+for example,
+
+	coincor (spectra, newgroup, dead_time)
+.fi
+.ke
+
+
+If a series of spectra are to be processed it seems reasonable to add the
+processed spectra to a new or existing data group (possibly the same as an
+input datagroup).  If the operation is to be performed in place a special
+notation (e.g. the null string) can be given as the output filename.
+At the \fBonedspec\fR level output filenames will not be defaulted.
+
+.nh 2
+Examples
+
+    Some examples of image templates might be useful to give a more concrete
+idea of the functionality which will be available.  Bear in mind that what we
+are describing here is really the usage of one of the fundamental IRAF system
+interfaces, the DBMS database management subsystem, albeit from the point of
+view of \fBonedspec\fR.  The same facilities will be available in any program
+which operates upon images, and in some non-image applications as well (e.g.
+the new \fBfinder\fR).  Our philosopy, as always, is to make standard usage
+simple, with considerable sophistication available for those with time to
+learn more about the system.
+
+The simplest case occurs when there is one spectrum per data group (file).
+For example, assuming that the file "a" contains a single spectrum, the
+command
+
+	cl> coincor a, b, .2
+
+would perform coincidence correction for spectrum A, placing the result in
+B, using a dead time parameter of .2.  For a more complex example, consider
+the following command:
+
+	cl> coincor "a.type=obj&coincor=no,b", a, .2
+
+This would perform coincidence correction for all spectra in group B plus all
+object spectra in group A which have not already been coincidence corrected,
+adding the corrected spectra to group A (notation approximate only).  If the
+user does not trust the database explicit record numbers may be used and
+referenced via range list expressions, e.g.,
+
+	cl> coincor "a.recnum=(1,5,7:11),b", a, .2
+
+would select records 1, 5, and 7 through 11 from data group A.  Alternatively
+the database utilities could be used to list the spectra matching the selection
+criteria prior to the operation if desired.  For example,
+
+	cl> db.select "a.type=obj"
+
+would write a table on the standard output (the terminal) wherein each spectrum
+in data group A is shown as a row of field values.  If one wanted to generate
+an explicit list of records to be processed with help from the database
+utilities, a set of records could be selected from a data group and selected
+fields from each record written into a text file:
+
+	cl> db.select "a.type=obj", "recnum, history" > reclistfile
+
+The output file "reclistfile" produced by this command would contain the
+fields "recnum" (record number) and "history" (description of processing
+performed to generate the record).  The editor could be used to delete
+unwanted records, producing a list of record numbers suitable for use as
+an image template:
+
+	cl> coincor "a.recnum=@reclistfile", a, .2
+
+.nh
+Reduction Operators
+
+.nh 2
+Line List Preparation
+
+    I suggest maintaining the line lists as text files so that the user can
+edit them with the text editor, or process them with the \fBlists\fR operators.
+A master line list might be maintained in a database and the DBMS \fBselect\fR
+operator used to extract ASCII linelists in the wavelength region of interest,
+but this would only be necessary if the linelist is quite large or if a linelist
+record contains many fields.  I don't think we need the \fBline_list\fR task.
+
+.nh 2
+Dispersion Solution
+
+    The problem with selecting a line list and doing the dispersion solution
+in separate operations is that the dispersion solution is invaluable as an aid
+for identifying lines and for rejecting lines.  Having a routine which merely
+tweaks up the positions of lines in an existing lineset (e.g., \fBalinid\fR)
+is not all that useful.  I would like to suggest the following alternate
+procedure for performing the dispersion solution for a set of calibration
+spectra which have roughly the same dispersion.
+
+.ls
+.ls [1] Generate Lineset [and fit dispersion]
+.sp
+Interactively determine the lineset to be used, i.e., wavelength (or whatever)
+and approximate line position in pixel units for N lines.  Input is one or more
+comparison spectra and optionally a list of candidate lines in the region
+of interest.  Output is the order for the dispersion curve and a linelist of
+the following (basic) form:
+
+	L#  X  Wavelength [Weight]
+
+It would be very useful if the program, given a rough guess at the dispersion,
+could match the standard linelist with the spectra and attempt to automatically
+identify the lines thus detected.  The user would then interactively edit the
+resultant line set using plots of the fitted dispersion curve to reject
+misidentified or blended lines and to adjust weights until a final lineset
+is produced.
+.le
+
+.ls [2] Fit Dispersion
+.sp
+Given the order and functional type of the curve to be fitted and a lineset
+determined in step [1] (or a lineset produced some any other means, e.g. with
+the editor), for each spectrum in the input data group tweak the center of
+each line in the lineset via an automatic centering algorithm, fit the
+dispersion curve, and save the coefficients of the fitted curve in the
+image header.  The approximate line positions would be used to find and measure
+the positions of the actual lines, and the dispersion curve would be fitted and
+saved in the image header of each calibration spectrum.
+
+While this operator would be intended to be used noninteractively, the default
+textual and graphics output devices could be the terminal.  To use the program
+in batch mode the user would redirect both the standard output and the graphics
+output (if any), e.g.,
+
+.nf
+	cl> dispsol "night1.type=comp", linelistfile, order,
+	>>> device=stdplot, > dispsol.spool &
+.fi
+
+Line shifts, correlation functions, statistical errors, the computed residuals
+in the fitted dispersion curves, plots of various terms of the dispersion
+curves, etc. may be generated to provide a means for later checking for
+erroneous solutions to the individual spectra.  There is considerable room for
+innovation in this area.
+.le
+
+.ls [3] Second Order Correction
+.sp
+If it is desired to interpolate the dispersion curve in some additional
+dimension such as time or hour angle, fit the individual dispersion solutions
+produced by [1] or [2] as a group to one or more additional dimensions,
+generating a dispersion solution of one, two or more dimensions as output.
+If the output is another one dimensional dispersion solution, the input
+solutions are simply averaged with optional weights.  This "second order"
+correction to a group of dispersion solutions is probably best performed by
+a separate program, rather than building it into \fBalineid\fR, \fBdispsol\fR,
+etc.  This makes the other programs simpler and makes it possible to exclude
+spectra from the higher dimensional fit without repeating the dispersion
+solutions.
+.le
+.le
+
+If the batch run [2] fails for selected spectra the dispersion solution for
+those spectra can be repeated interactively with operator [1].
+The curve fitting package should be used to fit the dispersion curve (we can
+extend the package to support \fBonedspec\fR if necessary).
+
+.nh 2
+Dispersion Correction
+
+    This function of this procedure is to change the dispersion of a
+spectrum or group of spectra from one functional form to another.
+At a mimimum it must be possible to produce spectra linear in wavelength or
+log wavelength (as specified), but it might also be useful to be able
+to match the dispersion of a spectrum to that of a second spectrum, e.g., to
+minimize the amount of interpolation required to register spectra, or
+to introduce a nonlinear dispersion for testing purposes.  This might be
+implemented at the CL parameter level by having a string parameter which
+takes on the values "linear" (default), "log", or the name of a record
+defining the dispersion solution to be matched.
+
+It should be possible for the output spectrum to be a different size than
+the input spectrum, e.g., since we are already interpolating the data,
+it might be nice to produce an output spectrum of length 2**N if fourier
+analysis is to be performed subsequently.  It should be possible to
+extract only a portion of a spectrum (perform subraster extraction) in the
+process of correcting the dispersion, producing an output spectrum of a
+user-definable size.  It should be possible for an output pixel to lie at
+a point outside the bounds of the input spectrum, setting the value of the
+output pixel to INDEF or to an artificially generated value.  Note that
+this kind of generality can be implemented at the \fBonedspec\fR level
+without compromising the simplicity of dispersion correction for a particular
+instrument at the \fBimred\fR level.
+
+.nh 3
+Line Centering Algorithms
+
+    For most data, the best algorithm in the set described is probably the
+parabola algorithm.  To reject nearby lines and avoid degradation of the
+signal to noise the centering should be performed within a small aperture,
+but the aperture should be allowed to move several pixels in either direction
+to find the peak of the line.
+
+The parabola algorithm described has these features,
+but as described it finds the extrema within a window about the
+initial position.  It might be preferable to simply walk up the peak nearest
+to the initial center.  This has the advantage that it is possible to center
+on a line which has a nearby, stronger neighbor which cannot itself be used
+for some reason, but which might fall within \fBparextent\fR pixels of the
+starting center.  The parabola algorithm as described also finds a local extrema
+rather than a local maximum; probably not what is desired for a dispersion
+solution.  The restriction to 3 pixels in the final center determination is
+bad; the width of the centering function must be a variable to accommodate
+the wide range of samplings expected.
+
+The parabola algorithm described is basically a grid search over
+2*\fIparextent\fR pixels for the local extrema.  What I am suggesting is
+an iterative gradient search for the local maximum.  The properties of the
+two algorithms are probably sufficiently different to warrant implementation
+of both as an option (the running times are comparable).  I suspect that
+everyone else who has done this will have their own favorite algorithm as
+well; probably we should study half a dozen but implement only one or two.
+
+.nh 2
+Field Flattening
+
+    It is not clear that we need special flat fielding operators for
+\fBonedspec\fR.  We have a two-dimensional operator which fits image lines
+independently which might already do the job.  Probably we should experiment
+with both the smoothing spline and possibly fourier filtering for removing
+the difficult medium frequency fluctuations.  The current \fBimred\fR flat field
+operator implements the cubic smoothing spline (along with the Chebyshev and
+Legendre polynomials), and is available for experimentation.
+
+Building interactive graphics into the operator which fits a smooth curve to
+the continuum is probably not necessary.  If a noninteractive \fBimred\fR or
+\fBimages\fR operator is used to fit the continuum the interactive graphics
+can still be available, but might better reside in a higher level CL script.
+The basic operator should behave like a subroutine and not write any output
+to the terminal unless enabled by a hidden parameter (we have been calling
+this parameter \fIverbose\fR in other programs).
+
+.nh 3
+Extinction Correction and Flux Calibration
+
+    I did not have time to review any of this.
+
+.nh
+Standard Library Packages
+
+    The following standard IRAF math library packages should be used in
+\fBonedspec\fR.  The packages are very briefly described here but are
+fully documented under \fBhelp\fR on the online (kpnob:xcl) system.
+
+.nh 2
+Curve Fitting
+
+    The curve fitting package (\fBcurfit\fR) is currently capable of fitting
+the Chebyshev and Legendre polynomials and the cubic smoothing spline.
+Weighting is supported as an option.
+We need to add a piecewise linear function to support the
+dispersion curves for the high resolution FTS spectra.  We may have to add a
+double precision version of the package to provide the 8-10 digits of
+precision needed for typical comparison line wavelength values, but
+normalization of the wavelength values may make this unnecessary for moderate
+resolution spectra.
+
+Ordinary polynomials are not supported because their numerical properties are
+very much inferior to those of orthogonal polynomials (the ls matrix can have
+a disastrously high condition number, and lacking normalization the function
+begin fitted is not invariant with respect to scale changes and translations
+in the input data).  For low order fits the Chebyshev polynomials are
+considered to have the best properties from an approximation theoretic point
+of view, and for high order fits the smoothing spline is probably best because
+it can follow arbitrary trends in the data.
+
+.nh 2
+Interpolation
+
+    The image interpolation package (\fBiminterp\fR) currently supports the
+nearest neighbor, linear, third and fifth order divided differences,
+cubic interpolating spline, and sinc function interpolators.
+We should add the zeroth and first order partial pixel ("flux conserving")
+interpolants because they offer unique properties not provided by any
+of the other interpolants.
+
+.nh 2
+Interactive Graphics
+
+    We will define a standard interactive graphics utility package for
+interactive operations upon data vectors (to be available in a system library
+in object form).  It should be possible to define a general package which
+can be used anywhere a data vector is to be plotted and
+examined interactively (not just in \fBonedspec\fR).  Standard keystrokes
+should be defined for common operations such as expanding a region of
+the plot and restoring the original scale.  This will not be attempted
+until an interactive version of the GIO interface is available later this
+fall.
+.endhelp