Initial commit

1 files changed, 787 insertions, 0 deletions

diff --git a/noao/imred/ccdred/doc/ccdred.ms b/noao/imred/ccdred/doc/ccdred.ms
new file mode 100644
index 00000000..645514ec
--- /dev/null
+++ b/noao/imred/ccdred/doc/ccdred.ms
@@ -0,0 +1,787 @@
+.RP
+.TL
+The IRAF CCD Reduction Package -- CCDRED
+.AU
+Francisco Valdes
+.AI
+IRAF Group - Central Computer Services
+.K2
+P.O. Box 26732, Tucson, Arizona 85726
+September 1987
+.AB
+The IRAF\(dg CCD reduction package, \fBccdred\fR, provides tools
+for the easy and efficient reduction of CCD images.  The standard
+reduction operations are replacement of bad pixels, subtraction of an
+overscan or prescan bias, subtraction of a zero level image,
+subtraction of a dark count image, division by a flat field calibration
+image, division by an illumination correction, subtraction of a fringe
+image, and trimming unwanted lines or columns.  Another common
+operation provided by the package is scaling and combining images with
+a number of algorithms for rejecting cosmic rays.  Data in the image
+header is used to make the reductions largely automated and
+self-documenting though the package may still be used in the absence of
+this data.  Also a translation mechanism is used to relate image header
+parameters to those used by the package to allow data from a variety of
+observatories and instruments to be processed.  This paper describes
+the design goals for the package and the main tasks and algorithms
+which satisfy these goals.
+.PP
+This paper is to be published as part of the proceedings of the
+Santa Cruz Summer Workshop in Astronomy and Astrophysics,
+\fIInstrumentation for Ground-Based Optical Astronomy:  Present and
+Future\fR, edited by Lloyd B. Robinson and published by
+Springer-Verlag.
+.LP
+\(dgImage Reduction and Analysis Facility (IRAF), a software system
+distributed by the National Optical Astronomy Observatories (NOAO).
+.AE
+.NH
+Introduction
+.PP
+The IRAF CCD reduction package, \fBccdred\fR, provides tools
+for performing the standard instrumental corrections and calibrations
+to CCD images.  The major design goals were:
+.IP
+.nf
+\(bu To be easy to use
+\(bu To be largely automated
+\(bu To be image header driven if the data allows
+\(bu To be usable for a variety of instruments and observatories
+\(bu To be efficient and capable of processing large volumes of data
+.fi
+.LP
+This paper describes the important tasks and algorithms and shows how
+these design goals were met.  It is not intended to describe every
+task, parameter, and usage in detail; the package has full
+documentation on each task plus a user's guide.
+.PP
+The standard CCD correction and calibration operations performed are
+replacement of bad columns and lines by interpolation from neighboring
+columns and lines, subtraction of a bias level determined from overscan
+or prescan columns or lines, subtraction of a zero level using a zero
+length exposure calibration image, subtraction of a dark count
+calibration image appropriately scaled to the dark time exposure of the
+image, division by a scaled flat field calibration image, division by
+an illumination image (derived from a blank sky image), subtraction of
+a scaled fringe image (also derived from a blank sky image), and
+trimming the image of unwanted lines or columns such as the overscan
+strip.  The processing may change the pixel datatype on disk (IRAF allows
+seven image datatypes); usually from 16 bit integer to real format.
+Two special operations are also supported for scan mode and one
+dimensional zero level and flat field calibrations; i.e. the same
+calibration is applied to each CCD readout line.  Any set of operations
+may be done simultaneously over a list of images in a highly efficient
+manner.  The reduction operations are recorded in the image header and
+may also be logged on the terminal and in a log file.
+.PP
+The package also provides tools for combining multiple exposures
+of object and calibration images to improve the statistical accuracy of
+the observations and to remove transient bad pixels.  The combining
+operation scales images of different exposure times, adjusts for
+variable sky background, statistically weights the images by their
+signal-to-noise, and provides a number of useful algorithms for
+detecting and rejecting transient bad pixels.
+.PP
+Other tasks are provided for listing reduction information about
+the images, deriving secondary calibration images (such as sky
+corrected flat fields or illumination correction images), and easily
+setting the package parameters for different instruments.
+.PP
+This paper is organized as follows.  There is a section giving an
+overview of how the package is used to reduce CCD data.  This gives the
+user's perspective and illustrates the general ease of use.  The next
+section describes many of the features of the package contributing to
+its ease of use, automation, and generality.  The next two sections
+describe the major tools and algorithms in some detail.  This includes
+discussions about achieving high efficiency.  Finally the status of the
+package and its use at NOAO is given.  References to additional
+documentation about IRAF and the CCD reduction package and an appendix
+listing the individual tasks in the package are found at the end of
+this paper.
+.NH
+A User's Overview
+.PP
+This section provides an overview of reducing data with the IRAF CCD
+reduction package.  There are many variations in usage depending on the
+type of data, whether the image headers contain information about the
+data which may be used by the tasks, and the scientific goal.  Only a
+brief example is given.  A more complete discussion of usage and
+examples is given in \fIA User's Guide to the IRAF CCDRED Package\fR.
+The package was developed within the IRAF system and so makes use of
+all the sophisticated features provided.  These features are also
+summarized here for those not familiar with IRAF since they are an
+important part of using the package.
+.PP
+Since the IRAF system is widely distributed and runs on a wide variety
+of computers, the site of the CCD reductions might be at the telescope,
+a system at the observatory provided for this purpose, or at the
+user's home computer.  The CCD images to be processed are either
+available immediately as the data is taken, transferred from the data taking
+computer via a network link (the method adopted at NOAO), or transferred
+to the reduction computer via a medium such as magnetic tape in FITS
+format.  The flexibility in reduction sites and hardware is one of the
+virtues of the IRAF-based CCD reduction package.
+.PP
+IRAF tasks typically have a number of parameters which give the user
+control over most aspects of the program.  This is possible since the
+parameters are kept in parameter files so that the user need not enter
+a large number of parameters every time the task is run.  The user may
+change any of these parameters as desired in several ways, such as by
+explicit assignment and using an easy to learn and use,
+fill-in-the-value type of screen editor.  The parameter values are
+\fIlearned\fR so that once a user sets the values they are maintained
+until the user changes them again; even between login sessions.
+.PP
+The first step in using the CCD reduction package is to set the default
+processing parameters for the data to be reduced.  These parameters include
+a database file describing the image header keyword translations and
+default values, the processing operations desired (operations
+required vary with instrument and observer), the calibration image names,
+and certain special parameters for special types of observations such
+as scan mode.  A special script task (a command procedure) is available
+to automatically set the default values, given the instrument name, to standard
+values defined by the support staff.  Identifying the instrument in this
+way may be all the novice user need do though most people quickly learn
+to adjust parameters at will.
+.PP
+As an example suppose there is an instrument identified as \fLrca4m\fR
+for an RCA CCD at the NOAO 4 meter telescope.  The user gives the command
+
+.ft L
+    cl> setinstrument rca4m
+.ft R
+
+which sets the default parameters to values suggested by the support staff
+for this instrument.  The user may then change these suggested values if
+desired.  In this example the processing switches are set to perform
+overscan bias subtraction, zero level image subtraction, flat fielding,
+and trimming.
+.PP
+The NOAO image headers contain information identifying the type of
+image, such as object, zero level, and flat field, the filter used to
+match flat fields with object images, the location of the overscan bias
+data, the trim size for the data, and whether the image has been
+processed.  With this information the user need not worry about
+selecting images, pairing object images with calibration images, or
+inadvertently reprocessing an image.
+.PP
+The first step is to combine multiple zero level and flat field observations
+to reduce the effects of statistical noise.  This is done by the
+commands
+
+.nf
+.ft L
+	cl> zerocombine *.imh
+	cl> flatcombine *.imh
+.ft R
+.fi
+
+The "cl> " is the IRAF command language prompt.  The first command says
+look through all the images and combine the zero level images.  The
+second command says look through all the images and combine the flat
+field images by filter.  What could be simpler?  Some \fIhidden\fR (default)
+parameters the user may modify are the combined image name, whether to
+process the images first, and the type of combining algorithm to use.
+.PP
+The next step is to process the images using the combined calibration
+images.  The command is
+
+.ft L
+	cl> ccdproc *.imh
+.ft R
+
+This command says look through all the images, find the object images,
+find the overscan data based on the image header and subtract the
+bias, subtract the zero level calibration image, divide by the flat field
+calibration image, and trim the bias data and edge lines and columns.
+During this operation the task recognizes that the
+zero level and flat field calibration images have not been processed
+and automatically processes them when they are needed.  The log output
+of this task, which may be to the terminal, to a file, or both, shows
+how this works.
+
+.nf
+.ft L
+  ccd003: Jun  1 15:12 Trim data section is [3:510,3:510]
+  ccd003: Jun  1 15:12 Overscan section is [520:540,*], mean=485.0
+  Dark:   Jun  1 15:12 Trim data section is [3:510,3:510]
+  Dark:   Jun  1 15:13 Overscan section is [520:540,*], mean=484.6
+  ccd003: Jun  1 15:13 Dark count image is Dark.imh
+  FlatV:  Jun  1 15:13 Trim data section is [3:510,3:510]
+  FlatV:  Jun  1 15:14 Overscan section is [520:540,*], mean=486.4
+  ccd003: Jun  1 15:15 Flat field image is FlatV.imh, scale=138.2
+  ccd004: Jun  1 15:16 Trim data section is [3:510,3:510]
+  ccd004: Jun  1 15:16 Overscan section is [520:540,*], mean=485.2
+  ccd004: Jun  1 15:16 Dark count image is Dark.imh
+  ccd004: Jun  1 15:16 Flat field image is FlatV.imh, scale=138.2
+                \fI<... more ...>\fL
+  ccd013: Jun  1 15:22 Trim data section is [3:510,3:510]
+  ccd013: Jun  1 15:23 Overscan section is [520:540,*], mean=482.4
+  ccd013: Jun  1 15:23 Dark count image is Dark.imh
+  FlatB:  Jun  1 15:23 Trim data section is [3:510,3:510]
+  FlatB:  Jun  1 15:23 Overscan section is [520:540,*], mean=486.4
+  ccd013: Jun  1 15:24 Flat field image is FlatB.imh, scale=132.3
+                \fI<... more ...>\fL
+.ft R
+.fi
+
+.PP
+The log gives the name of the image and a time stamp for each entry.
+The first image is ccd003.  It is to be trimmed to the specified
+size given as an \fIimage section\fR, an array notation used commonly
+in IRAF to specify subsections of images.  The location of the
+overscan data is also given by an image section which, in this case,
+was found in the image header.  The mean bias level of the overscan
+is also logged though the overscan is actually a function of the
+readout line with the order of the function selected by the user.
+.PP
+When the task comes to subtracting the zero level image it first
+notes that the calibration image has not been processed and switches
+to processing the zero level image.  Since it knows it is a zero level
+image the task does not attempt to zero level or flat field correct
+this image.  After the zero level image has been processed the task
+returns to the object image only to find that the flat field image
+also has not been processed.  It determines that the object image was
+obtained with a V filter and selects the flat field image having the same
+filter.  The flat field image is processed through the zero level correction
+and then the task again returns to the object image, ccd003, which it
+finishes processing.
+.PP
+The next image, ccd004, is also a V filter
+observation.  Since the zero level and V filter flat field have been
+processed the object image is processed directly.  This continues
+for all the object images except for a detour to process the B filter flat
+field when the task first encounters a B filter object image.
+.PP
+In summary, the basic usage of the CCD reduction package is quite simple.
+First, the instrument is identified and some parameters for the data
+are set.  Calibration images are then combined if needed.  Finally,
+the processing is done with the simple command
+
+.ft L
+    cl> ccdproc *.imh&
+.ft R
+
+where the processing is performed as a \fIbackground job\fR in this example.
+This simplicity was a major goal of the package.
+.NH
+Features of the Package
+.PP
+This section describes some of the special features of the package
+which contribute to its ease of use, generality, and efficiency.
+The major criteria for ease of use are to minimize the user's record keeping
+involving input and output image names, the types of images, subset
+parameters such as filters which must be kept separate, and the state
+of processing of each image.  The goal is to allow input images to
+be specified using simple wildcards, such as "*.imh" to specify all
+images, with the knowledge that the task will only operate on images
+for which it makes sense.  To accomplish this the tasks must be able to
+determine the type of image, subset, and the state of processing from
+the image itself.  This is done by making use of image header parameters.
+.PP
+For generality the package does not require any image header information
+except the exposure time.  It is really not very much more difficult to
+reduce such data.  Mainly, the user must be more explicit about specifying
+images and setting task parameters or add the information to the image
+headers.  Some default header information may also be set in the image
+header translation file (discussed below).
+.PP
+One important image header parameter is the image type.  This
+discriminates between object images and various types of calibration
+images such as flat field, zero level, dark count, comparison arcs,
+illumination, and fringe images.  This information is used in two
+ways.  For most of the tasks the user may select that only one type of
+image be considered.  Thus, all the flat field images may be selected
+for combining or only the processing status of the object images be
+listed.  The second usage is to allow the processing tasks to identify
+the standard calibration images and apply only those operations which
+make sense.  For example, flat field images are not divided by a
+flat field.  This allows the user to set the processing operations
+desired for the object images without fear of misprocessing the
+calibration images.  The image type is also used to automatically
+select calibration images from a list of images to be processed instead
+of explicitly identifying them.
+.PP
+A related parameter specifies the subset.  For certain operations the
+images must have a common value for this parameter.  This parameter is
+often the filter but it may also apply to a grating or aperture, for example.
+The subset parameter is used to identify the appropriate flat field
+image to apply to an image or to select common flat fields to be combined
+into a higher quality flat field.  This is automatic and the user need not
+keep track of which image was taken with which filter or grating.
+.PP
+The other important image header parameters are the processing flags.
+These identify when an image has been processed and also act as a history
+of the operation including calibration images used and other parameter
+information.  The usage of these parameters is obvious; it allows the
+user to include processed images in a wildcard list knowing that the
+processing will not be repeated and to quickly determine the processing
+status of the image.
+.PP
+Use of image header parameters often ties the software to the a
+particular observatory.  To maintain generality and usefulness for data
+other than that at NOAO, the CCD reduction package was designed to
+provide a translation between parameters requested by the package and
+those actually found in the image header.  This translation is defined
+in a simple text file which maps one keyword to another and also gives
+a default value to be used if the image header does not include a
+value.  In addition the translation file maps the arbitrary strings
+which may identify image types to the standard types which the package
+recognizes.  This is a relatively simple scheme and does not allow for
+forming combinations or for interpreting values which are not simple
+such as embedding an exposure time as part of a string.  A more complex
+translation scheme may prove desirable as experience is gained with
+other types of image header formats, but by then a general header translation
+ability and/or new image database structure may be a standard IRAF
+feature.
+.PP
+This feature has proven useful at NOAO.  During the course of
+developing the package the data taking system was modernized by
+updating keywords and adding new information in the image headers,
+generally following the lines laid out by the \fBccdred\fR package.
+However, there is a period of transition and it is also desirable to
+reduce preexisting data.  There are several different formats for this
+data.  The header translation files make coping with these different
+formats relatively easy.
+.PP
+A fundamental aspect of the package is that the processing
+modifies the images.  In other words, the reduction operations are
+performed directly on the image.  This "feature" further simplifies
+record keeping, frees the user from having to form unique output image
+names, and minimizes the amount of disk space required.  There
+are two safety features in this process.  First, the modifications do
+not take effect until the operation is completed on the image.  This
+allows the user to abort the task without leaving the image data in a
+partially processed state and protects data if the computer
+crashes.  The second feature is that there is a parameter which may be
+set to make a backup of the input data with a particular prefix; for
+example "b", "orig", or "imdir$" (a logical directory prefix).  This
+backup feature may be used when there is sufficient disk space, when
+learning to use the package, or just to be cautious.
+.PP
+In a similar effort to efficiently manage disk space, when combining
+images into a master object or calibration image, there is an option to
+delete the input images upon completion of the combining operation.
+Generally this is desirable when there are many calibration exposures,
+such as zero level or flat field images, which are not used after they
+are combined into a final calibration image.
+.PP
+The goal of generality for many instruments at
+different observatories inherently conflicts with the goal of ease of
+use.  Generality requires many parameters and options.  This is
+feasible in the CCD reduction package, as well as the other IRAF packages,
+because of the IRAF parameter handling mechanism.  In \fBccdred\fR
+there still remains the problem of setting the parameters appropriately
+for a particular instrument, image header format, and observatory.
+.PP
+To make this convenient there is a task, \fBsetinstrument\fR, that,
+based on an instrument name, runs a setup script for the instrument.
+An example of this task was given in the previous section.
+The script may do any type of operation but mainly it sets default
+parameters.  The setup scripts are generally created by the support staff
+for the instrument.  The combination of the setup script and the
+instrument translation file make the package, in a sense, programmable
+and achieves the desired instrument/observatory generality with ease of use.
+.NH
+CCD Processing
+.PP
+This section describes in some detail how the CCD processing is performed.
+The task which does the basic CCD processing is call \fBccdproc\fR.
+From the point of view of usage the task is very simple but a great deal
+is required to achieve this simplicity.  The approach we take in describing
+the task is to follow the flow of control as the task runs with digressions
+as appropriate.
+.PP
+The highest level of control is a loop over the input images; all the
+operations are performed successively on each image.  It is common for
+IRAF tasks which operate on individual images to allow the operation to
+be repeated automatically over a list of input images.  This is important
+in the \fBccdred\fR package because data sets are often large and the
+processing is generally the same for each image.  It would be tedious
+to have to give the processing command for each image to be processed.
+If an error occurs while processing an image the error is
+printed as a warning and processing continues with the next image.
+This provides protection primarily against mistyped or nonexistent images.
+.PP
+Before the first image is processed the calibration images are
+identified.  There are two ways to specify calibration images;
+explicitly via task parameters or implicitly as part of the list of
+images to be processed.  Explicitly identifying calibration images
+takes precedence over calibration images in the input list.  Specifying
+calibration images as part of the input image list requires that the
+image types can be determined from the image header.  Using the input
+list provides a mechanism for breaking processing up into sets of
+images (possibly using files containing the image names for each set)
+each having their own calibration images.  One can, of course,
+selectively specify input and calibration images, but whenever possible
+one would like to avoid having to specify explicit images to process
+since this requires record keeping by the user.
+.PP
+The first step in processing an image is to check that it is of the
+appropriate image type.  The user may select to process images of only
+one type.  Generally this is object images since calibration images are
+automatically processed as needed.  Images which are not of the desired
+type are skipped and the next image is considered.
+.PP
+A temporary output image is created next.  The output pixel datatype on
+disk may be changed at this point as selected by the user.
+For example it is common for the raw CCD images to be digitized as 16
+bit integers but after calibration it is sometimes desirable to have
+real format pixels.  If no output pixel datatype is specified the
+output image takes the same pixel datatype as the input image.  The
+processing is done by operating on the input image and writing the
+results to a temporary output image.  When the processing is complete
+the output image replaces the input image.  This gives the effect of
+processing the images in place but with certain safeguards.  If the
+computer crashes or the processing is interrupted the integrity of the
+input image is maintained.  The reasons for chosing to process the
+images in this way are to avoid having to generate new image names (a
+tiresome record keeping process for the user), to minimize disk
+usage, and generally the unprocessed images are not used once they have
+been processed.  When dealing with large volumes of data these reasons
+become fairly important.  However, the user may specify a backup prefix
+for the images in which case, once the processing is completed, the
+original input image is renamed by appending it to the prefix (or with
+an added digit if a previous backup image of the same name exits)
+before the processed output image takes the original input name.
+.PP
+The next step is to determine the image geometry.  Only a subsection of
+the raw image may contain the CCD data.  If this region is specified by
+a header parameter then the processing will affect only this region.
+This allows calibration and other data to be part of the image.
+Normally, the only other data in a image is overscan or prescan data.
+The location of this bias data is determined from the image header or
+from a task parameter (which overrides the image header value).  To
+relate calibration images of different sizes and to allow for readout
+of only a portion of the CCD detector, a header parameter may relate
+the image data coordinates to the full CCD coordinates.  Application of
+calibration image data and identifying bad pixel regions via a bad
+pixel file is done in this CCD coordinate system.  The final
+geometrical information is the region of the input image to be output
+after processing; an operation called trimming.  This is defined by an
+image header parameter or a task parameter.  Trimming of the image is
+selected by the user.  Any or all of this geometry information may be
+absent from the image and appropriate defaults are used.
+.PP
+Each selected operation which is appropriate for the image type is then
+considered.  If the operation has been performed previously it will not
+be repeated.  If all selected operations have been performed then the
+temporary output image is deleted and the input image is left
+unchanged.  The next image is then processed.
+.PP
+For each selected operation to be performed the pertinent data is
+determined.  This consists of such things as the name of the
+calibration image, scaling factors, the overscan bias function, etc.
+Note that at this point only the parameters are determined, the
+operation is not yet performed.  This is because operations are not
+performed sequentially but simultaneously as described below.  Consider
+flat fielding as an example.  First the input image is checked to see
+if it has been flat fielded.  Then the flat field calibration image is
+determined.  The flat field image is checked to see if it has been
+processed.  If it has not been processed then it is processed by
+calling a procedure which is essentially a copy of the main processing
+program.  After the flat field image has been processed, parameters
+affecting the processing, such as the flat field scale factor
+(essentially the mean of the flat field image), are determined.  A log
+of the operation is then printed if desired.
+.PP
+Once all the processing operations and parameters have been defined the
+actual processing begins.  One of the key design goals was that the
+processing be efficient.  There are two primary methods used to achieve
+this goal; separate processing paths for 16 bit integer data and
+floating point data and simultaneous operations.  If the image, the
+calibration images, and the output image (as selected by the user) are
+16 bit integer pixel datatypes then the image data is read and written
+as integer data.  This eliminates internal datatype conversions both
+during I/O and during computations.  However, many operations include
+use of real factors such as the overscan bias, dark count exposure
+scaling, and flat field scaling which causes the computation to be done
+in real arithmetic before the result is stored again as an integer
+value.  In any case there is never any loss of precision except when
+converting the output pixel to short integer.  If any of the images are
+not integer then a real internal data path is used in which input and
+output image data are converted to real as necessary.
+.PP
+For each data path the processing proceeds line-by-line.  For each line
+in the output image data region (ignoring pixels outside the data area
+and pixels which are trimmed) the appropriate input data and
+calibration data are obtained.  The calibration data is determined from
+the CCD coordinates of the output image and are not necessarily from
+the same image line or columns.  The input data is copied to the output
+array while applying bad pixel corrections and trimming.  The line is
+then processed using a specially optimized procedure.  This procedure
+applies all operations simultaneously for all combinations of
+operations.  As an example, consider subtracting an overscan bias,
+subtracting a zero level, and dividing by a flat field.  The basic
+kernel of the task, where the bulk of the CPU time is used, is
+
+.nf
+.ft L
+  do i = 1, n
+      out[i] = (out[i] - overscan - zero[i]) * flatscale / flat[i]
+.ft R
+.fi
+
+Here, \fIn\fR is the number of pixels in the line, \fIoverscan\fR is
+the overscan bias value for the line, \fIzero\fR is the zero level data
+from the zero level image, \fIflatscale\fR is the mean of the flat
+field image, and \fIflat\fR is the flat field data from the flat
+field image.  Note the operations are not applied sequentially but
+in a single statement.  This is the most efficient method and there is
+no need for intermediate images.
+.PP
+Though the processing is logically performed line-by-line in the program,
+the image I/O from the disk is not done this way.  The IRAF virtual
+operating system image interface automatically provides multi-line
+buffering for maximal I/O efficiency.
+.PP
+In many image processing systems it has been standard to apply operations
+sequentially over an image.  This requires producing intermediate images.
+Since this is clearly inefficient in terms of I/O it has been the practice
+to copy the images into main memory and operate upon them there until
+the final image is ready to be saved.  This has led to the perception
+that in order to be efficient an image processing system \fImust\fR
+store images in memory.  This is not true and the IRAF CCD reduction
+package illustrates this.  The CCD processing does not use intermediate
+images and does not need to keep the entire image in main memory.
+Furthermore, though of lesser importance than I/O, the single statement method
+illustrated above is more efficient than multiple passes through the
+images even when the images are kept in main memory.  Finally, as CCD
+detectors increase in size and small, fast, and cheap processors become
+common it is a distinct advantage to not require the large amounts of
+memory needed to keep entire images in memory.
+.PP
+There is one area in which use of main memory can improve performance
+and \fBccdproc\fR does take advantage of it if desired.  The calibration
+images usually are the same for many input images.  By specifying the
+maximum amount of memory available for storing images in memory
+the calibration images may be stored in memory up to that amount.
+By parameterizing the memory requirement there is no builtin dependence
+on large memory!
+.PP
+After processing the input image the last steps are to log the operations
+in the image header using processing keywords and replace the input
+image by the output image as described earlier.  The CCD coordinates
+of the data are recorded in the header, even if not there previously, to
+allow further processing on the image after the image has been trimmed.
+.NH
+Combining Images
+.PP
+The second important tool in the CCD reduction package is a task to combine
+many images into a single, higher quality image.  While this may also be
+done with more general image processing tools (the IRAF task \fBimsum\fR
+for example) the \fBccdred\fR tasks include special CCD dependent features such
+as recognizing the image types and using the image header translation
+file.  Combining images is often done
+with calibration images, which are easy to obtain in number, where it
+is important to minimize the statistical noise so as to not affect the
+object images.  Sometimes object images also are combined.
+The task is called \fBcombine\fR and there are special versions of
+this task called \fBzerocombine, darkcombine\fR, and \fBflatcombine\fR
+for the standard calibration images.
+.PP
+The task takes a list of input images to be combined.  As output there
+is the combined image, an optional sigma image, and optional log output either
+to the terminal, to a log file, or both.  A subset or subsets
+of the input images may be selected based on the image type and a
+subset parameter such as the filter.  As with the processing task,
+this allows selecting images without having to explicitly list each
+image from a large data set.  When combining based on a subset parameter
+there is an output image, and possibly a sigma image, for each separate subset.
+The output image pixel datatype may also be changed during combining;
+usually from 16 bit integer input to real output.
+The sigma image is the standard deviation of the input images about the
+output image.
+.PP
+Except for summing the images together,
+combining images may require correcting for variations between the images
+due to differing exposure times, sky background, extinctions, and
+positions.  Currently, extinction corrections and registration are
+not included but scaling and shifting corrections are included.
+The scaling corrections may be done by exposure times or by computing
+the mode in each image.  Additive shifting is also done by computing
+the mode in the images.  The region of the image in which the mode
+is computed can be specified but by default the whole image is used.
+A scaling correction is used when the flux level or sensitivity is varying.
+The offset correction is used when the sky brightness is varying independently
+of the object brightness.  If the images are not scaled then special
+data paths combine the images more efficiently.
+.PP
+Except for medianing and summing, the images are combined by averaging.
+The average may be weighted by
+
+.nf
+.ft L
+	weight = (N * scale / mode) ** 2
+.ft R
+.fi
+
+where \fIN\fR is the number of images previously combined (the task
+records the number of images combined in the image header), \fIscale\fR
+is the relative scale (applied by dividing) from the exposure time or
+mode, and \fImode\fR is the background mode estimate used when adding a
+variable offset.
+.PP
+The combining operation is the heart of the task.  There are a number
+algorithms which may be used as well as applying statistical weights.
+The algorithms are used to detect and reject deviant pixels, such as
+cosmic rays.
+The choice of algorithm depends on the data, the number of images,
+and the importance of rejecting cosmic rays.  The more complex the
+algorithm the more time consuming the operation.
+The list below summarizes the algorithms.
+Further algorithms may be added in time.
+
+.IP "Sum - sum the input images"
+.br
+The input images are combined by summing.  Care must be taken
+not to exceed the range of the 16 bit integer datatype when summing if the
+output datatype is of this type.  Summing is the only algorithm in which
+scaling and weighting are not used.  Also no sigma image is produced.
+.IP "Average - average the input images"
+.br
+The input images are combined by averaging.  The images may be scaled
+and weighted.  There is no pixel rejection.  A sigma image is produced
+if more than one image is combined.
+.IP "Median - median the input images"
+.br
+The input images are combined by medianing each pixel.  Unless the images
+are at the same exposure level they should be scaled.  The sigma image
+is based on all the input images and is only an approximation to the
+uncertainty in the median estimates.
+.IP "Minreject, maxreject, minmaxreject - reject extreme pixels"
+.br
+At each pixel the minimum, maximum, or both are excluded from the
+average.  The images should be scaled and the average may be
+weighted.  The sigma image requires at least two pixels after rejection
+of the extreme values.  These are relatively fast algorithms and are
+a good choice if there are many images (>15).
+.IP "Threshold - reject pixels above and below specified thresholds"
+.br
+The input images are combined with pixels above and below specified
+threshold values (before scaling) excluded.  The images may be scaled
+and the average weighted.  The sigma image also has the rejected
+pixels excluded.
+.IP "Sigclip - apply a sigma clipping algorithm to each pixel"
+.br
+The input images are combined by applying a sigma clipping algorithm
+at each pixel.  The images should be scaled.  This only rejects highly
+deviant points and so
+includes more of the data than the median or minimum and maximum 
+algorithms.  It requires many images (>10-15) to work effectively.
+Otherwise the bad pixels bias the sigma significantly.  The mean
+used to determine the sigmas is based on the "minmaxrej" algorithm
+to eliminate the effects of bad pixels on the mean.  Only one
+iteration is performed and at most one pixel is rejected at each
+point in the output image.  After the deviant pixels are rejected the final
+mean is computed from all the data.  The sigma image excludes the
+rejected pixels.
+.IP "Avsigclip - apply a sigma clipping algorithm to each pixel"
+.br
+The input images are combined with a variant of the sigma clipping
+algorithm which works well with only a few images.  The images should
+be scaled.  For each line the mean is first estimated using the
+"minmaxrej" algorithm.  The sigmas at each point in the line are scaled
+by the square root of the mean, that is a Poisson scaling of the noise
+is assumed.  These sigmas are averaged to get a line estimate of the
+sigma.  Then the sigma at each point in the line is estimated by
+multiplying the line sigma by the square root of the mean at that point.  As
+with the sigma clipping algorithm only one iteration is performed and
+at most one pixel is rejected at each point.  After the deviant pixels
+are rejected the file mean is computed from all the data.  The sigma
+image excludes the rejected pixels.
+.RE
+.PP
+The "avsigclip" algorithm is the best algorithm for rejecting cosmic
+rays, especially with a small number of images, but it is also the most
+time consuming.  With many images (>10-15) it might be advisable to use
+one of the other algorithms ("maxreject", "median", "minmaxrej") because
+of their greater speed.
+.PP
+This task also has several design features to make it efficient and
+versatile.  There are separate data paths for integer data and real
+data; as with processing, if all input images and the output image are
+of the same datatype then the I/O is done with no internal conversions.
+With mixed datatypes the operations are done as real.  Even in the
+integer path the operations requiring real arithmetic to preserve the
+accuracy of the calculation are performed in that mode.  There is
+effectively no limit to the number of images which may be combined.
+Also, the task determines the amount of memory available and buffers
+the I/O as much as possible.  This is a case where operating on images
+from disk rather than in memory is essential.
+.NH
+Status and Conclusion
+.PP
+The initial implementation of the IRAF \fBccdred\fR package was
+completed in June 1987.  It has been in use at the National Optical
+Astronomy Observatories since April 1987.  The package was not
+distributed with Version 2.5 of IRAF (released in August 1987) but is
+available as a separate installation upon request.  It will be part of
+future releases of IRAF.
+.PP
+At NOAO the CCD reduction package is available at the telescopes as the
+data is obtained.  This is accomplished by transferring the images from
+the data taking computer to a Sun workstation (Sun Microsystems, Inc.)
+initially via tape and later by a direct link.  There are several
+reasons for adopting this architecture.  First, the data acquisition
+system is well established and is dedicated to its real-time function.
+The second computer was phased in without disrupting the essential
+operation of the telescopes and if it fails data taking may continue
+with data being stored on tape.  The role of the second computer is to
+provide faster and more powerful reduction and analysis capability not
+required in a data acquisition system.  In the future it can be more
+easily updated to  follow the state of the art in small computers.  As
+CCD detectors get larger the higher processing speeds will be essential
+to keep up with the data flow.
+.PP
+By writing the reduction software in the high level, portable, IRAF
+system the users have the capability to process their data from the
+basic CCD reductions to a full analysis at the telescope.  Furthermore,
+the same software is widely available on a variety of computers if
+later processing or reprocessing is desired; staff and visitors at NOAO
+may also reduce their data at the headquarters facilities.  The use of
+a high level system was also essential in achieving the design goals;
+it would be difficult to duplicate this complex package without
+the rich programming environment provided by the IRAF system.
+.NH
+References
+.PP
+The following documentation is distributed by the National Optical
+Astronomy Observatories, Central Computer Services, P.O. Box 26732,
+Tucson, Arizona, 85726.  A comprehensive description of the IRAF system
+is given in \fIThe IRAF Data Reduction and Analysis System\fR by Doug
+Tody (also appearing in \fIProceedings of the SPIE - Instrumentation in
+Astronomy VI\fR, Vol. 627, 1986).  A general guide to using IRAF is \fIA
+User's Introduction to the IRAF Command Language\fR by Peter Shames
+and Doug Tody.  Both these documents are also part of the IRAF
+documentation distributed with the system.
+.PP
+A somewhat more tutorial description of the \fBccdred\fR package is
+\fIA User's Guide to the IRAF CCDRED Package\fR by the author.
+Detailed task descriptions and supplementary documentation are
+given in the on-line help library and are part of the user's guide.
+.NH
+Appendix
+.PP
+The current set of tasks making up the IRAF CCD Reduction Package,
+\fBccdred\fR, are summarized below.
+
+.nf
+.ft L
+  badpiximage - Create a bad pixel mask image from a bad pixel file
+    ccdgroups - Group CCD images into image lists
+     ccdhedit - CCD image header editor
+      ccdlist - List CCD processing information
+      ccdproc - Process CCD images
+      combine - Combine CCD images
+  darkcombine - Combine and process dark count images
+  flatcombine - Combine and process flat field images
+  mkfringecor - Make fringe correction images from sky images
+   mkillumcor - Make flat field illumination correction images
+  mkillumflat - Make illumination corrected flat fields
+     mkskycor - Make sky illumination correction images
+    mkskyflat - Make sky corrected flat field images
+setinstrument - Set instrument parameters
+  zerocombine - Combine and process zero level images
+.fi
+.ft R