.RP .TL The IRAF CCD Reduction Package -- CCDRED .AU Francisco Valdes .AI IRAF Group - Central Computer Services .K2 P.O. Box 26732, Tucson, Arizona 85726 September 1987 .AB The IRAF\(dg CCD reduction package, \fBccdred\fR, provides tools for the easy and efficient reduction of CCD images. The standard reduction operations are replacement of bad pixels, subtraction of an overscan or prescan bias, subtraction of a zero level image, subtraction of a dark count image, division by a flat field calibration image, division by an illumination correction, subtraction of a fringe image, and trimming unwanted lines or columns. Another common operation provided by the package is scaling and combining images with a number of algorithms for rejecting cosmic rays. Data in the image header is used to make the reductions largely automated and self-documenting though the package may still be used in the absence of this data. Also a translation mechanism is used to relate image header parameters to those used by the package to allow data from a variety of observatories and instruments to be processed. This paper describes the design goals for the package and the main tasks and algorithms which satisfy these goals. .PP This paper is to be published as part of the proceedings of the Santa Cruz Summer Workshop in Astronomy and Astrophysics, \fIInstrumentation for Ground-Based Optical Astronomy: Present and Future\fR, edited by Lloyd B. Robinson and published by Springer-Verlag. .LP \(dgImage Reduction and Analysis Facility (IRAF), a software system distributed by the National Optical Astronomy Observatories (NOAO). .AE .NH Introduction .PP The IRAF CCD reduction package, \fBccdred\fR, provides tools for performing the standard instrumental corrections and calibrations to CCD images. The major design goals were: .IP .nf \(bu To be easy to use \(bu To be largely automated \(bu To be image header driven if the data allows \(bu To be usable for a variety of instruments and observatories \(bu To be efficient and capable of processing large volumes of data .fi .LP This paper describes the important tasks and algorithms and shows how these design goals were met. It is not intended to describe every task, parameter, and usage in detail; the package has full documentation on each task plus a user's guide. .PP The standard CCD correction and calibration operations performed are replacement of bad columns and lines by interpolation from neighboring columns and lines, subtraction of a bias level determined from overscan or prescan columns or lines, subtraction of a zero level using a zero length exposure calibration image, subtraction of a dark count calibration image appropriately scaled to the dark time exposure of the image, division by a scaled flat field calibration image, division by an illumination image (derived from a blank sky image), subtraction of a scaled fringe image (also derived from a blank sky image), and trimming the image of unwanted lines or columns such as the overscan strip. The processing may change the pixel datatype on disk (IRAF allows seven image datatypes); usually from 16 bit integer to real format. Two special operations are also supported for scan mode and one dimensional zero level and flat field calibrations; i.e. the same calibration is applied to each CCD readout line. Any set of operations may be done simultaneously over a list of images in a highly efficient manner. The reduction operations are recorded in the image header and may also be logged on the terminal and in a log file. .PP The package also provides tools for combining multiple exposures of object and calibration images to improve the statistical accuracy of the observations and to remove transient bad pixels. The combining operation scales images of different exposure times, adjusts for variable sky background, statistically weights the images by their signal-to-noise, and provides a number of useful algorithms for detecting and rejecting transient bad pixels. .PP Other tasks are provided for listing reduction information about the images, deriving secondary calibration images (such as sky corrected flat fields or illumination correction images), and easily setting the package parameters for different instruments. .PP This paper is organized as follows. There is a section giving an overview of how the package is used to reduce CCD data. This gives the user's perspective and illustrates the general ease of use. The next section describes many of the features of the package contributing to its ease of use, automation, and generality. The next two sections describe the major tools and algorithms in some detail. This includes discussions about achieving high efficiency. Finally the status of the package and its use at NOAO is given. References to additional documentation about IRAF and the CCD reduction package and an appendix listing the individual tasks in the package are found at the end of this paper. .NH A User's Overview .PP This section provides an overview of reducing data with the IRAF CCD reduction package. There are many variations in usage depending on the type of data, whether the image headers contain information about the data which may be used by the tasks, and the scientific goal. Only a brief example is given. A more complete discussion of usage and examples is given in \fIA User's Guide to the IRAF CCDRED Package\fR. The package was developed within the IRAF system and so makes use of all the sophisticated features provided. These features are also summarized here for those not familiar with IRAF since they are an important part of using the package. .PP Since the IRAF system is widely distributed and runs on a wide variety of computers, the site of the CCD reductions might be at the telescope, a system at the observatory provided for this purpose, or at the user's home computer. The CCD images to be processed are either available immediately as the data is taken, transferred from the data taking computer via a network link (the method adopted at NOAO), or transferred to the reduction computer via a medium such as magnetic tape in FITS format. The flexibility in reduction sites and hardware is one of the virtues of the IRAF-based CCD reduction package. .PP IRAF tasks typically have a number of parameters which give the user control over most aspects of the program. This is possible since the parameters are kept in parameter files so that the user need not enter a large number of parameters every time the task is run. The user may change any of these parameters as desired in several ways, such as by explicit assignment and using an easy to learn and use, fill-in-the-value type of screen editor. The parameter values are \fIlearned\fR so that once a user sets the values they are maintained until the user changes them again; even between login sessions. .PP The first step in using the CCD reduction package is to set the default processing parameters for the data to be reduced. These parameters include a database file describing the image header keyword translations and default values, the processing operations desired (operations required vary with instrument and observer), the calibration image names, and certain special parameters for special types of observations such as scan mode. A special script task (a command procedure) is available to automatically set the default values, given the instrument name, to standard values defined by the support staff. Identifying the instrument in this way may be all the novice user need do though most people quickly learn to adjust parameters at will. .PP As an example suppose there is an instrument identified as \fLrca4m\fR for an RCA CCD at the NOAO 4 meter telescope. The user gives the command .ft L cl> setinstrument rca4m .ft R which sets the default parameters to values suggested by the support staff for this instrument. The user may then change these suggested values if desired. In this example the processing switches are set to perform overscan bias subtraction, zero level image subtraction, flat fielding, and trimming. .PP The NOAO image headers contain information identifying the type of image, such as object, zero level, and flat field, the filter used to match flat fields with object images, the location of the overscan bias data, the trim size for the data, and whether the image has been processed. With this information the user need not worry about selecting images, pairing object images with calibration images, or inadvertently reprocessing an image. .PP The first step is to combine multiple zero level and flat field observations to reduce the effects of statistical noise. This is done by the commands .nf .ft L cl> zerocombine *.imh cl> flatcombine *.imh .ft R .fi The "cl> " is the IRAF command language prompt. The first command says look through all the images and combine the zero level images. The second command says look through all the images and combine the flat field images by filter. What could be simpler? Some \fIhidden\fR (default) parameters the user may modify are the combined image name, whether to process the images first, and the type of combining algorithm to use. .PP The next step is to process the images using the combined calibration images. The command is .ft L cl> ccdproc *.imh .ft R This command says look through all the images, find the object images, find the overscan data based on the image header and subtract the bias, subtract the zero level calibration image, divide by the flat field calibration image, and trim the bias data and edge lines and columns. During this operation the task recognizes that the zero level and flat field calibration images have not been processed and automatically processes them when they are needed. The log output of this task, which may be to the terminal, to a file, or both, shows how this works. .nf .ft L ccd003: Jun 1 15:12 Trim data section is [3:510,3:510] ccd003: Jun 1 15:12 Overscan section is [520:540,*], mean=485.0 Dark: Jun 1 15:12 Trim data section is [3:510,3:510] Dark: Jun 1 15:13 Overscan section is [520:540,*], mean=484.6 ccd003: Jun 1 15:13 Dark count image is Dark.imh FlatV: Jun 1 15:13 Trim data section is [3:510,3:510] FlatV: Jun 1 15:14 Overscan section is [520:540,*], mean=486.4 ccd003: Jun 1 15:15 Flat field image is FlatV.imh, scale=138.2 ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] ccd004: Jun 1 15:16 Overscan section is [520:540,*], mean=485.2 ccd004: Jun 1 15:16 Dark count image is Dark.imh ccd004: Jun 1 15:16 Flat field image is FlatV.imh, scale=138.2 \fI<... more ...>\fL ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] ccd013: Jun 1 15:23 Overscan section is [520:540,*], mean=482.4 ccd013: Jun 1 15:23 Dark count image is Dark.imh FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] FlatB: Jun 1 15:23 Overscan section is [520:540,*], mean=486.4 ccd013: Jun 1 15:24 Flat field image is FlatB.imh, scale=132.3 \fI<... more ...>\fL .ft R .fi .PP The log gives the name of the image and a time stamp for each entry. The first image is ccd003. It is to be trimmed to the specified size given as an \fIimage section\fR, an array notation used commonly in IRAF to specify subsections of images. The location of the overscan data is also given by an image section which, in this case, was found in the image header. The mean bias level of the overscan is also logged though the overscan is actually a function of the readout line with the order of the function selected by the user. .PP When the task comes to subtracting the zero level image it first notes that the calibration image has not been processed and switches to processing the zero level image. Since it knows it is a zero level image the task does not attempt to zero level or flat field correct this image. After the zero level image has been processed the task returns to the object image only to find that the flat field image also has not been processed. It determines that the object image was obtained with a V filter and selects the flat field image having the same filter. The flat field image is processed through the zero level correction and then the task again returns to the object image, ccd003, which it finishes processing. .PP The next image, ccd004, is also a V filter observation. Since the zero level and V filter flat field have been processed the object image is processed directly. This continues for all the object images except for a detour to process the B filter flat field when the task first encounters a B filter object image. .PP In summary, the basic usage of the CCD reduction package is quite simple. First, the instrument is identified and some parameters for the data are set. Calibration images are then combined if needed. Finally, the processing is done with the simple command .ft L cl> ccdproc *.imh& .ft R where the processing is performed as a \fIbackground job\fR in this example. This simplicity was a major goal of the package. .NH Features of the Package .PP This section describes some of the special features of the package which contribute to its ease of use, generality, and efficiency. The major criteria for ease of use are to minimize the user's record keeping involving input and output image names, the types of images, subset parameters such as filters which must be kept separate, and the state of processing of each image. The goal is to allow input images to be specified using simple wildcards, such as "*.imh" to specify all images, with the knowledge that the task will only operate on images for which it makes sense. To accomplish this the tasks must be able to determine the type of image, subset, and the state of processing from the image itself. This is done by making use of image header parameters. .PP For generality the package does not require any image header information except the exposure time. It is really not very much more difficult to reduce such data. Mainly, the user must be more explicit about specifying images and setting task parameters or add the information to the image headers. Some default header information may also be set in the image header translation file (discussed below). .PP One important image header parameter is the image type. This discriminates between object images and various types of calibration images such as flat field, zero level, dark count, comparison arcs, illumination, and fringe images. This information is used in two ways. For most of the tasks the user may select that only one type of image be considered. Thus, all the flat field images may be selected for combining or only the processing status of the object images be listed. The second usage is to allow the processing tasks to identify the standard calibration images and apply only those operations which make sense. For example, flat field images are not divided by a flat field. This allows the user to set the processing operations desired for the object images without fear of misprocessing the calibration images. The image type is also used to automatically select calibration images from a list of images to be processed instead of explicitly identifying them. .PP A related parameter specifies the subset. For certain operations the images must have a common value for this parameter. This parameter is often the filter but it may also apply to a grating or aperture, for example. The subset parameter is used to identify the appropriate flat field image to apply to an image or to select common flat fields to be combined into a higher quality flat field. This is automatic and the user need not keep track of which image was taken with which filter or grating. .PP The other important image header parameters are the processing flags. These identify when an image has been processed and also act as a history of the operation including calibration images used and other parameter information. The usage of these parameters is obvious; it allows the user to include processed images in a wildcard list knowing that the processing will not be repeated and to quickly determine the processing status of the image. .PP Use of image header parameters often ties the software to the a particular observatory. To maintain generality and usefulness for data other than that at NOAO, the CCD reduction package was designed to provide a translation between parameters requested by the package and those actually found in the image header. This translation is defined in a simple text file which maps one keyword to another and also gives a default value to be used if the image header does not include a value. In addition the translation file maps the arbitrary strings which may identify image types to the standard types which the package recognizes. This is a relatively simple scheme and does not allow for forming combinations or for interpreting values which are not simple such as embedding an exposure time as part of a string. A more complex translation scheme may prove desirable as experience is gained with other types of image header formats, but by then a general header translation ability and/or new image database structure may be a standard IRAF feature. .PP This feature has proven useful at NOAO. During the course of developing the package the data taking system was modernized by updating keywords and adding new information in the image headers, generally following the lines laid out by the \fBccdred\fR package. However, there is a period of transition and it is also desirable to reduce preexisting data. There are several different formats for this data. The header translation files make coping with these different formats relatively easy. .PP A fundamental aspect of the package is that the processing modifies the images. In other words, the reduction operations are performed directly on the image. This "feature" further simplifies record keeping, frees the user from having to form unique output image names, and minimizes the amount of disk space required. There are two safety features in this process. First, the modifications do not take effect until the operation is completed on the image. This allows the user to abort the task without leaving the image data in a partially processed state and protects data if the computer crashes. The second feature is that there is a parameter which may be set to make a backup of the input data with a particular prefix; for example "b", "orig", or "imdir$" (a logical directory prefix). This backup feature may be used when there is sufficient disk space, when learning to use the package, or just to be cautious. .PP In a similar effort to efficiently manage disk space, when combining images into a master object or calibration image, there is an option to delete the input images upon completion of the combining operation. Generally this is desirable when there are many calibration exposures, such as zero level or flat field images, which are not used after they are combined into a final calibration image. .PP The goal of generality for many instruments at different observatories inherently conflicts with the goal of ease of use. Generality requires many parameters and options. This is feasible in the CCD reduction package, as well as the other IRAF packages, because of the IRAF parameter handling mechanism. In \fBccdred\fR there still remains the problem of setting the parameters appropriately for a particular instrument, image header format, and observatory. .PP To make this convenient there is a task, \fBsetinstrument\fR, that, based on an instrument name, runs a setup script for the instrument. An example of this task was given in the previous section. The script may do any type of operation but mainly it sets default parameters. The setup scripts are generally created by the support staff for the instrument. The combination of the setup script and the instrument translation file make the package, in a sense, programmable and achieves the desired instrument/observatory generality with ease of use. .NH CCD Processing .PP This section describes in some detail how the CCD processing is performed. The task which does the basic CCD processing is call \fBccdproc\fR. From the point of view of usage the task is very simple but a great deal is required to achieve this simplicity. The approach we take in describing the task is to follow the flow of control as the task runs with digressions as appropriate. .PP The highest level of control is a loop over the input images; all the operations are performed successively on each image. It is common for IRAF tasks which operate on individual images to allow the operation to be repeated automatically over a list of input images. This is important in the \fBccdred\fR package because data sets are often large and the processing is generally the same for each image. It would be tedious to have to give the processing command for each image to be processed. If an error occurs while processing an image the error is printed as a warning and processing continues with the next image. This provides protection primarily against mistyped or nonexistent images. .PP Before the first image is processed the calibration images are identified. There are two ways to specify calibration images; explicitly via task parameters or implicitly as part of the list of images to be processed. Explicitly identifying calibration images takes precedence over calibration images in the input list. Specifying calibration images as part of the input image list requires that the image types can be determined from the image header. Using the input list provides a mechanism for breaking processing up into sets of images (possibly using files containing the image names for each set) each having their own calibration images. One can, of course, selectively specify input and calibration images, but whenever possible one would like to avoid having to specify explicit images to process since this requires record keeping by the user. .PP The first step in processing an image is to check that it is of the appropriate image type. The user may select to process images of only one type. Generally this is object images since calibration images are automatically processed as needed. Images which are not of the desired type are skipped and the next image is considered. .PP A temporary output image is created next. The output pixel datatype on disk may be changed at this point as selected by the user. For example it is common for the raw CCD images to be digitized as 16 bit integers but after calibration it is sometimes desirable to have real format pixels. If no output pixel datatype is specified the output image takes the same pixel datatype as the input image. The processing is done by operating on the input image and writing the results to a temporary output image. When the processing is complete the output image replaces the input image. This gives the effect of processing the images in place but with certain safeguards. If the computer crashes or the processing is interrupted the integrity of the input image is maintained. The reasons for chosing to process the images in this way are to avoid having to generate new image names (a tiresome record keeping process for the user), to minimize disk usage, and generally the unprocessed images are not used once they have been processed. When dealing with large volumes of data these reasons become fairly important. However, the user may specify a backup prefix for the images in which case, once the processing is completed, the original input image is renamed by appending it to the prefix (or with an added digit if a previous backup image of the same name exits) before the processed output image takes the original input name. .PP The next step is to determine the image geometry. Only a subsection of the raw image may contain the CCD data. If this region is specified by a header parameter then the processing will affect only this region. This allows calibration and other data to be part of the image. Normally, the only other data in a image is overscan or prescan data. The location of this bias data is determined from the image header or from a task parameter (which overrides the image header value). To relate calibration images of different sizes and to allow for readout of only a portion of the CCD detector, a header parameter may relate the image data coordinates to the full CCD coordinates. Application of calibration image data and identifying bad pixel regions via a bad pixel file is done in this CCD coordinate system. The final geometrical information is the region of the input image to be output after processing; an operation called trimming. This is defined by an image header parameter or a task parameter. Trimming of the image is selected by the user. Any or all of this geometry information may be absent from the image and appropriate defaults are used. .PP Each selected operation which is appropriate for the image type is then considered. If the operation has been performed previously it will not be repeated. If all selected operations have been performed then the temporary output image is deleted and the input image is left unchanged. The next image is then processed. .PP For each selected operation to be performed the pertinent data is determined. This consists of such things as the name of the calibration image, scaling factors, the overscan bias function, etc. Note that at this point only the parameters are determined, the operation is not yet performed. This is because operations are not performed sequentially but simultaneously as described below. Consider flat fielding as an example. First the input image is checked to see if it has been flat fielded. Then the flat field calibration image is determined. The flat field image is checked to see if it has been processed. If it has not been processed then it is processed by calling a procedure which is essentially a copy of the main processing program. After the flat field image has been processed, parameters affecting the processing, such as the flat field scale factor (essentially the mean of the flat field image), are determined. A log of the operation is then printed if desired. .PP Once all the processing operations and parameters have been defined the actual processing begins. One of the key design goals was that the processing be efficient. There are two primary methods used to achieve this goal; separate processing paths for 16 bit integer data and floating point data and simultaneous operations. If the image, the calibration images, and the output image (as selected by the user) are 16 bit integer pixel datatypes then the image data is read and written as integer data. This eliminates internal datatype conversions both during I/O and during computations. However, many operations include use of real factors such as the overscan bias, dark count exposure scaling, and flat field scaling which causes the computation to be done in real arithmetic before the result is stored again as an integer value. In any case there is never any loss of precision except when converting the output pixel to short integer. If any of the images are not integer then a real internal data path is used in which input and output image data are converted to real as necessary. .PP For each data path the processing proceeds line-by-line. For each line in the output image data region (ignoring pixels outside the data area and pixels which are trimmed) the appropriate input data and calibration data are obtained. The calibration data is determined from the CCD coordinates of the output image and are not necessarily from the same image line or columns. The input data is copied to the output array while applying bad pixel corrections and trimming. The line is then processed using a specially optimized procedure. This procedure applies all operations simultaneously for all combinations of operations. As an example, consider subtracting an overscan bias, subtracting a zero level, and dividing by a flat field. The basic kernel of the task, where the bulk of the CPU time is used, is .nf .ft L do i = 1, n out[i] = (out[i] - overscan - zero[i]) * flatscale / flat[i] .ft R .fi Here, \fIn\fR is the number of pixels in the line, \fIoverscan\fR is the overscan bias value for the line, \fIzero\fR is the zero level data from the zero level image, \fIflatscale\fR is the mean of the flat field image, and \fIflat\fR is the flat field data from the flat field image. Note the operations are not applied sequentially but in a single statement. This is the most efficient method and there is no need for intermediate images. .PP Though the processing is logically performed line-by-line in the program, the image I/O from the disk is not done this way. The IRAF virtual operating system image interface automatically provides multi-line buffering for maximal I/O efficiency. .PP In many image processing systems it has been standard to apply operations sequentially over an image. This requires producing intermediate images. Since this is clearly inefficient in terms of I/O it has been the practice to copy the images into main memory and operate upon them there until the final image is ready to be saved. This has led to the perception that in order to be efficient an image processing system \fImust\fR store images in memory. This is not true and the IRAF CCD reduction package illustrates this. The CCD processing does not use intermediate images and does not need to keep the entire image in main memory. Furthermore, though of lesser importance than I/O, the single statement method illustrated above is more efficient than multiple passes through the images even when the images are kept in main memory. Finally, as CCD detectors increase in size and small, fast, and cheap processors become common it is a distinct advantage to not require the large amounts of memory needed to keep entire images in memory. .PP There is one area in which use of main memory can improve performance and \fBccdproc\fR does take advantage of it if desired. The calibration images usually are the same for many input images. By specifying the maximum amount of memory available for storing images in memory the calibration images may be stored in memory up to that amount. By parameterizing the memory requirement there is no builtin dependence on large memory! .PP After processing the input image the last steps are to log the operations in the image header using processing keywords and replace the input image by the output image as described earlier. The CCD coordinates of the data are recorded in the header, even if not there previously, to allow further processing on the image after the image has been trimmed. .NH Combining Images .PP The second important tool in the CCD reduction package is a task to combine many images into a single, higher quality image. While this may also be done with more general image processing tools (the IRAF task \fBimsum\fR for example) the \fBccdred\fR tasks include special CCD dependent features such as recognizing the image types and using the image header translation file. Combining images is often done with calibration images, which are easy to obtain in number, where it is important to minimize the statistical noise so as to not affect the object images. Sometimes object images also are combined. The task is called \fBcombine\fR and there are special versions of this task called \fBzerocombine, darkcombine\fR, and \fBflatcombine\fR for the standard calibration images. .PP The task takes a list of input images to be combined. As output there is the combined image, an optional sigma image, and optional log output either to the terminal, to a log file, or both. A subset or subsets of the input images may be selected based on the image type and a subset parameter such as the filter. As with the processing task, this allows selecting images without having to explicitly list each image from a large data set. When combining based on a subset parameter there is an output image, and possibly a sigma image, for each separate subset. The output image pixel datatype may also be changed during combining; usually from 16 bit integer input to real output. The sigma image is the standard deviation of the input images about the output image. .PP Except for summing the images together, combining images may require correcting for variations between the images due to differing exposure times, sky background, extinctions, and positions. Currently, extinction corrections and registration are not included but scaling and shifting corrections are included. The scaling corrections may be done by exposure times or by computing the mode in each image. Additive shifting is also done by computing the mode in the images. The region of the image in which the mode is computed can be specified but by default the whole image is used. A scaling correction is used when the flux level or sensitivity is varying. The offset correction is used when the sky brightness is varying independently of the object brightness. If the images are not scaled then special data paths combine the images more efficiently. .PP Except for medianing and summing, the images are combined by averaging. The average may be weighted by .nf .ft L weight = (N * scale / mode) ** 2 .ft R .fi where \fIN\fR is the number of images previously combined (the task records the number of images combined in the image header), \fIscale\fR is the relative scale (applied by dividing) from the exposure time or mode, and \fImode\fR is the background mode estimate used when adding a variable offset. .PP The combining operation is the heart of the task. There are a number algorithms which may be used as well as applying statistical weights. The algorithms are used to detect and reject deviant pixels, such as cosmic rays. The choice of algorithm depends on the data, the number of images, and the importance of rejecting cosmic rays. The more complex the algorithm the more time consuming the operation. The list below summarizes the algorithms. Further algorithms may be added in time. .IP "Sum - sum the input images" .br The input images are combined by summing. Care must be taken not to exceed the range of the 16 bit integer datatype when summing if the output datatype is of this type. Summing is the only algorithm in which scaling and weighting are not used. Also no sigma image is produced. .IP "Average - average the input images" .br The input images are combined by averaging. The images may be scaled and weighted. There is no pixel rejection. A sigma image is produced if more than one image is combined. .IP "Median - median the input images" .br The input images are combined by medianing each pixel. Unless the images are at the same exposure level they should be scaled. The sigma image is based on all the input images and is only an approximation to the uncertainty in the median estimates. .IP "Minreject, maxreject, minmaxreject - reject extreme pixels" .br At each pixel the minimum, maximum, or both are excluded from the average. The images should be scaled and the average may be weighted. The sigma image requires at least two pixels after rejection of the extreme values. These are relatively fast algorithms and are a good choice if there are many images (>15). .IP "Threshold - reject pixels above and below specified thresholds" .br The input images are combined with pixels above and below specified threshold values (before scaling) excluded. The images may be scaled and the average weighted. The sigma image also has the rejected pixels excluded. .IP "Sigclip - apply a sigma clipping algorithm to each pixel" .br The input images are combined by applying a sigma clipping algorithm at each pixel. The images should be scaled. This only rejects highly deviant points and so includes more of the data than the median or minimum and maximum algorithms. It requires many images (>10-15) to work effectively. Otherwise the bad pixels bias the sigma significantly. The mean used to determine the sigmas is based on the "minmaxrej" algorithm to eliminate the effects of bad pixels on the mean. Only one iteration is performed and at most one pixel is rejected at each point in the output image. After the deviant pixels are rejected the final mean is computed from all the data. The sigma image excludes the rejected pixels. .IP "Avsigclip - apply a sigma clipping algorithm to each pixel" .br The input images are combined with a variant of the sigma clipping algorithm which works well with only a few images. The images should be scaled. For each line the mean is first estimated using the "minmaxrej" algorithm. The sigmas at each point in the line are scaled by the square root of the mean, that is a Poisson scaling of the noise is assumed. These sigmas are averaged to get a line estimate of the sigma. Then the sigma at each point in the line is estimated by multiplying the line sigma by the square root of the mean at that point. As with the sigma clipping algorithm only one iteration is performed and at most one pixel is rejected at each point. After the deviant pixels are rejected the file mean is computed from all the data. The sigma image excludes the rejected pixels. .RE .PP The "avsigclip" algorithm is the best algorithm for rejecting cosmic rays, especially with a small number of images, but it is also the most time consuming. With many images (>10-15) it might be advisable to use one of the other algorithms ("maxreject", "median", "minmaxrej") because of their greater speed. .PP This task also has several design features to make it efficient and versatile. There are separate data paths for integer data and real data; as with processing, if all input images and the output image are of the same datatype then the I/O is done with no internal conversions. With mixed datatypes the operations are done as real. Even in the integer path the operations requiring real arithmetic to preserve the accuracy of the calculation are performed in that mode. There is effectively no limit to the number of images which may be combined. Also, the task determines the amount of memory available and buffers the I/O as much as possible. This is a case where operating on images from disk rather than in memory is essential. .NH Status and Conclusion .PP The initial implementation of the IRAF \fBccdred\fR package was completed in June 1987. It has been in use at the National Optical Astronomy Observatories since April 1987. The package was not distributed with Version 2.5 of IRAF (released in August 1987) but is available as a separate installation upon request. It will be part of future releases of IRAF. .PP At NOAO the CCD reduction package is available at the telescopes as the data is obtained. This is accomplished by transferring the images from the data taking computer to a Sun workstation (Sun Microsystems, Inc.) initially via tape and later by a direct link. There are several reasons for adopting this architecture. First, the data acquisition system is well established and is dedicated to its real-time function. The second computer was phased in without disrupting the essential operation of the telescopes and if it fails data taking may continue with data being stored on tape. The role of the second computer is to provide faster and more powerful reduction and analysis capability not required in a data acquisition system. In the future it can be more easily updated to follow the state of the art in small computers. As CCD detectors get larger the higher processing speeds will be essential to keep up with the data flow. .PP By writing the reduction software in the high level, portable, IRAF system the users have the capability to process their data from the basic CCD reductions to a full analysis at the telescope. Furthermore, the same software is widely available on a variety of computers if later processing or reprocessing is desired; staff and visitors at NOAO may also reduce their data at the headquarters facilities. The use of a high level system was also essential in achieving the design goals; it would be difficult to duplicate this complex package without the rich programming environment provided by the IRAF system. .NH References .PP The following documentation is distributed by the National Optical Astronomy Observatories, Central Computer Services, P.O. Box 26732, Tucson, Arizona, 85726. A comprehensive description of the IRAF system is given in \fIThe IRAF Data Reduction and Analysis System\fR by Doug Tody (also appearing in \fIProceedings of the SPIE - Instrumentation in Astronomy VI\fR, Vol. 627, 1986). A general guide to using IRAF is \fIA User's Introduction to the IRAF Command Language\fR by Peter Shames and Doug Tody. Both these documents are also part of the IRAF documentation distributed with the system. .PP A somewhat more tutorial description of the \fBccdred\fR package is \fIA User's Guide to the IRAF CCDRED Package\fR by the author. Detailed task descriptions and supplementary documentation are given in the on-line help library and are part of the user's guide. .NH Appendix .PP The current set of tasks making up the IRAF CCD Reduction Package, \fBccdred\fR, are summarized below. .nf .ft L badpiximage - Create a bad pixel mask image from a bad pixel file ccdgroups - Group CCD images into image lists ccdhedit - CCD image header editor ccdlist - List CCD processing information ccdproc - Process CCD images combine - Combine CCD images darkcombine - Combine and process dark count images flatcombine - Combine and process flat field images mkfringecor - Make fringe correction images from sky images mkillumcor - Make flat field illumination correction images mkillumflat - Make illumination corrected flat fields mkskycor - Make sky illumination correction images mkskyflat - Make sky corrected flat field images setinstrument - Set instrument parameters zerocombine - Combine and process zero level images .fi .ft R