diff options
Diffstat (limited to 'noao/imred/ccdred/doc')
28 files changed, 7183 insertions, 0 deletions
diff --git a/noao/imred/ccdred/doc/Notes b/noao/imred/ccdred/doc/Notes new file mode 100644 index 00000000..209faf30 --- /dev/null +++ b/noao/imred/ccdred/doc/Notes @@ -0,0 +1,96 @@ +12/15/93: +I have modified CCDPROC to more fully support scan table observations. In +combination with the ability to have the number of scan rows encoded in the +image header automatically, this allows such data to be processed in a +fairly foolproof and documented way. + +First if ccdproc.scancor=no then the NSCANROW keyword and nscan parameter +are ignored. For actual scanned data this may be useful to override +things. Otherwise the following steps are taken. The logic is slightly +complex so that everything is done in the right order and only as needed. + +The task wants to apply dark count and flat field calibration images which +have been scanned by the same number of rows. [Zero calibration images are +assumed not to be scanned. This made sense to me but if desired the zero +images can also be treated like the darks and flats.] This is similar to +the way flat fields are checked for subset (filter/grating). If the +appropriate dark or flat has not been scanned then it is scanned in +software; i.e. a moving average is taken over the unscanned image. + +The number of scan rows is determined for each object being processed from +the NSCANROW keyword or appropriate translation in the header translation +file. If this keyword is not found the nscan parameter value is used; +i.e. it is assumed the object image has been scanned by the specified +amount. This allows using the software in cases where the number of scan +rows is not encoded in the header. In the case of dark and flat images if +NSCANROW is not found a value of 1 (unscanned) is assumed. + +The set of possible calibration images (from the zero and flat parameters +or the list of input images) is searched for one which has been scanned +with the same number of lines as the object being processed. If one is +found it is processed as needed before applying to the object. If one is +not found then an unscanned one is sought. It is an error if neither can +be found. An unscanned image is first processed as necessary (overscan, +trim, etc.) and then scanned in software to create a new image. The new +image has the name of the unscanned image with the number of scan lines +appended, for example Flat1.32. It also has the NSCANROW keyword added as +well as a comment indicating the image from which it was created. This +approach allows the calibration image to be created only once for each +different scan format and the number of scan lines may be changed for +different observations and the appropriate calibration made from the +unscanned image. + +The following example shows how this all works. There are four object +images using two filters and two scan row values and unscanned +zero, dark, and flats. + +cc> dir +Dark.imh FlatV.imh obs019.imh obs021.imh pixels +FlatB.imh Zero.imh obs020.imh obs022.imh +cc> hselect obs* $I,filter,nscanrow yes +obs019.imh V 24 +obs020.imh V 32 +obs021.imh B 24 +obs022.imh B 32 +cc> ccdproc obs* overscan+ trim+ zerocor+ darkcor+ flatcor+ scancor+ +obs019.imh: Dec 15 17:53 Zero level correction image is Zero +Dark.imh: Dec 15 17:53 Zero level correction image is Zero +Dark.24.imh: Dec 15 17:53 Converted to shortscan from Dark.imh with nscan=24 +obs019.imh: Dec 15 17:53 Dark count correction image is Dark.24.imh +FlatV.imh: Dec 15 17:53 Zero level correction image is Zero +FlatV.imh: Dec 15 17:53 Dark count correction image is Dark.imh +FlatV.24.imh: Dec 15 17:53 Converted to shortscan from FlatV.imh with nscan=24 +obs019.imh: Dec 15 17:53 Flat field image is FlatV.24.imh +obs020.imh: Dec 15 17:53 Zero level correction image is Zero +Dark.32.imh: Dec 15 17:53 Converted to shortscan from Dark.imh with nscan=32 +obs020.imh: Dec 15 17:53 Dark count correction image is Dark.32.imh +FlatV.32.imh: Dec 15 17:53 Converted to shortscan from FlatV.imh with nscan=32 +obs020.imh: Dec 15 17:53 Flat field image is FlatV.32.imh +obs021.imh: Dec 15 17:53 Zero level correction image is Zero +obs021.imh: Dec 15 17:53 Dark count correction image is Dark.24.imh +FlatB.imh: Dec 15 17:53 Zero level correction image is Zero +FlatB.imh: Dec 15 17:53 Dark count correction image is Dark.imh +FlatB.24.imh: Dec 15 17:53 Converted to shortscan from FlatB.imh with nscan=24 +obs021.imh: Dec 15 17:53 Flat field image is FlatB.24.imh +obs022.imh: Dec 15 17:53 Zero level correction image is Zero +obs022.imh: Dec 15 17:53 Dark count correction image is Dark.32.imh +FlatB.32.imh: Dec 15 17:53 Converted to shortscan from FlatB.imh with nscan=32 +obs022.imh: Dec 15 17:53 Flat field image is FlatB.32.imh +cc> ccdlist *.imh +cc> ccdlist *.imh +Dark.24.imh[96,96][real][dark][][OTZ]: +Dark.32.imh[96,96][real][dark][][OTZ]: +Dark.imh[96,96][real][dark][][OTZ]: +FlatB.24.imh[96,96][real][flat][B][OTZD]: +FlatB.32.imh[96,96][real][flat][B][OTZD]: +FlatB.imh[96,96][real][flat][B][OTZD]: +FlatV.24.imh[96,96][real][flat][V][OTZD]: +FlatV.32.imh[96,96][real][flat][V][OTZD]: +FlatV.imh[96,96][real][flat][V][OTZD]: +Zero.imh[96,96][real][zero][][OT]: +obs019.imh[96,96][real][object][V][OTZDF]: +obs020.imh[96,96][real][object][V][OTZDF]: +obs021.imh[96,96][real][object][B][OTZDF]: +obs022.imh[96,96][real][object][B][OTZDF]: + +Frank diff --git a/noao/imred/ccdred/doc/badpiximage.hlp b/noao/imred/ccdred/doc/badpiximage.hlp new file mode 100644 index 00000000..46e13160 --- /dev/null +++ b/noao/imred/ccdred/doc/badpiximage.hlp @@ -0,0 +1,51 @@ +.help badpiximage Jun87 noao.imred.ccdred +.ih +NAME +badpiximage -- Create a bad pixel mask image from a bad pixel file +.ih +USAGE +badpiximage fixfile template image +.ih +PARAMETERS +.ls fixfile +Bad pixel file. +.le +.ls template +Template image used to define the size of the bad pixel mask image. +.le +.ls image +Bad pixel mask image to be created. +.le +.ls goodvalue = 1 +Integer value assigned to the good pixels. +.le +.ls badvalue = 0 +Integer value assigned to the bad pixels. +.le +.ih +DESCRIPTION +A bad pixel mask image is created from the specified bad pixel file. +The format of the bad pixel file is that used by \fBccdproc\fR to +correct CCD defects (see instruments). The bad pixel image is of pixel type short and +has the value given by the parameter \fBgoodvalue\fR for the good +pixels and the value given by the parameter \fBbadvalue\fR for the bad pixels. +The image size and header parameters are taken from the specified +template image. The bad pixel mask image may be used to view the +location of the bad pixels and blink against an data image using an +image display, to mask or flag bad pixels later by image arithmetic, +and to propagate the positions of the bad pixels through the +reductions. +.ih +EXAMPLES +1. To make a bad pixel mask image from the bad pixel file "cryocambp.dat" +using the image "ccd005" as the template: + + cl> badpiximage cryocambp.dat ccd005 cryocambp + +2. To make the bad pixel mask image with good values of 0 and bad values of 1: + + cl> badpixim cryomapbp.dat ccd005 cryocambp good=0 bad=1 +.ih +SEE ALSO +ccdproc, instruments +.endhelp diff --git a/noao/imred/ccdred/doc/ccdgeometry.hlp b/noao/imred/ccdred/doc/ccdgeometry.hlp new file mode 100644 index 00000000..a051ae5e --- /dev/null +++ b/noao/imred/ccdred/doc/ccdgeometry.hlp @@ -0,0 +1,73 @@ +.help ccdgeometry Sep87 noao.imred.ccdred +.ih +NAME +ccdgeometry - Discussion of CCD geometry and header parameters +.ih +DESCRIPTION +The \fBccdred\fR package maintains and updates certain geometry +information about the images. This geometry is described by four image +header parameters which may be present. These are defined below by the +parameter names used in the package. Note that these names may be +different in the image header using the image header translation +feature of the package. + +.ls DATASEC +The section of the image containing the CCD data. If absent the +entire image is assumed to be data. Only the pixels within the +data section are modified during processing. Therefore, there may be +additional calibration or observation information in the image. +If after processing, the data section is the entire image it is +not recorded in the image header. +.le +.ls CCDSEC +The section of the CCD to corresponding to the data section. This +refers to the physical format, columns and lines, of the detector. This is +the coordinate system used during processing to relate calibration +data to the image data; i.e. image data pixels are calibrated by +calibration pixels at the same CCD coordinates regardless of image pixel +coordinates. This allows recording only parts of the CCD during data +taking and calibrating with calibration frames covering some or all of +the CCD. The CCD section is maintained during trimming operations. +Note that changing the format of the images by image operators outside +of the \fBccdred\fR package will invalidate this coordinate system. +The size of the CCD section must agree with that of the data section. +If a CCD section is absent then it defaults to the data section such +that the first pixel of the data section has CCD coordinate (1,1). +.le +.ls BIASSEC +The section of the image containing prescan or overscan bias information. +It consists of a strip perpendicular to the readout axis. There may be +both a prescan and overscan but the package currently only uses one. +This parameter may be overridden during processing by the parameter +\fIccdproc.biassec\fR. Only the part of the bias section along the +readout is used and the length of the bias region is determined by +the trim section. If one wants to limit the region of the bias +strip used in the fit then the \fIsample\fR parameter should be used. +.le +.ls TRIMSEC +The section of the image extracted during processing when the trim +operation is selected (\fIccdproc.trim\fR). If absent when the trim +operation is selected it defaults to the data section; i.e. the processed +image consists only of the data section. This parameter may be overridden +during processing by the parameter \fIccdproc.trimsec\fR. After trimming +this parameter, if present, is removed from the image header. The +CCD section, data section, and bias section parameters are also modified +by trimming. +.le + +The geometry is as follows. When a CCD image is recorded it consists +of a data section corresponding to part or all of the CCD detector. +Regions outside of the data section may contain additional information +which are not affected except by trimming. Most commonly this consists +of prescan and overscan bias data. When recording only part of the +full CCD detector the package maintains information about that part and +correctly applies calibrations for that part of the detector. Also any +trimming operation updates the CCD coordinate information. If the +images include the data section, bias section, trim section, and ccd +section the processing may be performed entirely automatically. + +The sections are specified using the notation [c1:c2,l1:l2] where c1 +and c2 are the first and last columns and l1 and l2 are the first and +last lines. Currently c1 and l1 must be less than c2 and l2 +respectively and no subsampling is allowed. This may be added later. +.endhelp diff --git a/noao/imred/ccdred/doc/ccdgroups.hlp b/noao/imred/ccdred/doc/ccdgroups.hlp new file mode 100644 index 00000000..48c29b99 --- /dev/null +++ b/noao/imred/ccdred/doc/ccdgroups.hlp @@ -0,0 +1,163 @@ +.help ccdgroups Jun87 noao.imred.ccdred +.ih +NAME +ccdgroups -- Group CCD images into image lists +.ih +USAGE +ccdgroups images output +.ih +PARAMETERS +.ls images +List of CCD images to be grouped. +.le +.ls output +Output root group filename. The image group lists will be put in files +with this root name followed by a number. +.le +.ls group = "ccdtype" +Group type. There are currently four grouping types: +.ls ccdtype +Group by CCD image type. +.le +.ls subset +Group by subset parameter. +.le +.ls position +Group by position in right ascension (in hours) and declination (in degrees). +The groups are defined by a radius parameter (in arc seconds). +.le +.ls title +Group by identical titles. +.le +.ls date +Group by identical dates. +.le +.le +.ls radius = 60. +Grouping radius when grouping by positions. This is given in arc seconds. +.le +.ls ccdtype = "" +CCD image types to select from the input image list. If null ("") then +all image types are used. +.le +.ih +DESCRIPTION +The input images, possible restricted to a particular CCD image type, +are grouped into image lists. The "ccdtype" or "subset" groups +produce output image lists with the given root name and the CCD type +or subset as an extension (without a period). For the other group +types the +image lists have file names given by +the root output name and a numeric extension (without a period). +If the package parameter \fIccdred.verbose\fR is yes then the +image name and output group list is printed for each image. The image lists can +be used with the @ list feature for processing separate groups of observations. +Note that grouping by CCD image type and subset is often not necessary since +the \fBccdred\fR tasks automatically use this information (see +\fBccdtypes\fR and \fBsubsets\fR). + +Besides CCD image type and subsets there are currently three ways to +group images. These are by position in the sky, by title, and by +date. Further groups may be added as suggested. The title grouping is +useful if consistent titles are used when taking data. The date +grouping is useful if multiple nights of observations are not organized +by directories (it is recommended that data from separate nights be +kept in separate directories). The position grouping finds +observations within a given radius on the sky of the first member of +the group (this is not a clustering algorithm). The right ascension +and declination coordinates must be in standard units, hours and +degrees respectively. The grouping radius is in arc seconds. This +grouping type is useful for making sets of data in which separate +calibration images are taken at each position. + +The date, title, and coordinates are accessed through the instrument +translation file. The standard names used are "date-obs", "title", "ra", +and "dec". +.ih +EXAMPLES +1. For each object 5 exposures were taken to be combined in order to remove +cosmic rays. If the titles are the same then (with ccdred.verbose=yes): + +.nf + cl> ccdgroups *.imh group group=title ccdtype=object + ccd005.imh --> group1 + ccd006.imh --> group1 + ccd007.imh --> group1 + ccd008.imh --> group1 + ccd009.imh --> group1 + ccd012.imh --> group2 + ccd013.imh --> group2 + ccd014.imh --> group2 + ccd015.imh --> group2 + ccd016.imh --> group2 + [... etc ...] + cl> combine @group1 obj1 proc+ + cl> combine @group2 obj2 proc+ + [... etc ...] +.fi + +Note the numeric suffixes to the output root name "group". + +2. CCD observations were made in groups with a flat field, the object, and +a comparison spectrum at each position. To group and process this data: + +.nf + cl> ccdgroups *.imh obs group=position >> logfile + cl> ccdproc @obs1 + cl> ccdproc @obs2 + cl> ccdproc @obs3 +.fi + +Since no flat field is specified for the parameter \fIccdproc.flat\fR +the flat field is taken from the input image list. + +3. If for some reason you want to group by date and position it is possible +to use two steps. + +.nf + cl> ccdgroups *.imh date group=date + cl> ccdgroups @data1 pos1 + cl> ccdgroups @data2 pos2 +.fi + +4. To get groups by CCD image type: + +.nf + cl> ccdgroups *.imh "" group=ccdtype + ccd005.imh --> zero + ccd006.imh --> zero + ccd007.imh --> zero + ccd008.imh --> dark + ccd009.imh --> flat + ccd012.imh --> flat + ccd013.imh --> object + ccd014.imh --> object + ccd015.imh --> object + ccd016.imh --> object + [... etc ...] +.fi + +Note the use of a null root name and the extension is the standard +CCDRED types (not necessarily those used in the image header). + +5. To get groups by subset: + +.nf + cl> ccdgroups *.imh filt group=subset + ccd005.imh --> filt + ccd006.imh --> filtB + ccd007.imh --> filtB + ccd008.imh --> filtB + ccd009.imh --> filtV + ccd012.imh --> filtV + ccd013.imh --> filtV + ccd014.imh --> filtB + ccd015.imh --> filtB + ccd016.imh --> filtB + [... etc ...] +.fi + +.ih +SEE ALSO +ccdlist, ccdtypes, instruments, subsets +.endhelp diff --git a/noao/imred/ccdred/doc/ccdhedit.hlp b/noao/imred/ccdred/doc/ccdhedit.hlp new file mode 100644 index 00000000..1bc27d29 --- /dev/null +++ b/noao/imred/ccdred/doc/ccdhedit.hlp @@ -0,0 +1,108 @@ +.help ccdhedit Jun87 noao.imred.ccdred +.ih +NAME +ccdhedit -- CCD image header editor +.ih +USAGE +ccdhedit images parameter value +.ih +PARAMETERS +.ls images +List of CCD images to be edited. +.le +.ls parameter +Image header parameter. The image header parameter will be translated by +the header translation file for the images. +.le +.ls value +The parameter value. If the null string ("") is specified then the +parameter is deleted from the image header, otherwise it is added or +modified. If the parameter is "imagetyp" then the value string giving +the CCD image type is translated from the package CCD type to the +instrument specific string. +.le +.ls type = "string" +The parameter type. The parameter types are "string", "real", or "integer". +.le +.ih +DESCRIPTION +The image headers of the specified CCD images are edited to add, modify, +or delete a parameter. The parameters may be those used by the \fBccdred\fR +package. The parameter name is translated to an image header parameter by the +instrument translation file (see \fBinstruments\fR) if a translation is +given. Otherwise the parameter is that in the image header. If the parameter +is "imagetyp" the parameter value for the CCD image type may be that +used by the package; i.e. dark, object, flat, etc. The value string will be +translated to the instrument image string in this case. The translation +facility allows use of this task in an instrument independent way. + +The value string is used to determine whether to delete or modify the +image parameter. If the null string, "", is given the specified parameter +is deleted. If parameters are added the header type must be specified +as a string, real, or integer parameter. The numeric types convert the +value string to a number. +.ih +EXAMPLES +The \fBccdred\fR package is usable even with little image header information. +However, if desired the header information can be added to images which +lack it. In all the examples the parameters used are those of the package +and apply equally well to any image header format provided there is an +instrument translation file. + +.nf +1. cl> ccdhedit obj* imagetyp object +2. cl> ccdhedit flat* imagetyp flat +3. cl> ccdhedit zero* imagetyp zero +4. cl> ccdhedit obj0![1-3]* subset "V filter" +5. cl> ccdhedit obj0![45]* subset "R filter" +6. cl> ccdhedit flat001 subset "R filter" +7. cl> ccdhedit obj* exptime 500 type=integer +.fi + +8. The following is an example of a CL script which sets the CCD image type, +the subset, and the exposure time simultaneously. The user may expand +on this example to include other parameters or other initialization +operations. + +.nf + cl> edit ccdheader.cl + + ---------------------------------------------------------------- + # Program to set CCD header parameters. + + procedure ccdheader (images) + + string images {prompt="CCD images"} + string imagetyp {prompt="CCD image type"} + string subset {prompt="CCD subset"} + string exptime {prompt="CCD exposure time"} + + begin + string ims + + ims = images + ccdhedit (ims, "imagetyp", imagetyp, type="string") + ccdhedit (ims, "subset", subset, type="string") + ccdhedit (ims, "exptime", exptime, type="real") + end + ---------------------------------------------------------------- + + cl> task ccdheader=ccdheader.cl + cl> ccdheader obj* imagetyp=object subset="V" exptime=500 +.fi + +9. The image header may be changed to force processing a calibration image +as an object. For example to flatten a flat field: + +.nf + cl> ccdhedit testflat imagetyp other + cl> ccdproc testflat +.fi + +10. To delete processing flags: + + cl> ccdhedit obj042 flatcor "" +.ih +SEE ALSO +hedit, instruments, ccdtypes, subsets +.endhelp diff --git a/noao/imred/ccdred/doc/ccdinst.hlp b/noao/imred/ccdred/doc/ccdinst.hlp new file mode 100644 index 00000000..ea90f4a7 --- /dev/null +++ b/noao/imred/ccdred/doc/ccdinst.hlp @@ -0,0 +1,391 @@ +.help ccdinstrument Dec93 noao.imred.ccdred +.ih +NAME +ccdinstrument -- Setup and verify CCD instrument translation files +.ih +USAGE +ccdinstrument images +.ih +PARAMETERS +.ls images +List of images to be verified or used to setup a CCD instrument translation +file. +.le +.ls instrument = ")_.instrument" +CCD instrument translation file. The default is to use the translation +file defined in the \fBccdred\fR package parameters. Note that one would +need write permission to update this file though the task has a write +command to save any changes to a different file. +.le +.ls ssfile = ")_.ssfile" +Subset translation file. The default is to use the file defined in +the \fBccdred\fR package parameters. +.le +.ls edit = yes +Edit the instrument translation file? If "yes" an interactive +mode is entered allowing translation parameters to be modified while if +"no" the task is simply used to verify the translations noninteractively. +.le +.ls parameters = "basic" +Parameters to be displayed. The choices are "basic" to display only the +most basic parameters (those needed for the simplest automation of +\fBccdred\fR tasks), "common" to display the common parameters used +by the package (most of these are keywords to be written to the image +rather than translated), and "all" to display all the parameters +referenced by the package including the most obscure. For most uses +the "basic" set is all that is important and the other options are +included for completeness. +.le +.ih +DESCRIPTION +The purpose of this task is to provide an interface to simplify setting +up CCD instrument translation files and to verify the translations +for a set of images. Before this task was written users who needed to +set up translation files for new instruments and observatories had +to directly create the files with an editor. Many people encountered +difficulties and were prone to errors. Also there was no task that +directly verified the translations though \fBccdlist\fR provided some +clues. + +The \fBccdred\fR package was designed to make intelligent use of +information in image headers for determining things such as image +calibration or object type and exposure times. While the package may +be used without this capability it is much more convenient to be +able to use information from the image. The package was also intended +to be used with many different instruments, detectors, and observatories. +The key to providing image header access across different observatories +is the ability to translate the needs of the package to the appropriate +keywords in the image header. This is done through a file called +an "instrument translation file". For a complete description of +this file and other instrument setup features of the package see +\fBccdred.instruments\fR. + +The instrument translation file translates the parameter names used by +the \fBccdred\fR package into image specific parameters and also +supplies default values for parameters. The translation proceeds as +follows. When a package task needs a parameter for an image, for +example "imagetyp", it looks in the instrument translation file. If +the file is not found or none is specified then the image header +keyword that is requested is assumed to have the same name. If an +instrument translation file is defined then the requested parameter is +translated to an image header keyword, provided a translation entry is +given. If no translation is given the package name is used. For +example the package parameter "imagetyp" might be translated to +"data-typ" (the old NOAO CCD keyword). If the parameter is not found +then the default value specified in the translation file, if present, +is returned. + +For recording parameter information in the header, such +as processing flags, translation is also used. For example, if the +flag specifying that the image has been corrected by a flat field is to +be set then the package parameter name "flatcor" might be translated to +"ff-flag". If no translation is given then the new image header +parameter is entered as "flatcor". + +The CCD image type requires a second level of translation also defined +in the translation file. Once the image keyword which identifies the +type of CCD image, for example a flat field or object, is translated +to an imahe keyword the specific +string value must be translated to one of the CCD image types used +by the package. The translation works in the same way, the specific +string found is translated to the \fBccdred\fR type and returned to +the task. This translation is tricky in that the exact string +including all spaces and capitalizations must be correctly defined +in the translation file. The \fBccdinstrument\fR allows doing +this automatically thus minimizing typing errors. + +The basic display format of the task is a table of five columns +giving the parameter name used by the package, the image keyword +to which it is translated, the default value (if any), the value +the task will receive for the current image after translation, +and the actual keyword value in the image. A "?" is printed if +a value cannot be determined. The idea of the task is to make sure +that the value a \fBccdred\fR task sees is the correct one and if not +to modify the translation appropriately. In verify mode when the +\fBedit\fR parameter is not set the translation table is simply +printed for each input image. + +In edit mode the user interactively gives commands at the ccdinstrument +prompt to display or modify keywords. The modifications can then be +written to the instrument file or saved in a private copy. The +list of commands is shown below and may be printed using ? or help. + +.in 4 +.nf + CCDINSTRUMENT COMMANDS + +? Print command summary +help Print command summary +imheader Page image header +instrument Print current instrument translation file +next Next image +newimage Select a new image +quit Quit +read Read instrument translation file +show Show current translations +write Write instrument translation file + +translate Translate image string selected by the imagetyp + parameter to one of the CCDRED types given as an + argument or queried: + object, zero, dark, flat, comp, illum, fringe, other + +.fi +The following are CCDRED parameters which may be translated. You are +queried for the image keyword to use or it may be typed after the command. +An optional default value (returned if the image does not contain the +keyword) may be typed as the second argument of the command. +.nf + + BASIC PARAMETERS +imagetyp Image type parameter (see also translate) +subset Subset or filter parameter +exptime Exposure time +darktime Dark time (may be same as the exposure time) +.fi +.in -4 + +The commands may be followed by values such as file names for some of +the general commands or the keyword and default value for the parameters +to be translated. Note this is the only way to specify a default value. +If no arguments are given the user is prompted with the current value +which may then be changed. + +The set of parameters shown above are only those considered "basic". +In order to avoid confusion the task can limit the set of parameters +displayed. Without going into great detail, it is only the basic +parameters which are generally required to have valid translations to +allow the package to work well. However, for completeness, and if someone +wants to go wild with translations, further parameters may be displayed +and changed. The parameters displayed is controlled by the \fIparameters\fR +keyword. The additional parameters not shown above are: + +.in 4 +.nf + USEFUL DEFAULT GEOMETRY PARAMETERS +biassec Bias section (often has a default value) +trimsec Trim section (often has a default value) + + COMMON PROCESSING FLAGS +fixpix Bad pixel replacement flag +overscan Overscan correction flag +trim Trim flag +zerocor Zero level correction flag +darkcor Dark count correction flag +flatcor Flat field correction flag + + RARELY TRANSLATED PARAMETERS +ccdsec CCD section +datasec Data section +fixfile Bad pixel file + +fringcor Fringe correction flag +illumcor Ilumination correction flag +readcor One dimensional zero level read out correction +scancor Scan mode correction flag +nscanrow Number of scan rows + +illumflt Ilumination flat image +mkfringe Fringe image +mkillum Iillumination image +skyflat Sky flat image + +ccdmean Mean value +ccdmeant Mean value compute time +fringscl Fringe scale factor +ncombine Number of images combined +date-obs Date of observations +dec Declination +ra Right Ascension +title Image title +.fi +.in -4 +.ih +EXAMPLES +1. To verify the translations for a set of images using the default +translation file: + +.nf + cl> setinst "" review- + cl> ccdinst dev$pix edit- + Image: dev$pix + Instrument file: + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + -------------------------------- + imagetyp imagetyp none ? + subset subset ? + exptime exptime ? ? + darktime darktime ? ? + + cl> setinst "" site=kpno dir=ccddb$ review- + cl> ccdinst dev$pix edit- + Image: dev$pix + + Instrument file: ccddb$kpno/camera.dat + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + -------------------------------- + imagetyp data-typ object OBJECT (0) + subset f1pos 2 2 + exptime otime 600 600 + darktime ttime 600 600 +.fi + +2. Set up an instrument translation file from scratch. + +.nf + ccdinst ech???.imh instr=myccd edit+ + Warning: OPEN: File does not exist (myccd) + Image: ech001.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp imagetyp none ? + subset subset ? + exptime exptime ? ? + darktime darktime ? ? + + ccdinstrument> imagetyp + Image keyword for image type (imagetyp): ccdtype + imagetyp ccdtype unknown BIAS + ccdinstrument> translate + CCDRED image type for 'BIAS' (unknown): zero + imagetyp ccdtype zero BIAS + ccdinstrument> subset + Image keyword for subset parameter (subset): filters + subset filters 1 1 0 + ccdinstrument> exptime integ + exptime integ 0. 0. + ccdinstrument> darktime integ + darktime integ 0. 0. + ccdinstrument> show + Image: ech001.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype zero BIAS + subset filters 1 1 0 + exptime integ 0. 0. + darktime integ 0. 0. + + ccdinstrument> next + Image: ech002.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype unknown PROJECTOR FLAT + subset filters 1 1 0 + exptime integ 20. 20. + darktime integ 20. 20. + + ccdinstrument> trans + CCDRED image type for 'PROJECTOR FLAT' (unknown): flat + imagetyp ccdtype flat PROJECTOR FLAT + ccdinstrument> next + Image: ech003.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype unknown COMPARISON + subset filters 1 1 0 + exptime integ 300 300 + darktime integ 300 300 + + ccdinstrument> translate comp + imagetyp ccdtype comp COMPARISON + ccdinstrument> next + Image: ech004.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype unknown OBJECT + subset filters 1 1 0 + exptime integ 3600 3600 + darktime integ 3600 3600 + + ccdinstrument> translate object + imagetyp ccdtype object OBJECT + ccdinstrument> inst + imagetyp ccdtype + BIAS zero + subset filters + exptime integ + darktime integ + 'PROJECTOR FLAT' flat + COMPARISON comp + OBJECT object + + ccdinstrument> next + Update instrument file myccd (yes)? +.fi + +3. Set default geometry parameters. Note that to set a default the +arguments must be on the command line. + +.nf + cc> ccdinst ech001 instr=myccd param=common edit+ + Image: ech001 + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype zero BIAS + subset filters 1 1 0 + exptime integ 0. 0. + darktime integ 0. 0. + + biassec biassec ? ? + trimsec trimsec ? ? + + fixpix fixpix no ? + overscan overscan no ? + trim trim no ? + zerocor zerocor no ? + darkcor darkcor no ? + flatcor flatcor no ? + + ccdinstrument> biassec biassec [803:830,*] + biassec biassec [803:830,*] [803:830,*] ? + ccdinstrument> trimsec trimsec [2:798,2:798] + trimsec trimsec [2:798,2:798] [2:798,2:798] ? + ccdinstrument> instr + trimsec trimsec [2:798,2:798] + biassec biassec [803:830,*] + imagetyp ccdtype + BIAS zero + subset filters + exptime integ + darktime integ + 'PROJECTOR FLAT' flat + COMPARISON comp + OBJECT object + + ccdinstrument> q + Update instrument file myccd (yes)? +.fi +.ih +SEE ALSO +instruments, setinstrument +.endhelp diff --git a/noao/imred/ccdred/doc/ccdlist.hlp b/noao/imred/ccdred/doc/ccdlist.hlp new file mode 100644 index 00000000..9ce7dfdd --- /dev/null +++ b/noao/imred/ccdred/doc/ccdlist.hlp @@ -0,0 +1,133 @@ +.help ccdlist Jun87 noao.imred.ccdred +.ih +NAME +ccdlist -- List CCD processing information +.ih +USAGE +ccdlist images +.ih +PARAMETERS +.ls images +CCD images to be listed. A subset of the these may be selected using the +CCD image type parameter. +.le +.ls ccdtype = "" +CCD image type to be listed. If no type is specified then all the images +are listed. If an image type is specified then only images +of that type are listed. See \fBccdtypes\fR for a list of the package +image types. +.le +.ls names = no +List the image names only? Used with the CCD image type parameter to make +a list of the images of the specified type. +.le +.ls long = no +Long format listing? The images are listed in a long format containing some +image parameters and the processing history. +.le +.ls ccdproc (pset) +CCD processing parameter set. +.le +.ih +DESCRIPTION +Information from the specified input images is listed on the standard +output. A specific CCD image type may be selected from the input +images by the parameter \fIccdtype\fR. There are three list formats; +the default one line per image format, an image name only format, and a +multi-line long format. The default one line format consists of the +image name, image size, image pixel type, CCD image type, subset ID (if +defined), processing flags, and title. This format contains the same +information as that produced by \fBimheader\fR as well as CCD specific +information. The processing flags identifying the processing operations +performed on the image are given by the following single letter codes. + +.nf + B - Bad pixel replacement + O - Overscan bias subtraction + T - Trimming + Z - Zero level subtraction + D - Dark count subtraction + F - Flat field calibration + I - Iillumination correction + Q - Fringe correction +.fi + +The long format has the same first line as the default format plus additional +instrument information such as the exposure time and the full processing +history. In addition to listing the completed processing, the operations +not yet done (as specified by the \fBccdproc\fR parameters) are also +listed. + +The image name only format is intended to be used to generate lists of +images of the same CCD image type. These lists may be used as "@" file +lists in IRAF tasks. +.ih +EXAMPLES +1. To list the default format for all images: + +.nf + cl> ccdlist *.imh + ccd001.imh[544,512][short][unknown][V]:FOCUS L98-193 + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s + ccd045.imh[544,512][short][flat][V]:dflat 6v+blue 5s + ccd066.imh[544,512][short][flat][B]:dflat 6v+blue 5s + ccd103.imh[544,512][short][flat][R]:dflat 6v+blue 5s + ccd104.imh[544,512][short][zero][]:bias + ccd105.imh[544,512][short][dark][]:dark 3600s +.fi + +These images have not been processed. + +2. To restrict the listing to just the object images: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s +.fi + +3. The long list for image "ccd007" is obtained by: + +.nf + cl> ccdlist ccd007 l+ + ccd007[544,512][short][object][V]:N2968 R 600s + exptime = 200. darktime = 200. + [TO BE DONE] Overscan strip is [520:540,*] + [TO BE DONE] Trim image section is [3:510,3:510] + [TO BE DONE] Flat field correction +.fi + +4. After processing the images have the short listing: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[508,508][real][object][V][OTF]:N2968 V 600s + ccd015.imh[508,508][real][object][B][OTF]:N3098 B 500s + ccd024.imh[544,512][short][object][R][OTF]:N4036 R 600s +.fi + +The processing indicated is overscan subtraction, trimming, and flat fielding. + +5. The long listing for "ccd007" after processing is: + +.nf + cl> ccdlist ccd007 l+ + ccd007[508,508][real][object][V][OTF]:N2968 R 600s + exptime = 200. darktime = 200. + Jun 2 18:18 Overscan section is [520:540,*] with mean=481.8784 + Jun 2 18:18 Trim data section is [3:510,3:510] + Jun 2 18:19 Flat field image is FlatV.imh with scale=138.2713 +.fi + +6. To make a list file containing all the flat field images: + + cl> ccdlist *.imh ccdtype=flat name+ > flats + +This file can be used as an @ file for processing. +.ih +SEE ALSO +ccdtypes ccdgroups +.endhelp diff --git a/noao/imred/ccdred/doc/ccdmask.hlp b/noao/imred/ccdred/doc/ccdmask.hlp new file mode 100644 index 00000000..190ef016 --- /dev/null +++ b/noao/imred/ccdred/doc/ccdmask.hlp @@ -0,0 +1,138 @@ +.help ccdmask Jun96 noao.imred.ccdred +.ih +NAME +ccdmask -- create a pixel mask from a CCD image +.ih +USAGE +.nf +ccdmask image mask +.fi +.ih +PARAMETERS +.ls image +CCD image to use in defining bad pixels. Typically this is +a flat field image or, even better, the ratio of two flat field +images of different exposure levels. +.le +.ls mask +Pixel mask name to be created. A pixel list image, .pl extension, +is created so no extension is necessary. +.le +.ls ncmed = 7, nlmed = 7 +The column and line size of a moving median rectangle used to estimate the +uncontaminated local signal. The column median size should be at least 3 +pixels to span single bad columns. +.le +.ls ncsig = 15, nlsig = 15 +The column and line size of regions used to estimate the uncontaminated +local sigma using a percentile. The size of the box should contain +of order 100 pixels or more. +.le +.ls lsigma = 6, hsigma = 6 +Positive sigma factors to use for selecting pixels below and above +the median level based on the local percentile sigma. +.le +.ls ngood = 5 +Gaps of undetected pixels along the column direction of length less +than this amount are also flagged as bad pixels. +.le +.ls linterp = 2 +Mask code for pixels having a bounding good pixel separation which is +smaller along lines; i.e. to use line interpolation along the narrower +dimension. +.le +.ls cinterp = 3 +Mask code for pixels having a bounding good pixel separation which is +smaller along columns; i.e. to use columns interpolation along the narrower +dimension. +.le +.ls eqinterp = 2 +Mask code for pixels having a bounding good pixel separation which is +equal along lines and columns. +.le +.ih +DESCRIPTION +\fBCcdmask\fR makes a pixel mask from pixels deviating by a specified +statistical amount from the local median level. The input images may be of +any type but this task was designed primarily for detecting column oriented +CCD defects such as charge traps that cause bad columns and non-linear +sensitivities. The ideal input is a ratio of two flat fields having +different exposure levels so that all features which would normally flat +field properly are removed and only pixels which are not corrected by flat +fielding are found to make the pixel mask. A single flat field may also be +used but pixels of low or high sensitivity may be included as well as true +bad pixels. + +The input image is first subtracted by a moving box median. The median is +unaffected by bad pixels provided the median size is larger that twice +the size of a bad region. Thus, if 3 pixel wide bad columns are present +then the column median box size should be at least 7 pixels. The median +box can be a single pixel wide along one dimension if needed. This may be +appropriate for spectroscopic long slit data. + +The median subtracted image is then divided into blocks of size +\fInclsig\fR by \fInlsig\fR. In each block the pixel values are sorted and +the pixels nearest the 30.9 and 69.1 percentile points are found; this +would be the one sigma points in a Gaussian noise distribution. The +difference between the two count levels divided by two is then the local +sigma estimate. This algorithm is used to avoid contamination by the bad +pixel values. The block size must be at least 10 pixels in each dimension +to provide sufficient pixels for a good estimate of the percentile sigma. The +sigma uncertainty estimate of each pixel in the image is then the sigma +from the nearest block. + +The deviant pixels are found by comparing the median subtracted residual to +a specified sigma threshold factor times the local sigma above and below +zero (the \fIlsigma\fR and \fIhsigma\fR parameters). This is done for +individual pixels and then for column sums of pixels (excluding previously +flagged bad pixels) from two to the number of lines in the image. The sigma +of the sums is scaled by the square root of the number of pixels summed so +that statistically low or high column regions may be detected even though +individual pixels may not be statistically deviant. For the purpose of +this task one would normally select large sigma threshold factors such as +six or greater to detect only true bad pixels and not the extremes of the +noise distribution. + +As a final step each column is examined to see if there are small +segments of unflagged pixels between bad pixels. If the length +of a segment is less than that given by the \fIngood\fR parameter +all the pixels in the segment are also marked as bad. + +The bad pixel mask is created with good pixels identified by zero values +and the bad pixels by non-zero values. +The nearest good pixels along the columns and lines for +each bad pixel are located and the separation along the columns and lines +between those pixels is computed. The smaller separation is used to select +the mask value. If the smaller separation is along lines the \fIlinterp\fR +value is set, if the smaller separation is along columns the \fIcinterp\fR +value is set, and if the two are equal the \fIeqinterp\fR value is set. +The purpose of this is to allow interpolating across bad pixels using the +narrowest dimension. The task \fBfixpix\fR can select the type of pixel +replacement to use for each mask value. So one can chose, for example, +line interpolation for the linterp values and the eqinterp values, and +column interpolation for the cinterp values. + +In addition to this task, pixel mask images may be made in a variety of +ways. Any task which produces and modifies image values may be used. Some +useful tasks are \fBimexpr, imreplace, imcopy, text2mask\fR and +\fBmkpattern\fR. If a new image is specified with an explicit ".pl" +extension then the pixel mask format is produced. +.ih +EXAMPLES +1. Two flat fields of exposures 1 second and 3 seconds are taken, +overscan and zero corrected, and trimmed. These are then used +to generate a CCD mask. + +.nf + cl> imarith flat1 / flat2 ratio + cl> ccdmask ratio mask +.fi +.ih +REVISIONS +.ls CCDMASK V2.11 +This task is new. +.le +.ih +SEE ALSO +imreplace, imexpr, imcopy, imedit, fixpix, text2mask +.endhelp diff --git a/noao/imred/ccdred/doc/ccdproc.hlp b/noao/imred/ccdred/doc/ccdproc.hlp new file mode 100644 index 00000000..26ec6d1d --- /dev/null +++ b/noao/imred/ccdred/doc/ccdproc.hlp @@ -0,0 +1,825 @@ +.help ccdproc Dec93 noao.imred.ccdred +.ih +NAME +ccdproc -- Process CCD images +.ih +USAGE +ccdproc images +.ih +PARAMETERS +.ls images +List of input CCD images to process. The list may include processed +images and calibration images. +.le +.ls output = "" +List of output images. If no list is given then the processing will replace +the input images with the processed images. If a list is given it must +match the input image list. \fINote that any dependent calibration images +still be processed in-place with optional backup.\fR +.le +.ls ccdtype = "" +CCD image type to select from the input image list. If no type is given +then all input images will be selected. The recognized types are described +in \fBccdtypes\fR. +.le +.ls max_cache = 0 +Maximum image caching memory (in Mbytes). If there is sufficient memory +the calibration images, such as zero level, dark count, and flat fields, +will be cached in memory when processing many input images. This +reduces the disk I/O and makes the task run a little faster. If the +value is zero image caching is not used. +.le +.ls noproc = no +List processing steps only? +.le + +.ce +PROCESSING SWITCHES +.ls fixpix = yes +Fix bad CCD lines and columns by linear interpolation from neighboring +lines and columns? If yes then a bad pixel mask, image, or file must be +specified. +.le +.ls overscan = yes +Apply overscan or prescan bias correction? If yes then the overscan +image section and the readout axis must be specified. +.le +.ls trim = yes +Trim the image of the overscan region and bad edge lines and columns? +If yes then the data section must be specified. +.le +.ls zerocor = yes +Apply zero level correction? If yes a zero level image must be specified. +.le +.ls darkcor = yes +Apply dark count correction? If yes a dark count image must be specified. +.le +.ls flatcor = yes +Apply flat field correction? If yes flat field images must be specified. +.le +.ls illumcor = no +Apply iillumination correction? If yes iillumination images must be specified. +.le +.ls fringecor = no +Apply fringe correction? If yes fringe images must be specified. +.le +.ls readcor = no +Convert zero level images to readout correction images? If yes then +zero level images are averaged across the readout axis to form one +dimensional zero level readout correction images. +.le +.ls scancor = no +Convert zero level, dark count and flat field images to scan mode flat +field images? If yes then the form of scan mode correction is specified by +the parameter \fIscantype\fR. +.le + +.ce +PROCESSING PARAMETERS +.ls readaxis = "line" +Read out axis specified as "line" or "column". +.le +.ls fixfile +Bad pixel mask, image, or file. If "image" is specified then the name is +specified in the image header or instrument translation file. If "BPM" is +specified then the standard BPM image header keyword defines a bad pixel +mask. A bad pixel mask is a compact format (".pl" extension) with zero +values indicating good pixels and non-zero values indicating bad pixels. A +bad pixel image is a regular image in which zero values are good pixels and +non-zero values are bad pixels. A bad pixel file specifies bad pixels or +rectangular bad pixel regions as described later. The direction of +interpolation is determined by the mask value with a value of two +interpolating across columns, a value of three interpolating across lines, +and any other non-zero value interpolating along the narrowest dimension. +.le +.ls biassec +Overscan bias strip image section. If "image" is specified then the overscan +bias section is specified in the image header or instrument translation file. +Only the part of the bias section along the readout axis is used. The +length of the bias region fit is defined by the trim section. If one +wants to limit the region of the overscan used in the fit to be less +than that of the trim section then the sample region parameter, +\fIsample\fR, should be used. It is an error if no section or the +whole image is specified. +.le +.ls trimsec +image section for trimming. If "image" is specified then the trim +image section is specified in the image header or instrument translation file. +.le +.ls zero = "" +Zero level calibration image. The zero level image may be one or two +dimensional. The CCD image type and subset are not checked for these +images and they take precedence over any zero level calibration images +given in the input list. +.le +.ls dark = "" +Dark count calibration image. The CCD image type and subset are not checked +for these images and they take precedence over any dark count calibration +images given in the input list. +.le +.ls flat = "" +Flat field calibration images. The flat field images may be one or +two dimensional. The CCD image type is not checked for these +images and they take precedence over any flat field calibration images given +in the input list. The flat field image with the same subset as the +input image being processed is selected. +.le +.ls illum = "" +Iillumination correction images. The CCD image type is not checked for these +images and they take precedence over any iillumination correction images given +in the input list. The iillumination image with the same subset as the +input image being processed is selected. +.le +.ls fringe = "" +Fringe correction images. The CCD image type is not checked for these +images and they take precedence over any fringe correction images given +in the input list. The fringe image with the same subset as the +input image being processed is selected. +.le +.ls minreplace = 1. +When processing flat fields, pixel values below this value (after +all other processing such as overscan, zero, and dark corrections) are +replaced by this value. This allows flat fields processed by \fBccdproc\fR +to be certain to avoid divide by zero problems when applied to object +images. +.le +.ls scantype = "shortscan" +Type of scan format used in creating the CCD images. The modes are: +.ls "shortscan" +The CCD is scanned over a number of lines and then read out as a regular +two dimensional image. In this mode unscanned zero level, dark count and +flat fields are numerically scanned to form scanned flat fields comparable +to the observations. +.le +.ls "longscan" +In this mode the CCD is clocked and read out continuously to form a long +strip. Flat fields are averaged across the readout axis to +form a one dimensional flat field readout correction image. This assumes +that all recorded image lines are clocked over the entire active area of the +CCD. +.le +.le +.ls nscan +Number of object scan readout lines used in short scan mode. This parameter +is used when the scan type is "shortscan" and the number of scan lines +cannot be determined from the object image header (using the keyword +nscanrows or it's translation). +.le + + +.ce +OVERSCAN FITTING PARAMETERS + +There are two types of overscan (or prescan) determinations. One determines +a independent overscan value for each line and is only available for a +\fIreadaxis\fR of 1. The other averages the overscan along the readout +direction to make an overscan vector, fits a smoothing function to the vector, +and then evaluate and then evaluates the smooth function at each readout +line or column. The line-by-line determination only uses the +\fIfunction\fR parameter and the smoothing determinations uses all +the following parameters. + +.ls function = "legendre" +Line-by-line determination of the overscan is specified by: + +.nf + mean - the mean of the biassec columns at each line + median - the median of the biassec columns at each line + minmax - the mean at each line with the min and max excluded +.fi + +The smoothed overscan vector may be fit by one of the functions: + +.nf + legendre - legendre polynomial + chebyshev - chebyshev polynomial + spline1 - linear spline + spline3 - cubic spline +.fi +.le +.ls order = 1 +Number of polynomial terms or spline pieces in the overscan fit. +.le +.ls sample = "*" +Sample points to use in the overscan fit. The string "*" specified all +points otherwise an \fBicfit\fR range string is used. +.le +.ls naverage = 1 +Number of points to average or median to form fitting points. Positive +numbers specify averages and negative numbers specify medians. +.le +.ls niterate = 1 +Number of rejection iterations to remove deviant points from the overscan fit. +If 0 then no points are rejected. +.le +.ls low_reject = 3., high_reject = 3. +Low and high sigma rejection factors for rejecting deviant points from the +overscan fit. +.le +.ls grow = 0. +One dimensional growing radius for rejection of neighbors to deviant points. +.le +.ls interactive = no +Fit the overscan vector interactively? If yes and the overscan function type +is one of the \fBicfit\fR types then the average overscan vector is fit +interactively using the \fBicfit\fR package. If no then the fitting parameters +given below are used. +.le +.ih +DESCRIPTION +\fBCcdproc\fR processes CCD images to correct and calibrate for +detector defects, readout bias, zero level bias, dark counts, +response, iillumination, and fringing. It also trims unwanted +lines and columns and changes the pixel datatype. It is efficient +and easy to use; all one has to do is set the parameters and then +begin processing the images. The task takes care of most of the +record keeping and automatically does the prerequisite processing +of calibration images. Beneath this simplicity there is much that +is going on. In this section a simple description of the usage is +given. The following sections present more detailed discussions +on the different operations performed and the order and logic +of the processing steps. For a user's guide to the \fBccdred\fR +package see \fBguide\fR. Much of the ease of use derives from using +information in the image header. If this information is missing +see section 13. + +One begins by setting the task parameters. There are many parameters +but they may be easily reviewed and modified using the task \fBeparam\fR. +The input CCD images to be processed are given as an image list. +Previously processed images are ignored and calibration images are +recognized, provided the CCD image types are in the image header (see +\fBinstruments\fR and \fBccdtypes\fR). Therefore it is permissible to +use simple image templates such as "*.imh". The \fIccdtype\fR parameter +may be used to select only certain types of CCD images to process +(see \fBccdtypes\fR). + +The processing operations are selected by boolean (yes/no) parameters. +Because calibration images are recognized and processed appropriately, +the processing operations for object images should be set. +Any combination of operations may be specified and the operations are +performed simultaneously. While it is possible to do operations in +separate steps this is much less efficient. Two of the operation +parameters apply only to zero level and flat field images. These +are used for certain types of CCDs and modes of operation. + +The processing steps selected have related parameters which must be +set. These are things like image sections defining the overscan and +trim regions and calibration images. There are a number of parameters +used for fitting the overscan or prescan bias section. These are +parameters used by the standard IRAF curve fitting package \fBicfit\fR. +The parameters are described in more detail in the following sections. + +In addition to the task parameters there are package parameters +which affect \fBccdproc\fR. These include the instrument and subset +files, the text and plot log files, the output pixel datatype, +the amount of memory available for calibration image caching, +the verbose parameter for logging to the terminal, and the backup +prefix. These are described in \fBccdred\fR. + +Calibration images are specified by task parameters and/or in the +input image list. If more than one calibration image is specified +then the first one encountered is used and a warning is issued for the +extra images. Calibration images specified by +task parameters take precedence over calibration images in the input list. +These images also need not have a CCD image type parameter since the task +parameter identifies the type of calibration image. This method is +best if there is only one calibration image for all images +to be processed. This is almost always true for zero level and dark +count images. If no calibration image is specified by task parameter +then calibration images in the input image list are identified and +used. This requires that the images have CCD image types recognized +by the package. This method is useful if one may simply say "*.imh" +as the image list to process all images or if the images are broken +up into groups, in "@" files for example, each with their own calibration +frames. + +When an input image is processed the task first determines the processing +parameters and calibration images. If a requested operation has been +done it is skipped and if all requested operations have been completed then +no processing takes place. When it determines that a calibration image +is required it checks for the image from the task parameter and then +for a calibration image of the proper type in the input list. + +Having +selected a calibration image it checks if it has been processed for +all the operations selected by the CCDPROC parameters. +After the calibration images have been identified, and processed if +necessary, the images may be cached in memory. This is done when there +are more than two input images (it is actually less efficient to +cache the calibration images for one or two input images) and the parameter +\fImax_cache\fR is greater than zero. When caching, as many calibration +images as allowed by the specified memory are read into memory and +kept there for all the input images. Cached images are, therefore, +only read once from disk which reduces the amount of disk I/O. This +makes a modest decrease in the execution time. It is not dramatic +because the actual processing is fairly CPU intensive. + +Once the processing parameters and calibration images have been determined +the input image is processed for all the desired operations in one step; +i.e. there are no intermediate results or images. This makes the task +efficient. If a matching list of output images is given then the processed +image is written to the specified output image name. If no output image +list is given then the corrected image is output as a temporary image until +the entire image has been processed. When the image has been completely +processed then the original image is deleted (or renamed using the +specified backup prefix) and the corrected image replaces the original +image. Using a temporary image protects the data in the event of an abort +or computer failure. Keeping the original image name eliminates much of +the record keeping and the need to generate new image names. +.sh +1. Fixpix +Regions of bad lines and columns may be replaced by linear +interpolation from neighboring lines and columns when the parameter +\fIfixpix\fR is set. This algorithm is the same as used in the +task \fBfixpix\fR. The bad pixels may be specified by a pixel mask, +an image, or a text file. For the mask or image, values of zero indicate +good pixels and other values indicate bad pixels to be replaced. + +The text file consists of lines with four fields, the starting and +ending columns and the starting and ending lines. Any number of +regions may be specified. Comment lines beginning with the character +'#' may be included. The description applies directly to the input +image (before trimming) so different files are needed for previously +trimmed or subsection readouts. The data in this file is internally +turned into the same description as a bad pixel mask with values of +two for regions which are narrower or equal across the columns and +a value of three for regions narrower across lines. + +The direction of interpolation is determined from the values in the +mask, image, or the converted text file. A value of two interpolates +across columns, a value of three interpolates across lines, and any +other value interpolates across the narrowest dimension of bad pixels +and using column interpolation if the two dimensions are equal. + +The bad pixel description may be specified explicitly with the parameter +\fIfixfile\fR or indirectly if the parameter has the value "image". In the +latter case the instrument file must contain the name of the file. +.sh +2. Overscan +If an overscan or prescan correction is specified (\fIoverscan\fR +parameter) then the image section (\fIbiassec\fR parameter) defines +the overscan region. + +There are two types of overscan (or prescan) determinations. One determines +a independent overscan value for each line and is only available for a +\fIreadaxis\fR of 1. The other averages the overscan along the readout +direction to make an overscan vector, fits a smoothing function to the vector, +and then evaluate and then evaluates the smooth function at each readout +line or column. + +The line-by-line determination provides an mean, median, or +mean with the minimum and maximum values excluded. The median +is lowest value of the middle two when the number of overscan columns +is even rather than the mean. + +The smoothed overscan vector determination uses the \fBicfit\fR options +including interactive fitting. The fitting function is generally either a +constant (polynomial of 1 term) or a high order function which fits the +large scale shape of the overscan vector. Bad pixel rejection is also +available to eliminate cosmic ray events. The function fitting may be done +interactively using the standard \fBicfit\fR iteractive graphical curve +fitting tool. Regardless of whether the fit is done interactively, the +overscan vector and the fit may be recorded for later review in a metacode +plot file named by the parameter \fIccdred.plotfile\fR. The mean value of +the bias function is also recorded in the image header and log file. +.sh +3. Trim +When the parameter \fItrim\fR is set the input image will be trimmed to +the image section given by the parameter \fItrimsec\fR. This trim +should, of course, be the same as that used for the calibration images. +.sh +4. Zerocor +After the readout bias is subtracted, as defined by the overscan or prescan +region, there may still be a zero level bias. This level may be two +dimensional or one dimensional (the same for every readout line). A +zero level calibration is obtained by taking zero length exposures; +generally many are taken and combined. To apply this zero +level calibration the parameter \fIzerocor\fR is set. In addition if +the zero level bias is only readout dependent then the parameter \fIreadcor\fR +is set to reduce two dimensional zero level images to one dimensional +images. The zero level images may be specified by the parameter \fIzero\fR +or given in the input image list (provided the CCD image type is defined). + +When the zero level image is needed to correct an input image it is checked +to see if it has been processed and, if not, it is processed automatically. +Processing of zero level images consists of bad pixel replacement, +overscan correction, trimming, and averaging to one dimension if the +readout correction is specified. +.sh +5. Darkcor +Dark counts are subtracted by scaling a dark count calibration image to +the same exposure time as the input image and subtracting. The +exposure time used is the dark time which may be different than the +actual integration or exposure time. A dark count calibration image is +obtained by taking a very long exposure with the shutter closed; i.e. +an exposure with no light reaching the detector. The dark count +correction is selected with the parameter \fIdarkcor\fR and the dark +count calibration image is specified either with the parameter +\fIdark\fR or as one of the input images. The dark count image is +automatically processed as needed. Processing of dark count images +consists of bad pixel replacement, overscan and zero level correction, +and trimming. +.sh +6. Flatcor +The relative detector pixel response is calibrated by dividing by a +scaled flat field calibration image. A flat field image is obtained by +exposure to a spatially uniform source of light such as an lamp or +twilight sky. Flat field images may be corrected for the spectral +signature in spectroscopic images (see \fBresponse\fR and +\fBapnormalize\fR), or for iillumination effects (see \fBmkillumflat\fR +or \fBmkskyflat\fR). For more on flat fields and iillumination corrections +see \fBflatfields\fR. The flat field response is dependent on the +wavelength of light so if different filters or spectroscopic wavelength +coverage are used a flat field calibration for each one is required. +The different flat fields are automatically selected by a subset +parameter (see \fBsubsets\fR). + +Flat field calibration is selected with the parameter \fBflatcor\fR +and the flat field images are specified with the parameter \fBflat\fR +or as part of the input image list. The appropriate subset is automatically +selected for each input image processed. The flat field image is +automatically processed as needed. Processing consists of bad pixel +replacement, overscan subtraction, zero level subtraction, dark count +subtraction, and trimming. Also if a scan mode is used and the +parameter \fIscancor\fR is specified then a scan mode correction is +applied (see below). The processing also computes the mean of the +flat field image which is used later to scale the flat field before +division into the input image. For scan mode flat fields the ramp +part is included in computing the mean which will affect the level +of images processed with this flat field. Note that there is no check for +division by zero in the interest of efficiency. If division by zero +does occur a fatal error will occur. The flat field can be fixed by +replacing small values using a task such as \fBimreplace\fR or +during processing using the \fIminreplace\fR parameter. Note that the +\fIminreplace\fR parameter only applies to flat fields processed by +\fBccdproc\fR. +.sh +7. Illumcor +CCD images processed through the flat field calibration may not be +completely flat (in the absence of objects). In particular, a blank +sky image may still show gradients. This residual nonflatness is called +the iillumination pattern. It may be introduced even if the detector is +uniformly illuminated by the sky because the flat field lamp +iillumination may be nonuniform. The iillumination pattern is found from a +blank sky, or even object image, by heavily smoothing and rejecting +objects using sigma clipping. The iillumination calibration image is +divided into the data being processed to remove the iillumination +pattern. The iillumination pattern is a function of the subset so there +must be an iillumination correction image for each subset to be +processed. The tasks \fBmkillumcor\fR and \fBmkskycor\fR are used to +create the iillumination correction images. For more on iillumination +corrections see \fBflatfields\fR. + +An alternative to treating the iillumination correction as a separate +operation is to combine the flat field and iillumination correction +into a corrected flat field image before processing the object +images. This will save some processing time but does require creating +the flat field first rather than correcting the images at the same +time or later. There are two methods, removing the large scale +shape of the flat field and combining a blank sky image iillumination +with the flat field. These methods are discussed further in the +tasks which create them; \fBmkillumcor\fR and \fBmkskycor\fR. +.sh +8. Fringecor +There may be a fringe pattern in the images due to the night sky lines. +To remove this fringe pattern a blank sky image is heavily smoothed +to produce an iillumination image which is then subtracted from the +original sky image. The residual fringe pattern is scaled to the +exposure time of the image to be fringe corrected and then subtracted. +Because the intensity of the night sky lines varies with time an +additional scaling factor may be given in the image header. +The fringe pattern is a function of the subset so there must be +a fringe correction image for each subset to be processed. +The task \fBmkfringecor\fR is used to create the fringe correction images. +.sh +9. Readcor +If a zero level correction is desired (\fIzerocor\fR parameter) +and the parameter \fIreadcor\fR is yes then a single zero level +correction vector is applied to each readout line or column. Use of a +readout correction rather than a two dimensional zero level image +depends on the nature of the detector or if the CCD is operated in +longscan mode (see below). The readout correction is specified by a +one dimensional image (\fIzero\fR parameter) and the readout axis +(\fIreadaxis\fR parameter). If the zero level image is two dimensional +then it is automatically processed to a one dimensional image by +averaging across the readout axis. Note that this modifies the zero +level calibration image. +.sh +10. Scancor +CCD detectors may be operated in several modes in astronomical +applications. The most common is as a direct imager where each pixel +integrates one point in the sky or spectrum. However, the design of most CCD's +allows the sky to be scanned across the CCD while shifting the +accumulating signal at the same rate. \fBCcdproc\fR provides for two +scanning modes called "shortscan" and "longscan". The type of scan +mode is set with the parameter \fIscanmode\fR. + +In "shortscan" mode the detector is scanned over a specified number of +lines (not necessarily at sideral rates). The lines that scroll off the +detector during the integration are thrown away. At the end of the +integration the detector is read out in the same way as an unscanned +observation. The advantage of this mode is that the small scale, zero +level, dark count and flat field responses are averaged in one dimension +over the number of lines scanned. A zero level, dark count or flat field may be +observed in the same way in which case there is no difference in the +processing from unscanned imaging and the parameter \fIscancor\fR may be +no. If it is yes, though, checking is done to insure that the calibration +image used has the same number of scan lines as the object being +processed. However, one obtains an increase in the statistical accuracy of +if they are not scanned during the observation but +digitally scanned during the processing. In shortscan mode with +\fIscancor\fR set to yes, zero level, dark count and flat field images are +digitally scanned, if needed, by the same number of scan lines as the +object. The number of scan lines is determined from the object image +header using the keyword nscanrow (or it's translation). If not found the +object is assumed to have been scanned with the value given by the +\fInscan\fR parameter. Zero, dark and flat calibration images are assumed +to be unscanned if the header keyword is not found. + +If a scanned zero level, dark count or flat field image is not found +matching the object then one may be created from the unscanned calibration +image. The image will have the root name of the unscanned image with an +extension of the number of scan rows; i.e. Flat1.32 is created from Flat1 +with a digital scanning of 32 lines. + +In "longscan" mode the detector is continuously read out to produce an +arbitrarily long strip. Provided data which has not passed over the entire +detector is thrown away, the zero level, dark count, and flat field +corrections will be one dimensional. If \fIscancor\fR is specified and the +scan mode is "longscan" then a one dimensional zero level, dark count, and +flat field correction will be applied. +.sh +11. Processing Steps +The following describes the steps taken by the task. This detailed +outline provides the most detailed specification of the task. + +.ls 5 (1) +An image to be processed is first checked that it is of the specified +CCD image type. If it is not the desired type then go on to the next image. +.le +.ls (2) +A temporary output image is created of the specified pixel data type +(\fBccdred.pixeltype\fR). The header parameters are copied from the +input image. +.le +.ls (3) +If trimming is specified and the image has not been trimmed previously, +the trim section is determined. +.le +.ls (4) +If bad pixel replacement is specified and this has not been done +previously, the bad pixel file is determined either from the task +parameter or the instrument translation file. The bad pixel regions +are read. If the image has been trimmed previously and the bad pixel +file contains the word "untrimmed" then the bad pixel coordinates are +translated to those of the trimmed image. +.le +.ls (5) +If an overscan correction is specified and this correction has not been +applied, the overscan section is averaged along the readout axis. If +trimming is to be done the overscan section is trimmed to the same +limits. A function is fit either interactively or noninteractively to +the overscan vector. The function is used to produce the overscan +vector to be subtracted from the image. This is done in real +arithmetic. +.le +.ls (6) +If the image is a zero level image go to processing step 12. +If a zero level correction is desired and this correction has not been +performed, find the zero level calibration image. If the zero level +calibration image has not been processed it is processed at this point. +This is done by going to processing step 1 for this image. After the +calibration image has been processed, processing of the input image +continues from this point. +The processed calibration image may be +cached in memory if it has not been previously and if there is enough memory. +.le +.ls (7) +If the image is a dark count image go to processing step 12. +If a dark count correction is desired and this correction has not been +performed, find the dark count calibration image. If the dark count +calibration image has not been processed it is processed at this point. +This is done by going to processing step 1 for this image. After the +calibration image has been processed, processing of the input image +continues from this point. The ratio of the input image dark time +to the dark count image dark time is determined to be multiplied with +each pixel of the dark count image before subtracting from the input +image. +The processed calibration image may be +cached in memory if it has not been previously and if there is enough memory. +.le +.ls (8) +If the image is a flat field image go to processing step 12. If a flat +field correction is desired and this correction has not been performed, +find the flat field calibration image of the appropriate subset. If +the flat field calibration image has not been processed it is processed +at this point. This is done by going to processing step 1 for this +image. After the calibration image has been processed, processing of +the input image continues from this point. The mean of the image +is determined from the image header to be used for scaling. If no +mean is found then a unit scaling is used. +The processed calibration image may be +cached in memory if it has not been previously and if there is enough memory. +.le +.ls (9) +If the image is an iillumination image go to processing step 12. If an +iillumination correction is desired and this correction has not been performed, +find the iillumination calibration image of the appropriate subset. +The iillumination image must have the "mkillum" processing flag or the +\fBccdproc\fR will abort with an error. The mean of the image +is determined from the image header to be used for scaling. If no +mean is found then a unit scaling is used. The processed calibration +image may be +cached in memory if it has not been previously and there is enough memory. +.le +.ls (10) +If the image is a fringe image go to processing step 12. If a fringe +correction is desired and this correction has not been performed, +find the fringe calibration image of the appropriate subset. +The iillumination image must have the "mkfringe" processing flag or the +\fBccdproc\fR will abort with an error. The ratio of the input +image exposure time to the fringe image exposure time is determined. +If there is a fringe scaling in the image header then this factor +is multiplied by the exposure time ratio. This factor is used +for scaling. The processed calibration image may be +cached in memory if it has not been previously and there is enough memory. +.le +.ls (11) +If there are no processing operations flagged, delete the temporary output +image, which has been opened but not used, and go to 14. +.le +.ls (12) +The input image is processed line by line with trimmed lines ignored. +A line of the input image is read. Bad pixel replacement and trimming +is applied to the image. Image lines from the calibration images +are read from disk or the image cache. If the calibration is one +dimensional (such as a readout zero +level correction or a longscan flat field correction) then the image +vector is read only once. Note that IRAF image I/O is buffered for +efficiency and accessing a line at a time does not mean that image +lines are read from disk a line at a time. Given the input line, the +calibration images, the overscan vector, and the various scale factors +a special data path for each combination of corrections is used to +perform all the processing in the most efficient manner. If the +image is a flat field any pixels less than the \fIminreplace\fR +parameter are replaced by that minimum value. Also a mean is +computed for the flat field and stored as the CCDMEAN keyword and +the time, in a internal format, when this value was calculated is stored +in the CCDMEANT keyword. The time is checked against the image modify +time to determine if the value is valid or needs to be recomputed. +.le +.ls (13) +The input image is deleted or renamed to a backup image. The temporary +output image is renamed to the input image name. +.le +.ls (14) +If the image is a zero level image and the readout correction is specified +then it is averaged to a one dimensional readout correction. +.le +.ls (15) +If the image is a zero level, dark count, or flat field image and the scan +mode correction is specified then the correction is applied. For shortscan +mode a modified two dimensional image is produced while for longscan mode a +one dimensional average image is produced. +.le +.ls (16) +The processing is completed and either the next input image is processed +beginning at step 1 or, if it is a calibration image which is being +processed for an input image, control returns to the step which initiated +the calibration image processing. +.le +.sh +12. Processing Arithmetic +The \fBccdproc\fR task has two data paths, one for real image pixel datatypes +and one for short integer pixel datatype. In addition internal arithmetic +is based on the rules of FORTRAN. For efficiency there is +no checking for division by zero in the flat field calibration. +The following rules describe the processing arithmetic and data paths. + +.ls (1) +If the input, output, or any calibration image is of type real the +real data path is used. This means all image data is converted to +real on input. If all the images are of type short all input data +is kept as short integers. Thus, if all the images are of the same type +there is no datatype conversion on input resulting in greater +image I/O efficiency. +.le +.ls (2) +In the real data path the processing arithmetic is always real and, +if the output image is of short pixel datatype, the result +is truncated. +.le +.ls (3) +The overscan vector and the scale factors for dark count, flat field, +iillumination, and fringe calibrations are always of type real. Therefore, +in the short data path any processing which includes these operations +will be coerced to real arithmetic and the result truncated at the end +of the computation. +.le +.sh +13. In the Absence of Image Header Information +The tasks in the \fBccdred\fR package are most convenient to use when +the CCD image type, subset, and exposure time are contained in the +image header. The ability to redefine which header parameters contain +this information makes it possible to use the package at many different +observatories (see \fBinstruments\fR). However, in the absence of any +image header information the tasks may still be used effectively. +There are two ways to proceed. One way is to use \fBccdhedit\fR +to place the information in the image header. + +The second way is to specify the processing operations more explicitly +than is needed when the header information is present. The parameter +\fIccdtype\fR is set to "" or to "none". The calibration images are +specified explicitly by task parameter since they cannot be recognized +in the input list. Only one subset at a time may be processed. + +If dark count and fringe corrections are to be applied the exposure +times must be added to all the images. Alternatively, the dark count +and fringe images may be scaled explicitly for each input image. This +works because the exposure times default to 1 if they are not given in +the image header. +.ih +EXAMPLES +The user's \fBguide\fR presents a tutorial in the use of this task. + +1. In general all that needs to be done is to set the task parameters +and enter + + cl> ccdproc *.imh & + +This will run in the background and process all images which have not +been processed previously. +.ih +TIME REQUIREMENTS +.nf +o SUN-3, 15 MHz 68020 with 68881 floating point hardware (no FPA) +o 8 Mb RAM, 2 Fuji Eagle disks. +o Input images = 544 x 512 short +o Output image = 500 x 500 real +o Operations are overscan subtraction (O), trimming to 500x500 (T), + zero level subtraction (Z), dark count scaling and subtraction (D), + and flat field scaling and subtraction (F). +o UNIX statistics + (user, system, and clock time, and misc. memory and i/o statistics): + +[OTF] One calibration image and 9 object images: +No caching: 110.6u 25.5s 3:18 68% 28+ 40K 3093+1645io 9pf+0w +Caching: 111.2u 23.0s 2:59 74% 28+105K 2043+1618io 9pf+0w + +[OTZF] Two calibration images and 9 object images: +No caching: 119.2u 29.0s 3:45 65% 28+ 50K 4310+1660io 9pf+0w +Caching: 119.3u 23.0s 3:07 75% 28+124K 2179+1601io 9pf+0w + +[OTZDF] Three calibration images and 9 object images: +No caching: 149.4u 31.6s 4:41 64% 28+ 59K 5501+1680io 19pf+0w +Caching: 151.5u 29.0s 4:14 70% 27+227K 2346+1637io 148pf+0w + +[OTZF] 2 calibration images and 20 images processed: +No caching: 272.7u 63.8u 8:47 63% 28+ 50K 9598+3713io 12pf+0w +Caching: 271.2u 50.9s 7:00 76% 28+173K 4487+3613io 51pf+0w +.fi +.ih +REVISIONS +.ls CCDPROC V2.11.2 +A new "output" parameter is available to specify an output image leaving +the input image unchanged. If this parameter is not specified then +the previous behavior of "in-place" operation with an optional backup +occurs. +.le +.ls CCDPROC V2.11 +The bad pixel fixing was modified to allow use of pixel masks, +images, or the text file description. Bad pixel masks are the +desired description and use of text files is only supported for +backward compatibility. Note that support for the trimmed +or untrimmed conversion from text files has been eliminated. + +Line-by-line overscan/prescan subtraction is now provided with +three simple algorithms. +.le +.ls CCDPROC: V2.10.3 +The output pixel datatypes (specified by the package parameter +\fIpixeltype\fR have been extended to include unsigned short +integers. Also it was previously possible to have the output +pixel datatype be of lower precision than the input. Now the +output pixel datatype is not allowed to lose precision; i.e. +a real input image may not be processed to a short datatype. + +For short scan data the task now looks for the number of scan lines in the +image header. Also when a calibration image is software scanned a new +image is created. This allows processing objects with different numbers of +scan lines and preserving the unscanned calibration image. + +It is an error if no biassec is specified rather than defaulting to +the whole image. + +The time, in a internal format, when the CCDMEAN value is calculated is +stored in the CCDMEANT keyword. The time is checked against the image +modify time to determine if the value is valid or needs to be recomputed. +.le +.ih +SEE ALSO +.nf +instruments, ccdtypes, flatfields, icfit, ccdred, guide, mkillumcor, +mkskycor, mkfringecor +.fi +.endhelp diff --git a/noao/imred/ccdred/doc/ccdred.hlp b/noao/imred/ccdred/doc/ccdred.hlp new file mode 100644 index 00000000..f2cca5bd --- /dev/null +++ b/noao/imred/ccdred/doc/ccdred.hlp @@ -0,0 +1,104 @@ +.help package Dec93 noao.imred +.ih +NAME +ccdred -- CCD image reduction package +.ih +USAGE +ccdred +.ih +PARAMETERS +.ls pixeltype = "real real" +Output pixel datatype and calculation datatype. When images are processed +or created the output pixel datatype is determined by this parameter if the +specified datatype is of equal or higher precision otherwise the input +image datatype is preserved. For example if the output datatype is +specified as "input" then input images which are "short" or "ushort" will +be output as integer but any real datatype input images will remain real. +The allowed types and order of precision are "short", "ushort", "int", +"long", "real", or "double", for short signed integer, short unsigned +integer, integer, long integers, and real or double floating point. Note +that if short input images are processed into real images the disk space +required will generally increase. The calculation datatypes may only be +short and real with a default of real if none is specified. +.le +.ls verbose = no +Print log information to the standard output? +.le +.ls logfile = "logfile" +Text log file. If no filename is specified then no log file is kept. +.le +.ls plotfile = "" +Log metacode plot file for the overscan bias vector fits. If no filename +is specified then no metacode plot file is kept. +.le +.ls backup = "" +Backup prefix for backup images. If no prefix is specified then no backup +images are kept when processing. If specified then the backup image +has the specified prefix. +.le +.ls instrument = "" +CCD instrument translation file. This is usually set with +\fBsetinstrument\fR. +.le +.ls ssfile = "subsets" +Subset translation file used to define the subset identifier. See +\fBsubsets\fR for more. +.le +.ls graphics = "stdgraph" +Interactive graphics output device when fitting the overscan bias vector. +.le +.ls cursor = "" +Graphics cursor input. The default is the standard graphics cursor. +.le +.ls version = "June 1987" +Package version. +.le +.ih +DESCRIPTION +The CCD reduction package is loaded when this command is entered. The +package contains parameters which affect the operation of the tasks it +defines. When images are processed or new image are created the output +pixel datatype is that specified by the parameter \fBpixeltype\fR. Note +that CCD processing replaces the original image by the processed image so +the pixel type of the CCD images may change during processing. The output +pixel type is not allowed to change to a lower precision but it is common +for input short images to be processed to real images. Processing images +from short to real pixel datatypes will generally increase the amount of +disk space required (a factor of 2 on most computers). + +The tasks produce log output which may be printed on the standard +output (the terminal unless redirected) and appended to a file. The +parameter \fIverbose\fR determines whether processing information +is printed. This may be desirable initially, but when using background +jobs the verbose output should be turned off. The user may look at +the end of the log file (for example with \fBtail\fR) to determine +the status of the processing. + +The package was designed to work with data from many different observatories +and instruments. In order to accomplish this an instrument translation +file is used to define a mapping between the package parameters and +the particular image header format. The instrument translation file +is specified to the package by the parameter \fIinstrument\fR. This +parameter is generally set by the task \fBsetinstrument\fR. The other +file used is a subset file. This is generally created and maintained +by the package and the user need not do anything. For more sophisticated +users see \fBinstruments\fR and \fBsubsets\fR. + +The package has very little graphics +output. The exception is the overscan bias subtraction. The bias +vector is logged in the metacode plot file if given. The plot file +may be examined with the tasks in the \fBplot\fR package such as +\fBgkimosaic\fR. When interactively fitting the overscan vector +the graphics input and output devices must be specified. The defaults +should apply in most cases. + +Because processing replaces the input image by the processed image it +may be desired to save the original image. This may be done by +specifying a backup prefix with the parameter \fIbackup\fR. For +example, if the prefix is "orig" and the image is "ccd001", the backup +image will be "origccd001". The prefix may be a directory but it must +end with '/' or '$' (for logical directories). +.ih +SEE ALSO +ccdproc, instruments, setinstrument, subsets +.endhelp diff --git a/noao/imred/ccdred/doc/ccdred.ms b/noao/imred/ccdred/doc/ccdred.ms new file mode 100644 index 00000000..645514ec --- /dev/null +++ b/noao/imred/ccdred/doc/ccdred.ms @@ -0,0 +1,787 @@ +.RP +.TL +The IRAF CCD Reduction Package -- CCDRED +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +September 1987 +.AB +The IRAF\(dg CCD reduction package, \fBccdred\fR, provides tools +for the easy and efficient reduction of CCD images. The standard +reduction operations are replacement of bad pixels, subtraction of an +overscan or prescan bias, subtraction of a zero level image, +subtraction of a dark count image, division by a flat field calibration +image, division by an illumination correction, subtraction of a fringe +image, and trimming unwanted lines or columns. Another common +operation provided by the package is scaling and combining images with +a number of algorithms for rejecting cosmic rays. Data in the image +header is used to make the reductions largely automated and +self-documenting though the package may still be used in the absence of +this data. Also a translation mechanism is used to relate image header +parameters to those used by the package to allow data from a variety of +observatories and instruments to be processed. This paper describes +the design goals for the package and the main tasks and algorithms +which satisfy these goals. +.PP +This paper is to be published as part of the proceedings of the +Santa Cruz Summer Workshop in Astronomy and Astrophysics, +\fIInstrumentation for Ground-Based Optical Astronomy: Present and +Future\fR, edited by Lloyd B. Robinson and published by +Springer-Verlag. +.LP +\(dgImage Reduction and Analysis Facility (IRAF), a software system +distributed by the National Optical Astronomy Observatories (NOAO). +.AE +.NH +Introduction +.PP +The IRAF CCD reduction package, \fBccdred\fR, provides tools +for performing the standard instrumental corrections and calibrations +to CCD images. The major design goals were: +.IP +.nf +\(bu To be easy to use +\(bu To be largely automated +\(bu To be image header driven if the data allows +\(bu To be usable for a variety of instruments and observatories +\(bu To be efficient and capable of processing large volumes of data +.fi +.LP +This paper describes the important tasks and algorithms and shows how +these design goals were met. It is not intended to describe every +task, parameter, and usage in detail; the package has full +documentation on each task plus a user's guide. +.PP +The standard CCD correction and calibration operations performed are +replacement of bad columns and lines by interpolation from neighboring +columns and lines, subtraction of a bias level determined from overscan +or prescan columns or lines, subtraction of a zero level using a zero +length exposure calibration image, subtraction of a dark count +calibration image appropriately scaled to the dark time exposure of the +image, division by a scaled flat field calibration image, division by +an illumination image (derived from a blank sky image), subtraction of +a scaled fringe image (also derived from a blank sky image), and +trimming the image of unwanted lines or columns such as the overscan +strip. The processing may change the pixel datatype on disk (IRAF allows +seven image datatypes); usually from 16 bit integer to real format. +Two special operations are also supported for scan mode and one +dimensional zero level and flat field calibrations; i.e. the same +calibration is applied to each CCD readout line. Any set of operations +may be done simultaneously over a list of images in a highly efficient +manner. The reduction operations are recorded in the image header and +may also be logged on the terminal and in a log file. +.PP +The package also provides tools for combining multiple exposures +of object and calibration images to improve the statistical accuracy of +the observations and to remove transient bad pixels. The combining +operation scales images of different exposure times, adjusts for +variable sky background, statistically weights the images by their +signal-to-noise, and provides a number of useful algorithms for +detecting and rejecting transient bad pixels. +.PP +Other tasks are provided for listing reduction information about +the images, deriving secondary calibration images (such as sky +corrected flat fields or illumination correction images), and easily +setting the package parameters for different instruments. +.PP +This paper is organized as follows. There is a section giving an +overview of how the package is used to reduce CCD data. This gives the +user's perspective and illustrates the general ease of use. The next +section describes many of the features of the package contributing to +its ease of use, automation, and generality. The next two sections +describe the major tools and algorithms in some detail. This includes +discussions about achieving high efficiency. Finally the status of the +package and its use at NOAO is given. References to additional +documentation about IRAF and the CCD reduction package and an appendix +listing the individual tasks in the package are found at the end of +this paper. +.NH +A User's Overview +.PP +This section provides an overview of reducing data with the IRAF CCD +reduction package. There are many variations in usage depending on the +type of data, whether the image headers contain information about the +data which may be used by the tasks, and the scientific goal. Only a +brief example is given. A more complete discussion of usage and +examples is given in \fIA User's Guide to the IRAF CCDRED Package\fR. +The package was developed within the IRAF system and so makes use of +all the sophisticated features provided. These features are also +summarized here for those not familiar with IRAF since they are an +important part of using the package. +.PP +Since the IRAF system is widely distributed and runs on a wide variety +of computers, the site of the CCD reductions might be at the telescope, +a system at the observatory provided for this purpose, or at the +user's home computer. The CCD images to be processed are either +available immediately as the data is taken, transferred from the data taking +computer via a network link (the method adopted at NOAO), or transferred +to the reduction computer via a medium such as magnetic tape in FITS +format. The flexibility in reduction sites and hardware is one of the +virtues of the IRAF-based CCD reduction package. +.PP +IRAF tasks typically have a number of parameters which give the user +control over most aspects of the program. This is possible since the +parameters are kept in parameter files so that the user need not enter +a large number of parameters every time the task is run. The user may +change any of these parameters as desired in several ways, such as by +explicit assignment and using an easy to learn and use, +fill-in-the-value type of screen editor. The parameter values are +\fIlearned\fR so that once a user sets the values they are maintained +until the user changes them again; even between login sessions. +.PP +The first step in using the CCD reduction package is to set the default +processing parameters for the data to be reduced. These parameters include +a database file describing the image header keyword translations and +default values, the processing operations desired (operations +required vary with instrument and observer), the calibration image names, +and certain special parameters for special types of observations such +as scan mode. A special script task (a command procedure) is available +to automatically set the default values, given the instrument name, to standard +values defined by the support staff. Identifying the instrument in this +way may be all the novice user need do though most people quickly learn +to adjust parameters at will. +.PP +As an example suppose there is an instrument identified as \fLrca4m\fR +for an RCA CCD at the NOAO 4 meter telescope. The user gives the command + +.ft L + cl> setinstrument rca4m +.ft R + +which sets the default parameters to values suggested by the support staff +for this instrument. The user may then change these suggested values if +desired. In this example the processing switches are set to perform +overscan bias subtraction, zero level image subtraction, flat fielding, +and trimming. +.PP +The NOAO image headers contain information identifying the type of +image, such as object, zero level, and flat field, the filter used to +match flat fields with object images, the location of the overscan bias +data, the trim size for the data, and whether the image has been +processed. With this information the user need not worry about +selecting images, pairing object images with calibration images, or +inadvertently reprocessing an image. +.PP +The first step is to combine multiple zero level and flat field observations +to reduce the effects of statistical noise. This is done by the +commands + +.nf +.ft L + cl> zerocombine *.imh + cl> flatcombine *.imh +.ft R +.fi + +The "cl> " is the IRAF command language prompt. The first command says +look through all the images and combine the zero level images. The +second command says look through all the images and combine the flat +field images by filter. What could be simpler? Some \fIhidden\fR (default) +parameters the user may modify are the combined image name, whether to +process the images first, and the type of combining algorithm to use. +.PP +The next step is to process the images using the combined calibration +images. The command is + +.ft L + cl> ccdproc *.imh +.ft R + +This command says look through all the images, find the object images, +find the overscan data based on the image header and subtract the +bias, subtract the zero level calibration image, divide by the flat field +calibration image, and trim the bias data and edge lines and columns. +During this operation the task recognizes that the +zero level and flat field calibration images have not been processed +and automatically processes them when they are needed. The log output +of this task, which may be to the terminal, to a file, or both, shows +how this works. + +.nf +.ft L + ccd003: Jun 1 15:12 Trim data section is [3:510,3:510] + ccd003: Jun 1 15:12 Overscan section is [520:540,*], mean=485.0 + Dark: Jun 1 15:12 Trim data section is [3:510,3:510] + Dark: Jun 1 15:13 Overscan section is [520:540,*], mean=484.6 + ccd003: Jun 1 15:13 Dark count image is Dark.imh + FlatV: Jun 1 15:13 Trim data section is [3:510,3:510] + FlatV: Jun 1 15:14 Overscan section is [520:540,*], mean=486.4 + ccd003: Jun 1 15:15 Flat field image is FlatV.imh, scale=138.2 + ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] + ccd004: Jun 1 15:16 Overscan section is [520:540,*], mean=485.2 + ccd004: Jun 1 15:16 Dark count image is Dark.imh + ccd004: Jun 1 15:16 Flat field image is FlatV.imh, scale=138.2 + \fI<... more ...>\fL + ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] + ccd013: Jun 1 15:23 Overscan section is [520:540,*], mean=482.4 + ccd013: Jun 1 15:23 Dark count image is Dark.imh + FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] + FlatB: Jun 1 15:23 Overscan section is [520:540,*], mean=486.4 + ccd013: Jun 1 15:24 Flat field image is FlatB.imh, scale=132.3 + \fI<... more ...>\fL +.ft R +.fi + +.PP +The log gives the name of the image and a time stamp for each entry. +The first image is ccd003. It is to be trimmed to the specified +size given as an \fIimage section\fR, an array notation used commonly +in IRAF to specify subsections of images. The location of the +overscan data is also given by an image section which, in this case, +was found in the image header. The mean bias level of the overscan +is also logged though the overscan is actually a function of the +readout line with the order of the function selected by the user. +.PP +When the task comes to subtracting the zero level image it first +notes that the calibration image has not been processed and switches +to processing the zero level image. Since it knows it is a zero level +image the task does not attempt to zero level or flat field correct +this image. After the zero level image has been processed the task +returns to the object image only to find that the flat field image +also has not been processed. It determines that the object image was +obtained with a V filter and selects the flat field image having the same +filter. The flat field image is processed through the zero level correction +and then the task again returns to the object image, ccd003, which it +finishes processing. +.PP +The next image, ccd004, is also a V filter +observation. Since the zero level and V filter flat field have been +processed the object image is processed directly. This continues +for all the object images except for a detour to process the B filter flat +field when the task first encounters a B filter object image. +.PP +In summary, the basic usage of the CCD reduction package is quite simple. +First, the instrument is identified and some parameters for the data +are set. Calibration images are then combined if needed. Finally, +the processing is done with the simple command + +.ft L + cl> ccdproc *.imh& +.ft R + +where the processing is performed as a \fIbackground job\fR in this example. +This simplicity was a major goal of the package. +.NH +Features of the Package +.PP +This section describes some of the special features of the package +which contribute to its ease of use, generality, and efficiency. +The major criteria for ease of use are to minimize the user's record keeping +involving input and output image names, the types of images, subset +parameters such as filters which must be kept separate, and the state +of processing of each image. The goal is to allow input images to +be specified using simple wildcards, such as "*.imh" to specify all +images, with the knowledge that the task will only operate on images +for which it makes sense. To accomplish this the tasks must be able to +determine the type of image, subset, and the state of processing from +the image itself. This is done by making use of image header parameters. +.PP +For generality the package does not require any image header information +except the exposure time. It is really not very much more difficult to +reduce such data. Mainly, the user must be more explicit about specifying +images and setting task parameters or add the information to the image +headers. Some default header information may also be set in the image +header translation file (discussed below). +.PP +One important image header parameter is the image type. This +discriminates between object images and various types of calibration +images such as flat field, zero level, dark count, comparison arcs, +illumination, and fringe images. This information is used in two +ways. For most of the tasks the user may select that only one type of +image be considered. Thus, all the flat field images may be selected +for combining or only the processing status of the object images be +listed. The second usage is to allow the processing tasks to identify +the standard calibration images and apply only those operations which +make sense. For example, flat field images are not divided by a +flat field. This allows the user to set the processing operations +desired for the object images without fear of misprocessing the +calibration images. The image type is also used to automatically +select calibration images from a list of images to be processed instead +of explicitly identifying them. +.PP +A related parameter specifies the subset. For certain operations the +images must have a common value for this parameter. This parameter is +often the filter but it may also apply to a grating or aperture, for example. +The subset parameter is used to identify the appropriate flat field +image to apply to an image or to select common flat fields to be combined +into a higher quality flat field. This is automatic and the user need not +keep track of which image was taken with which filter or grating. +.PP +The other important image header parameters are the processing flags. +These identify when an image has been processed and also act as a history +of the operation including calibration images used and other parameter +information. The usage of these parameters is obvious; it allows the +user to include processed images in a wildcard list knowing that the +processing will not be repeated and to quickly determine the processing +status of the image. +.PP +Use of image header parameters often ties the software to the a +particular observatory. To maintain generality and usefulness for data +other than that at NOAO, the CCD reduction package was designed to +provide a translation between parameters requested by the package and +those actually found in the image header. This translation is defined +in a simple text file which maps one keyword to another and also gives +a default value to be used if the image header does not include a +value. In addition the translation file maps the arbitrary strings +which may identify image types to the standard types which the package +recognizes. This is a relatively simple scheme and does not allow for +forming combinations or for interpreting values which are not simple +such as embedding an exposure time as part of a string. A more complex +translation scheme may prove desirable as experience is gained with +other types of image header formats, but by then a general header translation +ability and/or new image database structure may be a standard IRAF +feature. +.PP +This feature has proven useful at NOAO. During the course of +developing the package the data taking system was modernized by +updating keywords and adding new information in the image headers, +generally following the lines laid out by the \fBccdred\fR package. +However, there is a period of transition and it is also desirable to +reduce preexisting data. There are several different formats for this +data. The header translation files make coping with these different +formats relatively easy. +.PP +A fundamental aspect of the package is that the processing +modifies the images. In other words, the reduction operations are +performed directly on the image. This "feature" further simplifies +record keeping, frees the user from having to form unique output image +names, and minimizes the amount of disk space required. There +are two safety features in this process. First, the modifications do +not take effect until the operation is completed on the image. This +allows the user to abort the task without leaving the image data in a +partially processed state and protects data if the computer +crashes. The second feature is that there is a parameter which may be +set to make a backup of the input data with a particular prefix; for +example "b", "orig", or "imdir$" (a logical directory prefix). This +backup feature may be used when there is sufficient disk space, when +learning to use the package, or just to be cautious. +.PP +In a similar effort to efficiently manage disk space, when combining +images into a master object or calibration image, there is an option to +delete the input images upon completion of the combining operation. +Generally this is desirable when there are many calibration exposures, +such as zero level or flat field images, which are not used after they +are combined into a final calibration image. +.PP +The goal of generality for many instruments at +different observatories inherently conflicts with the goal of ease of +use. Generality requires many parameters and options. This is +feasible in the CCD reduction package, as well as the other IRAF packages, +because of the IRAF parameter handling mechanism. In \fBccdred\fR +there still remains the problem of setting the parameters appropriately +for a particular instrument, image header format, and observatory. +.PP +To make this convenient there is a task, \fBsetinstrument\fR, that, +based on an instrument name, runs a setup script for the instrument. +An example of this task was given in the previous section. +The script may do any type of operation but mainly it sets default +parameters. The setup scripts are generally created by the support staff +for the instrument. The combination of the setup script and the +instrument translation file make the package, in a sense, programmable +and achieves the desired instrument/observatory generality with ease of use. +.NH +CCD Processing +.PP +This section describes in some detail how the CCD processing is performed. +The task which does the basic CCD processing is call \fBccdproc\fR. +From the point of view of usage the task is very simple but a great deal +is required to achieve this simplicity. The approach we take in describing +the task is to follow the flow of control as the task runs with digressions +as appropriate. +.PP +The highest level of control is a loop over the input images; all the +operations are performed successively on each image. It is common for +IRAF tasks which operate on individual images to allow the operation to +be repeated automatically over a list of input images. This is important +in the \fBccdred\fR package because data sets are often large and the +processing is generally the same for each image. It would be tedious +to have to give the processing command for each image to be processed. +If an error occurs while processing an image the error is +printed as a warning and processing continues with the next image. +This provides protection primarily against mistyped or nonexistent images. +.PP +Before the first image is processed the calibration images are +identified. There are two ways to specify calibration images; +explicitly via task parameters or implicitly as part of the list of +images to be processed. Explicitly identifying calibration images +takes precedence over calibration images in the input list. Specifying +calibration images as part of the input image list requires that the +image types can be determined from the image header. Using the input +list provides a mechanism for breaking processing up into sets of +images (possibly using files containing the image names for each set) +each having their own calibration images. One can, of course, +selectively specify input and calibration images, but whenever possible +one would like to avoid having to specify explicit images to process +since this requires record keeping by the user. +.PP +The first step in processing an image is to check that it is of the +appropriate image type. The user may select to process images of only +one type. Generally this is object images since calibration images are +automatically processed as needed. Images which are not of the desired +type are skipped and the next image is considered. +.PP +A temporary output image is created next. The output pixel datatype on +disk may be changed at this point as selected by the user. +For example it is common for the raw CCD images to be digitized as 16 +bit integers but after calibration it is sometimes desirable to have +real format pixels. If no output pixel datatype is specified the +output image takes the same pixel datatype as the input image. The +processing is done by operating on the input image and writing the +results to a temporary output image. When the processing is complete +the output image replaces the input image. This gives the effect of +processing the images in place but with certain safeguards. If the +computer crashes or the processing is interrupted the integrity of the +input image is maintained. The reasons for chosing to process the +images in this way are to avoid having to generate new image names (a +tiresome record keeping process for the user), to minimize disk +usage, and generally the unprocessed images are not used once they have +been processed. When dealing with large volumes of data these reasons +become fairly important. However, the user may specify a backup prefix +for the images in which case, once the processing is completed, the +original input image is renamed by appending it to the prefix (or with +an added digit if a previous backup image of the same name exits) +before the processed output image takes the original input name. +.PP +The next step is to determine the image geometry. Only a subsection of +the raw image may contain the CCD data. If this region is specified by +a header parameter then the processing will affect only this region. +This allows calibration and other data to be part of the image. +Normally, the only other data in a image is overscan or prescan data. +The location of this bias data is determined from the image header or +from a task parameter (which overrides the image header value). To +relate calibration images of different sizes and to allow for readout +of only a portion of the CCD detector, a header parameter may relate +the image data coordinates to the full CCD coordinates. Application of +calibration image data and identifying bad pixel regions via a bad +pixel file is done in this CCD coordinate system. The final +geometrical information is the region of the input image to be output +after processing; an operation called trimming. This is defined by an +image header parameter or a task parameter. Trimming of the image is +selected by the user. Any or all of this geometry information may be +absent from the image and appropriate defaults are used. +.PP +Each selected operation which is appropriate for the image type is then +considered. If the operation has been performed previously it will not +be repeated. If all selected operations have been performed then the +temporary output image is deleted and the input image is left +unchanged. The next image is then processed. +.PP +For each selected operation to be performed the pertinent data is +determined. This consists of such things as the name of the +calibration image, scaling factors, the overscan bias function, etc. +Note that at this point only the parameters are determined, the +operation is not yet performed. This is because operations are not +performed sequentially but simultaneously as described below. Consider +flat fielding as an example. First the input image is checked to see +if it has been flat fielded. Then the flat field calibration image is +determined. The flat field image is checked to see if it has been +processed. If it has not been processed then it is processed by +calling a procedure which is essentially a copy of the main processing +program. After the flat field image has been processed, parameters +affecting the processing, such as the flat field scale factor +(essentially the mean of the flat field image), are determined. A log +of the operation is then printed if desired. +.PP +Once all the processing operations and parameters have been defined the +actual processing begins. One of the key design goals was that the +processing be efficient. There are two primary methods used to achieve +this goal; separate processing paths for 16 bit integer data and +floating point data and simultaneous operations. If the image, the +calibration images, and the output image (as selected by the user) are +16 bit integer pixel datatypes then the image data is read and written +as integer data. This eliminates internal datatype conversions both +during I/O and during computations. However, many operations include +use of real factors such as the overscan bias, dark count exposure +scaling, and flat field scaling which causes the computation to be done +in real arithmetic before the result is stored again as an integer +value. In any case there is never any loss of precision except when +converting the output pixel to short integer. If any of the images are +not integer then a real internal data path is used in which input and +output image data are converted to real as necessary. +.PP +For each data path the processing proceeds line-by-line. For each line +in the output image data region (ignoring pixels outside the data area +and pixels which are trimmed) the appropriate input data and +calibration data are obtained. The calibration data is determined from +the CCD coordinates of the output image and are not necessarily from +the same image line or columns. The input data is copied to the output +array while applying bad pixel corrections and trimming. The line is +then processed using a specially optimized procedure. This procedure +applies all operations simultaneously for all combinations of +operations. As an example, consider subtracting an overscan bias, +subtracting a zero level, and dividing by a flat field. The basic +kernel of the task, where the bulk of the CPU time is used, is + +.nf +.ft L + do i = 1, n + out[i] = (out[i] - overscan - zero[i]) * flatscale / flat[i] +.ft R +.fi + +Here, \fIn\fR is the number of pixels in the line, \fIoverscan\fR is +the overscan bias value for the line, \fIzero\fR is the zero level data +from the zero level image, \fIflatscale\fR is the mean of the flat +field image, and \fIflat\fR is the flat field data from the flat +field image. Note the operations are not applied sequentially but +in a single statement. This is the most efficient method and there is +no need for intermediate images. +.PP +Though the processing is logically performed line-by-line in the program, +the image I/O from the disk is not done this way. The IRAF virtual +operating system image interface automatically provides multi-line +buffering for maximal I/O efficiency. +.PP +In many image processing systems it has been standard to apply operations +sequentially over an image. This requires producing intermediate images. +Since this is clearly inefficient in terms of I/O it has been the practice +to copy the images into main memory and operate upon them there until +the final image is ready to be saved. This has led to the perception +that in order to be efficient an image processing system \fImust\fR +store images in memory. This is not true and the IRAF CCD reduction +package illustrates this. The CCD processing does not use intermediate +images and does not need to keep the entire image in main memory. +Furthermore, though of lesser importance than I/O, the single statement method +illustrated above is more efficient than multiple passes through the +images even when the images are kept in main memory. Finally, as CCD +detectors increase in size and small, fast, and cheap processors become +common it is a distinct advantage to not require the large amounts of +memory needed to keep entire images in memory. +.PP +There is one area in which use of main memory can improve performance +and \fBccdproc\fR does take advantage of it if desired. The calibration +images usually are the same for many input images. By specifying the +maximum amount of memory available for storing images in memory +the calibration images may be stored in memory up to that amount. +By parameterizing the memory requirement there is no builtin dependence +on large memory! +.PP +After processing the input image the last steps are to log the operations +in the image header using processing keywords and replace the input +image by the output image as described earlier. The CCD coordinates +of the data are recorded in the header, even if not there previously, to +allow further processing on the image after the image has been trimmed. +.NH +Combining Images +.PP +The second important tool in the CCD reduction package is a task to combine +many images into a single, higher quality image. While this may also be +done with more general image processing tools (the IRAF task \fBimsum\fR +for example) the \fBccdred\fR tasks include special CCD dependent features such +as recognizing the image types and using the image header translation +file. Combining images is often done +with calibration images, which are easy to obtain in number, where it +is important to minimize the statistical noise so as to not affect the +object images. Sometimes object images also are combined. +The task is called \fBcombine\fR and there are special versions of +this task called \fBzerocombine, darkcombine\fR, and \fBflatcombine\fR +for the standard calibration images. +.PP +The task takes a list of input images to be combined. As output there +is the combined image, an optional sigma image, and optional log output either +to the terminal, to a log file, or both. A subset or subsets +of the input images may be selected based on the image type and a +subset parameter such as the filter. As with the processing task, +this allows selecting images without having to explicitly list each +image from a large data set. When combining based on a subset parameter +there is an output image, and possibly a sigma image, for each separate subset. +The output image pixel datatype may also be changed during combining; +usually from 16 bit integer input to real output. +The sigma image is the standard deviation of the input images about the +output image. +.PP +Except for summing the images together, +combining images may require correcting for variations between the images +due to differing exposure times, sky background, extinctions, and +positions. Currently, extinction corrections and registration are +not included but scaling and shifting corrections are included. +The scaling corrections may be done by exposure times or by computing +the mode in each image. Additive shifting is also done by computing +the mode in the images. The region of the image in which the mode +is computed can be specified but by default the whole image is used. +A scaling correction is used when the flux level or sensitivity is varying. +The offset correction is used when the sky brightness is varying independently +of the object brightness. If the images are not scaled then special +data paths combine the images more efficiently. +.PP +Except for medianing and summing, the images are combined by averaging. +The average may be weighted by + +.nf +.ft L + weight = (N * scale / mode) ** 2 +.ft R +.fi + +where \fIN\fR is the number of images previously combined (the task +records the number of images combined in the image header), \fIscale\fR +is the relative scale (applied by dividing) from the exposure time or +mode, and \fImode\fR is the background mode estimate used when adding a +variable offset. +.PP +The combining operation is the heart of the task. There are a number +algorithms which may be used as well as applying statistical weights. +The algorithms are used to detect and reject deviant pixels, such as +cosmic rays. +The choice of algorithm depends on the data, the number of images, +and the importance of rejecting cosmic rays. The more complex the +algorithm the more time consuming the operation. +The list below summarizes the algorithms. +Further algorithms may be added in time. + +.IP "Sum - sum the input images" +.br +The input images are combined by summing. Care must be taken +not to exceed the range of the 16 bit integer datatype when summing if the +output datatype is of this type. Summing is the only algorithm in which +scaling and weighting are not used. Also no sigma image is produced. +.IP "Average - average the input images" +.br +The input images are combined by averaging. The images may be scaled +and weighted. There is no pixel rejection. A sigma image is produced +if more than one image is combined. +.IP "Median - median the input images" +.br +The input images are combined by medianing each pixel. Unless the images +are at the same exposure level they should be scaled. The sigma image +is based on all the input images and is only an approximation to the +uncertainty in the median estimates. +.IP "Minreject, maxreject, minmaxreject - reject extreme pixels" +.br +At each pixel the minimum, maximum, or both are excluded from the +average. The images should be scaled and the average may be +weighted. The sigma image requires at least two pixels after rejection +of the extreme values. These are relatively fast algorithms and are +a good choice if there are many images (>15). +.IP "Threshold - reject pixels above and below specified thresholds" +.br +The input images are combined with pixels above and below specified +threshold values (before scaling) excluded. The images may be scaled +and the average weighted. The sigma image also has the rejected +pixels excluded. +.IP "Sigclip - apply a sigma clipping algorithm to each pixel" +.br +The input images are combined by applying a sigma clipping algorithm +at each pixel. The images should be scaled. This only rejects highly +deviant points and so +includes more of the data than the median or minimum and maximum +algorithms. It requires many images (>10-15) to work effectively. +Otherwise the bad pixels bias the sigma significantly. The mean +used to determine the sigmas is based on the "minmaxrej" algorithm +to eliminate the effects of bad pixels on the mean. Only one +iteration is performed and at most one pixel is rejected at each +point in the output image. After the deviant pixels are rejected the final +mean is computed from all the data. The sigma image excludes the +rejected pixels. +.IP "Avsigclip - apply a sigma clipping algorithm to each pixel" +.br +The input images are combined with a variant of the sigma clipping +algorithm which works well with only a few images. The images should +be scaled. For each line the mean is first estimated using the +"minmaxrej" algorithm. The sigmas at each point in the line are scaled +by the square root of the mean, that is a Poisson scaling of the noise +is assumed. These sigmas are averaged to get a line estimate of the +sigma. Then the sigma at each point in the line is estimated by +multiplying the line sigma by the square root of the mean at that point. As +with the sigma clipping algorithm only one iteration is performed and +at most one pixel is rejected at each point. After the deviant pixels +are rejected the file mean is computed from all the data. The sigma +image excludes the rejected pixels. +.RE +.PP +The "avsigclip" algorithm is the best algorithm for rejecting cosmic +rays, especially with a small number of images, but it is also the most +time consuming. With many images (>10-15) it might be advisable to use +one of the other algorithms ("maxreject", "median", "minmaxrej") because +of their greater speed. +.PP +This task also has several design features to make it efficient and +versatile. There are separate data paths for integer data and real +data; as with processing, if all input images and the output image are +of the same datatype then the I/O is done with no internal conversions. +With mixed datatypes the operations are done as real. Even in the +integer path the operations requiring real arithmetic to preserve the +accuracy of the calculation are performed in that mode. There is +effectively no limit to the number of images which may be combined. +Also, the task determines the amount of memory available and buffers +the I/O as much as possible. This is a case where operating on images +from disk rather than in memory is essential. +.NH +Status and Conclusion +.PP +The initial implementation of the IRAF \fBccdred\fR package was +completed in June 1987. It has been in use at the National Optical +Astronomy Observatories since April 1987. The package was not +distributed with Version 2.5 of IRAF (released in August 1987) but is +available as a separate installation upon request. It will be part of +future releases of IRAF. +.PP +At NOAO the CCD reduction package is available at the telescopes as the +data is obtained. This is accomplished by transferring the images from +the data taking computer to a Sun workstation (Sun Microsystems, Inc.) +initially via tape and later by a direct link. There are several +reasons for adopting this architecture. First, the data acquisition +system is well established and is dedicated to its real-time function. +The second computer was phased in without disrupting the essential +operation of the telescopes and if it fails data taking may continue +with data being stored on tape. The role of the second computer is to +provide faster and more powerful reduction and analysis capability not +required in a data acquisition system. In the future it can be more +easily updated to follow the state of the art in small computers. As +CCD detectors get larger the higher processing speeds will be essential +to keep up with the data flow. +.PP +By writing the reduction software in the high level, portable, IRAF +system the users have the capability to process their data from the +basic CCD reductions to a full analysis at the telescope. Furthermore, +the same software is widely available on a variety of computers if +later processing or reprocessing is desired; staff and visitors at NOAO +may also reduce their data at the headquarters facilities. The use of +a high level system was also essential in achieving the design goals; +it would be difficult to duplicate this complex package without +the rich programming environment provided by the IRAF system. +.NH +References +.PP +The following documentation is distributed by the National Optical +Astronomy Observatories, Central Computer Services, P.O. Box 26732, +Tucson, Arizona, 85726. A comprehensive description of the IRAF system +is given in \fIThe IRAF Data Reduction and Analysis System\fR by Doug +Tody (also appearing in \fIProceedings of the SPIE - Instrumentation in +Astronomy VI\fR, Vol. 627, 1986). A general guide to using IRAF is \fIA +User's Introduction to the IRAF Command Language\fR by Peter Shames +and Doug Tody. Both these documents are also part of the IRAF +documentation distributed with the system. +.PP +A somewhat more tutorial description of the \fBccdred\fR package is +\fIA User's Guide to the IRAF CCDRED Package\fR by the author. +Detailed task descriptions and supplementary documentation are +given in the on-line help library and are part of the user's guide. +.NH +Appendix +.PP +The current set of tasks making up the IRAF CCD Reduction Package, +\fBccdred\fR, are summarized below. + +.nf +.ft L + badpiximage - Create a bad pixel mask image from a bad pixel file + ccdgroups - Group CCD images into image lists + ccdhedit - CCD image header editor + ccdlist - List CCD processing information + ccdproc - Process CCD images + combine - Combine CCD images + darkcombine - Combine and process dark count images + flatcombine - Combine and process flat field images + mkfringecor - Make fringe correction images from sky images + mkillumcor - Make flat field illumination correction images + mkillumflat - Make illumination corrected flat fields + mkskycor - Make sky illumination correction images + mkskyflat - Make sky corrected flat field images +setinstrument - Set instrument parameters + zerocombine - Combine and process zero level images +.fi +.ft R diff --git a/noao/imred/ccdred/doc/ccdtypes.hlp b/noao/imred/ccdred/doc/ccdtypes.hlp new file mode 100644 index 00000000..2cec33ea --- /dev/null +++ b/noao/imred/ccdred/doc/ccdtypes.hlp @@ -0,0 +1,124 @@ +.help ccdtypes Jun87 noao.imred.ccdred +.ih +NAME +ccdtypes -- Description of the CCD image types +.ih +CCDTYPES +The following CCD image types may be specified as the value of the parameter +\fIccdtype\fR: + +.nf + "" - (the null string) all image types + object - object images + zero - zero level images such as a bias or preflash + dark - dark count images + flat - flat field images + illum - iillumination images + fringe - fringe correction images + other - other image types defined in the translation file + none - images without an image type parameter + unknown - image types not defined in the translation file +.fi +.ih +DESCRIPTION +The \fBccdred\fR package recognizes certain standard CCD image types +identified in the image header. The tasks may select images of a +particular CCD image type from image lists with the parameter +\fIccdtype\fR and also recognize and take special actions for +calibration images. + +In order to make use of CCD image type information the header keyword +identifying the image type must be specified in the instrument +translation file. This entry has the form + + imagetyp keyword + +where keyword is the image header keyword. This allows the package to +access the image type string. There must also be a translation between +the image type strings and the CCD types as recognized by the package. +This information consists of lines in the instrument translation file +of the form + + header package + +where header is the exact string given in the image header and package +is one of the types recognized by the package. The image header string +can be virtually anything and if it contains blanks it must be +quoted. The package image types are those given above except for +the null string, "none", and "unknown". That is, these types may +be specified as a CCD image type in selecting images but not as a translations +of image type strings. + +There may be more than one image type that maps to the same package +type. In particular other standard CCD image types, such as comparison +spectra, multiple exposure, standard star, etc., should be mapped to +object or other. There may also be more than one type of flat field, i.e. dome +flat, sky flat, and lamp flat. For more on the instrument translation +file see the help for \fBinstruments\fR. +.ih +EXAMPLES +1. The example entries in the instrument translation file are from the 1986 +NOAO CCD image header format produced by the CAMERA format tape writer. + +.nf + imagetyp data-typ + + 'OBJECT (0)' object + 'DARK (1)' dark + 'PROJECTOR FLAT (2)' flat + 'SKY FLAT (3)' other + 'COMPARISON LAMP (4)' other + 'BIAS (5)' zero + 'DOME FLAT (6)' flat +.fi + +The image header keyword describing the image type is "data-typ". +The values of the image type strings in the header contain blanks so they +are quoted. Also the case of the strings is important. Note that there +are two types of flat field images and two types of other images. + +2. One way to check the image types is with the task \fBccdlist\fR. + +.nf + cl> ccdlist *.imh + Zero.imh[504,1][real][zero][1][OT]:FOCUS L98-193 + Flat1.imh[504,1][real][flat][1][OTZ]:dflat 6v+blue 5s + ccd002.imh[504,504][real][unknown][1][OTZF]:FOCUS L98-193 + ccd003.imh[544,512][short][object][1]:L98-193 + ccd004.imh[544,512][short][object][1]:L98-193 + ccd005.imh[544,512][short][object][1]:L98-193 + oldformat.imh[544,512][short][none][1]:M31 V +.fi + +The unknown type has a header image type of "MUL (8)". The old format +image does not have any header type. + +3. To select only images of a particular type: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd003.imh[544,512][short][object][1]:L98-193 + ccd004.imh[544,512][short][object][1]:L98-193 + ccd005.imh[544,512][short][object][1]:L98-193 + cl> ccdlist *.imh ccdtype=unknown + ccd002.imh[504,504][real][unknown][1][OTZF]:FOCUS L98-193 + cl> ccdlist *.imh ccdtype=none + oldformat.imh[544,512][short][none][1]:M31 V +.fi + +4. To process images with \fBccdproc\fR: + +.nf + cl> ccdproc *.imh + cl> ccdproc *.imh ccdtype=object +.fi + +In the first case all the images will be processed (the default value of +\fIccdtype\fR is ""). However, the task recognizes the calibration +images, such as zero level and flat fields, and processes them appropriately. +In the second case only object images are processed and all other images +are ignored (except if needed as a calibration image). +.ih +SEE ALSO +instruments +.endhelp diff --git a/noao/imred/ccdred/doc/combine.hlp b/noao/imred/ccdred/doc/combine.hlp new file mode 100644 index 00000000..474937bf --- /dev/null +++ b/noao/imred/ccdred/doc/combine.hlp @@ -0,0 +1,1146 @@ +.help combine Aug96 noao.imred.ccdred +.ih +NAME +combine -- Combine CCD images using various algorithms +.ih +USAGE +combine input output +.ih +PARAMETERS +.ls input +List of CCD images to combine. Images of a particular CCD image type may be +selected with the parameter \fIccdtype\fR with the remaining images ignored. +.le +.ls output +Output combined image or list of images. If the \fIproject\fR parameter is +no (the typical case for CCD acquisition) then there will be one output +image or, if the \fIsubsets\fR parameter is selected, one +output image per subset. If the images consist of stacks then +the \fIproject\fR option allows combining each input stack into separate +output images as given by the image list. +.le +.ls plfile = "" (optional) +Output pixel list file or list of files. If no name is given or the +list ends prematurely then no file is produced. The pixel list file +is a map of the number of pixels rejected or, equivalently, +the total number of input images minus the number of pixels actually used. +The file name is also added to the output image header under the +keyword BPM. +.le +.ls sigma = "" (optional) +Output sigma image or list of images. If no name is given or the list ends +prematurely then no image is produced. The sigma is standard deviation, +corrected for a finite population, of the input pixel values (excluding +rejected pixels) about the output combined pixel values. +.le + +.ls ccdtype = "" +CCD image type to combine. If specified only input images of the specified +type are combined. See \fBccdtypes\fR for the possible image types. +.le +.ls amps = yes +Combine images by amplifier? If yes then the input images are grouped by +the amplifier parameter and each group combined into a separate output +image. The amplifier identifier is appended to the output image name(s). +See \fBsubsets\fR for more on the amplifier parameter. +.le +.ls subsets = no +Combine images by subset parameter? If yes then the input images are +grouped by subset parameter and each group combined into a separate output +image. The subset identifier is appended to the output image +name(s). See \fBsubsets\fR for more on the subset parameter. +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? THIS OPTION IS NO LONGER SUPPORTED BUT +THE PARAMETER REMAINS FOR NOW FOR BACKWARD COMPATIBILITY. IF SET TO +yes AN ERROR ABORT WILL OCCUR. +.le + +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +offsetting, masking, thresholding, and rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "none" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation performed on the pixels remaining after offsetting, +masking and thresholding. The algorithms are discussed in the +DESCRIPTION section. The rejection choices are: + +.nf + none - No rejection + minmax - Reject the nlow and nhigh pixels + ccdclip - Reject pixels using CCD noise parameters + crreject - Reject only positive pixels using CCD noise parameters + sigclip - Reject pixels using a sigma clipping algorithm + avsigclip - Reject pixels using an averaged sigma clipping algorithm + pclip - Reject pixels using sigma based on percentiles +.fi + +.le +.ls project = no +Project (combine) across the highest dimension of the input images? If +no then all the input images are combined to a single output image. If +yes then the highest dimension elements of each input image are combined to +an output image and optional pixel list and sigma images. Each element of +the highest dimension may have a separate offset but there can only be one +mask image. +.le +.ls outtype = "real" (short|ushort|integer|long|real|double) +Output image pixel datatype. The pixel datatypes are "double", "real", +"long", "integer", unsigned short ("ushort") and "short" with highest +precedence first. If none is specified then the highest precedence +datatype of the input images is used. A mixture of short and unsigned +short images has a highest precedence of integer. +The datatypes may be abbreviated to +a single character. +.le +.ls offsets = "none" (none|wcs|grid|<filename>) +Integer offsets to add to each image axes. The options are: +.ls "none" +No offsets are applied. +.le +.ls "wcs" +The world coordinate system (wcs) in the image is used to derive the +offsets. The nearest integer offset that matches the world coordinate +at the center of the first input image is used. +.le +.ls "grid" +A uniform grid of offsets is specified by a string of the form + +.nf + grid [n1] [s1] [n2] [s2] ... +.fi + +where ni is the number of images in dimension i and si is the step +in dimension i. For example "grid 5 100 5 100" specifies a 5x5 +grid with origins offset by 100 pixels. +.le +.ls <filename> +The offsets are given in the specified file. The file consists +of one line per image with the offsets in each dimension forming the +columns. +.le +.le +.ls masktype = "none" (none|goodvalue|badvalue|goodbits|badbits) +Type of pixel masking to use. If "none" then no pixel masking is done +even if an image has an associated pixel mask. The other choices +are to select the value in the pixel mask to be treated as good +(goodvalue) or bad (badvalue) or the bits (specified as a value) +to be treated as good (goodbits) or bad (badbits). The pixel mask +file name comes from the image header keyword BPM. +Note that when +combining images by projection of the highest dimension only one +pixel mask is applied to all the images. \fBAlso if the number of +input images becomes too large (currently about 115 .imh or 57 .hhh +images) then the images are temporarily stacked and combined by projection +which also means the bad pixel mask from the first image will be used +for all images.\fR +.le +.ls maskvalue = 0 +Mask value used with the \fImasktype\fR parameter. If the mask type +selects good or bad bits the value may be specified using IRAF notation +for decimal, octal, or hexadecimal; i.e 12, 14b, 0cx to select bits 3 +and 4. +.le +.ls blank = 0. +Output value to be used when there are no pixels. +.le + +.ls scale = "none" (none|mode|median|mean|exposure|@<file>|!<keyword>) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, scale +by the exposure time in the image header, scale by the values in a specified +file, or scale by a specified image header keyword. When specified in +a file the scales must be one per line in the order of the input +images. +.le +.ls zero = "none" (none|mode|median|mean|@<file>|!<keyword>) +Additive zero level image shifts to be applied. The choices are none or +shift by the mode, median, or mean of the specified statistics section, +shift by values given in a file, or shift by values given by an image +header keyword. When specified in a file the zero values must be one +per line in the order of the input images. File or keyword zero offset +values do not allow a correction to the weights. +.le +.ls weight = "none" (none|mode|median|mean|exposure|@<file>|!<keyword>) +Weights to be applied during the final averaging. The choices are none, +the mode, median, or mean of the specified statistics section, the exposure +time, values given in a file, or values given by an image header keyword. +When specified in a file the weights must be one per line in the order of +the input images and the only adjustment made by the task is for the number of +images previously combined. In this case the weights should be those +appropriate for the scaled images which would normally be the inverse +of the variance in the scaled image. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling and +weighting. If no section is given then the entire region of the input is +sampled (for efficiency the images are sampled if they are big enough). +When the images are offset relative to each other one can precede the image +section with one of the modifiers "input", "output", "overlap". The first +interprets the section relative to the input image (which is equivalent to +not specifying a modifier), the second interprets the section relative to +the output image, and the last selects the common overlap and any following +section is ignored. +.le + +.ce +Algorithm Parameters +.ls lthreshold = INDEF, hthreshold = INDEF +Low and high thresholds to be applied to the input pixels. This is done +before any scaling, rejection, and combining. If INDEF the thresholds +are not used. +.le +.ls nlow = 1, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +These numbers are converted to fractions of the total number of input images +so that if no rejections have taken place the specified number of pixels +are rejected while if pixels have been rejected by masking, thresholding, +or nonoverlap, then the fraction of the remaining pixels, truncated +to an integer, is used. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject when +using the clipping algorithms (ccdclip, crreject, sigclip, avsigclip, or +pclip). When given as a positive value this is the minimum number to +keep. When given as a negative value the absolute value is the maximum +number to reject. If there are fewer pixels at some point due to +offsetting, thresholding, or masking then if the number to keep (positive +nkeep) is greater than the number of pixels no pixels will be rejected and +if the number to reject is given (negative nkeep) then up to that number +may be rejected. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. The noise model for a pixel is: + +.nf + variance in DN = (rdnoise/gain)^2 + DN/gain + (snoise*DN)^2 + variance in e- = (rdnoise)^2 + (gain*DN) + (snoise*(gain*DN))^2 + = rdnoise^2 + Ne + (snoise * Ne)^2 +.fi + +where DN is the data number and Ne is the number of electrons. Sensitivity +noise typically comes from noise introduced during flat fielding. +.le +.ls sigscale = 0.1 (ccdclip, crreject, sigclip, avsigclip) +This parameter determines when poisson corrections are made to the +computation of a sigma for images with different scale factors. If all +relative scales are within this value of unity and all relative zero level +offsets are within this fraction of the mean then no correction is made. +The idea is that if the images are all similarly though not identically +scaled, the extra computations involved in making poisson corrections for +variations in the sigmas can be skipped. A value of zero will apply the +corrections except in the case of equal images and a large value can be +used if the sigmas of pixels in the images are independent of scale and +zero level. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See the DESCRIPTION section for further details. +.le +.ls grow = 0 +Number of pixels to either side of a rejected pixel along image lines +to also be rejected. This applies only to pixels rejected by one of +the rejection algorithms and not the masked or threshold rejected pixels. +.le + +PACKAGE PARAMETERS + +The package parameters are used to specify verbose and log output and the +instrument and header definitions. +.ih +DESCRIPTION +A set of CCD images are combined by weighted averaging or medianing. Pixels +may be rejected from the combining by using pixel masks, threshold levels, +and rejection algorithms. The images may be scaled multiplicatively or +additively based on image statistics, image header keywords, or text files +before rejection. The images may be combined with integer pixel coordinate +offsets to produce an image bigger than any of the input images. +This task is a variant of the \fBimages.imcombine\fR task specialized +for CCD images. + +The input images to be combined are specified by a list. A subset or +subsets of the input list may be selected using the parameters +\fIccdtype\fR and \fIsubsets\fR. The \fIccdtype\fR parameter +selects only images of a specified standard CCD image type. +The \fIsubsets\fR parameter breaks up the input +list into sublists of common subset parameter (filter, grating, etc.). For +more information see \fBccdtypes\fR and \fBsubsets\fR. This selection +process is useful with wildcard templates to combine, for example, the flat +field images for each filter in one step (see \fBflatcombine\fR). When +subsets of the input list are used the output image and optional pixel file +and sigma image are given by root names with an amplifier and subset +identifier appended by the task. + +If the \fBproject\fR parameter is yes then the highest dimension elements +of each input image are combined to make an output image of one lower +dimension. There is no limit to the number of elements combined in this +case. This case is If the \fBproject\fR is no then the entire input list +is combined to form a single output image per subset. In this case the +images must all have the same dimensionality but they may have different +sizes. There is a software limit of approximately 100 images in this +case. + +The output image header is a copy of the first image in the combined set. +In addition, the number of images combined is recorded under the keyword +NCOMBINE, the exposure time is updated as the weighted average of the input +exposure times, and any pixel list file created is recorded under the +keyword BPM. The output pixel type is set by the parameter \fIouttype\fR. +If left blank then the input datatype of highest precision is used. +A mixture of short and unsigned short images has a highest precision of +integer. + +In addition to one or more output combined images there may also be a pixel +list image containing the number of pixels rejected at each point in the +output image, an image containing the sigmas of the pixels combined about +the final output combined pixels, and a log file. The pixel list image is +in the compact pixel list format which can be used as an image in other +programs. The sigma computation is the standard deviation corrected for a +finite population (the n/(n-1) factor) including weights if a weighted +average is used. + +Other input/output parameters are \fIdelete\fR and \fIclobber\fR. The +\fIdelete\fR parameter may be set to "yes" to delete the input images +used in producing an output image after it has been created. This is +useful for minimizing disk space, particularly with large +sets of calibration images needed to achieve high statistical accuracy +in the final calibration image. The \fBclobber\fR parameter allows +the output image names to be existing images which are overwritten (at +the end of the operation). + +An outline of the steps taken by the program is given below and the +following sections elaborate on the steps. + +.nf +o Set the input image offsets and the final output image size. +o Set the input image scales and weights +o Write the log file output +.fi + +For each output image line: + +.nf +o Get input image lines that overlap the output image line +o Reject masked pixels +o Reject pixels outside the threshold limits +o Reject pixels using the specified algorithm +o Reject neighboring pixels along each line +o Combine remaining pixels using the weighted average or median +o Compute sigmas of remaining pixels about the combined values +o Write the output image line, rejected pixel list, and sigmas +.fi + + +OFFSETS + +The images to be combined need not be of the same size or overlap. They +do have to have the same dimensionality which will also be the dimensionality +of the output image. Any dimensional images supported by IRAF may be +used. Note that if the \fIproject\fR flag is yes then the input images +are the elements of the highest dimension; for example the planes of a +three dimensional image. + +The overlap of the images is determined by a set of integer pixel offsets +with an offset for each dimension of each input image. For example +offsets of 0, 10, and 20 in the first dimension of three images will +result in combining the three images with only the first image in the +first 10 colums, the first two images in the next 10 columns and +all three images starting in the 31st column. At the 31st output column +the 31st column of the first image will be combined with the 21st column +of the second image and the 1st column of the third image. + +The output image size is set by the maximum extent in each dimension +of any input image after applying the offsets. In the above example if +all the images have 100 columns then the output image will have 130 +columns corresponding to the 30 column offset in the third image. + +The input image offsets are set using the \fIoffset\fR parameter. There +are four ways to specify the offsets. If the word "none" or the empty +string "" are used then all offsets will be zero and all pixels with the +same coordinates will be combined. The output image size will be equal to +the biggest dimensions of the input images. + +If "wcs" offsets are specified then the world coordinate systems (wcs) +in the image headers are used to derived the offsets. The world coordinate +at the center of the first input image is evaluated. Then integer pixel +offsets are determined for each image to bring the same world coordinate +to the same point. Note the following caveats. The world coordinate +systems must be of the same type, orientation, and scale and only the +nearest integer shift is used. + +If the input images have offsets in a regular grid or one wants to make +an output image in which the input images are "mosaiced" together in +a grid then the special offset string beginning with the word "grid" +is used. The format is + +.nf + grid [n1] [s1] [n2] [s2] ... +.fi + +where ni is the number of images in dimension i and si is the step in +dimension i. For example "grid 5 100 5 100" specifies a 5x5 grid with +origins offset by 100 pixels. Note that one must insure that the input +images are specified in the correct order. This may best be accomplished +using a "@" list. One useful application of the grid is to make a +nonoverlapping mosaic of a number of images for display purposes. Suppose +there are 16 images which are 100x100. The offset string "grid 4 101 4 +101" will produce a mosaic with a one pixel border having the value set +by \fIblank\fR parameter between the images. + +The offsets may be defined in a file by specifying the file name +in the \fIoffset\fR parameter. (Note that the special file name STDIN +may be used to type in the values terminated by the end-of-file +character). The file consists of a line for each input image. The lines +must be in the same order as the input images and so an "@" list may +be useful. The lines consist of whitespace separated offsets one for +each dimension of the images. In the first example cited above the +offset file might contain: + +.nf + 0 0 + 10 0 + 20 0 +.fi + +where we assume the second dimension has zero offsets. + +The offsets need not have zero for one of the images. The offsets may +include negative values or refer to some arbitrary common point. +When the offsets are read by the program it will find the minimum +value in each dimension and subtract it from all the other offsets +in that dimension. The above example could also be specified as: + +.nf + 225 15 + 235 15 + 245 15 +.fi + +There may be cases where one doesn't want the minimum offsets reset +to zero. If all the offsets are positive and the comment "# Absolute" +appears in the offset file then the images will be combined with +blank values between the first output pixel and the first overlapping +input pixel. Continuing with the above example, the file + +.nf + # Absolute + 10 10 + 20 10 + 30 10 +.fi + +will have the first pixel of the first image in the 11th pixel of the +output image. Note that there is no way to "pad" the other side of +the output image. + + +SCALES AND WEIGHTS + +In order to combine images with rejection of pixels based on deviations +from some average or median they must be scaled to a common level. There +are two types of scaling available, a multiplicative intensity scale and an +additive zero point shift. The intensity scaling is defined by the +\fIscale\fR parameter and the zero point shift by the \fIzero\fR +parameter. These parameters may take the values "none" for no scaling, +"mode", "median", or "mean" to scale by statistics of the image pixels, +"exposure" (for intensity scaling only) to scale by the exposure time +keyword in the image header, any other image header keyword specified by +the keyword name prefixed by the character '!', and the name of a file +containing the scale factors for the input image prefixed by the +character '@'. + +Examples of the possible parameter values are shown below where +"myval" is the name of an image header keyword and "scales.dat" is +a text file containing a list of scale factors. + +.nf + scale = none No scaling + zero = mean Intensity offset by the mean + scale = exposure Scale by the exposure time + zero = !myval Intensity offset by an image keyword + scale = @scales.dat Scales specified in a file +.fi + +The image statistics factors are computed by sampling a uniform grid +of points with the smallest grid step that yields less than 10000 +pixels; sampling is used to reduce the time need to compute the statistics. +If one wants to restrict the sampling to a region of the image the +\fIstatsec\fR parameter is used. This parameter has the following +syntax: + +.nf + [input|output|overlap] [image section] +.fi + +The initial modifier defaults to "input" if absent. The modifiers are useful +if the input images have offsets. In that case "input" specifies +that the image section refers to each input image, "output" specifies +that the image section refers to the output image coordinates, and +"overlap" specifies the mutually overlapping region of the input images. +In the latter case an image section is ignored. + +The statistics are as indicated by their names. In particular, the +mode is a true mode using a bin size which is a fraction of the +range of the pixels and is not based on a relationship between the +mode, median, and mean. Also masked pixels are excluded from the +computations as well as during the rejection and combining operations. + +The "exposure" option in the intensity scaling uses the exposure time +from the image header. If one wants to use a nonexposure time image +header keyword the !<keyword> syntax is available. + +If both an intensity scaling and zero point shift are selected the +multiplicative scaling is done first. Use of both makes sense +if the intensity scaling is the exposure time to correct for +different exposure times and then the zero point shift allows for +sky brightness changes. + +The image statistics and scale factors are recorded in the log file +unless they are all equal, which is equivalent to no scaling. The +intensity scale factors are normalized to a unit mean and the zero +point shifts are adjust to a zero mean. When the factors are specified +in an @file or by a keyword they are not normalized. + +Scaling affects not only the mean values between images but also the +relative pixel uncertainties. For example scaling an image by a +factor of 0.5 will reduce the effective noise sigma of the image +at each pixel by the square root of 0.5. Changes in the zero +point also changes the noise sigma if the image noise characteristics +are Poissonian. In the various rejection algorithms based on +identifying a noise sigma and clipping large deviations relative to +the scaled median or mean, one may need to account for the scaling induced +changes in the image noise characteristics. + +In those algorithms it is possible to eliminate the "sigma correction" +while still using scaling. The reasons this might be desirable are 1) if +the scalings are similar the corrections in computing the mean or median +are important but the sigma corrections may not be important and 2) the +image statistics may not be Poissonian, either inherently or because the +images have been processed in some way that changes the statistics. In the +first case because computing square roots and making corrections to every +pixel during the iterative rejection operation may be a significant +computational speed limit the parameter \fIsigscale\fR selects how +dissimilar the scalings must be to require the sigma corrections. This +parameter is a fractional deviation which, since the scale factors are +normalized to unity, is the actual minimum deviation in the scale factors. +For the zero point shifts the shifts are normalized by the mean shift +before adjusting the shifts to a zero mean. To always use sigma scaling +corrections the parameter is set to zero and to eliminate the correction in +all cases it is set to a very large number. + +If the final combining operation is "average" then the images may be +weighted during the averaging. The weights are specified in the +same way as the scale factors. In addition +the NCOMBINE keyword, if present, will be used in the weights. +The weights, scaled to a unit sum, are printed in the log output. + +The weights are only used for the final weighted average and sigma image +output. They are not used to form averages in the various rejection +algorithms. For weights in the case of no scaling or only multiplicative +scaling the weights are used as given or determined so that images with +lower signal levels will have lower weights. However, for cases in which +zero level scaling is used and the zero levels are determined from image +statistics (not from an input file or keyword) the weights are computed +from the initial weights (the exposure time, image statistics, or input +values) using the formula: + +.nf + weight_final = weight_initial / (scale * sky) +.fi + +where the sky values are those from the image statistics before conversion +to zero level shifts and adjustment to zero mean over all images. The +reasoning is that if the zero level is high the sky brightness is high and +so the S/N is lower and the weight should be lower. If any sky value +determined from the image statistics comes out to be negative a warning is +given and the none of the weight are adjusted for sky levels. + +The weights are not adjusted when the zero offsets are input from a file +or keyword since these values do not imply the actual image sky value. +In this case if one wants to account for different sky statistics +in the weights the user must specify the weights in a file taking +explicit account of changes in the weights due to different sky +statistics. + + +PIXEL MASKS + +A pixel mask is a type of IRAF file having the extension ".pl" which +identifies an integer value with each pixel of the images to which it is +applied. The integer values may denote regions, a weight, a good or bad +flag, or some other type of integer or integer bit flag. In the common +case where many values are the same this file is compacted to be small and +efficient to use. It is also most compact and efficient if the majority of +the pixels have a zero mask value so frequently zero is the value for good +pixels. Note that these files, while not stored as a strict pixel array, +may be treated as images in programs. This means they may be created by +programs such as \fBmkpattern\fR, edited by \fBimedit\fR, examined by +\fBimexamine\fR, operated upon by \fBimarith\fR, graphed by \fBimplot\fR, +and displayed by \fBdisplay\fR. + +At the time of introducing this task, generic tools for creating +pixel masks have yet to be written. There are two ways to create a +mask in V2.10. First if a regular integer image can be created +then it can be converted to pixel list format with \fBimcopy\fR: + +.nf + cl> imcopy template plfile.pl +.fi + +by specifically using the .pl extension on output. Other programs that +can create integer images (such \fBmkpattern\fR or \fBccdred.badpiximage\fR) +can create the pixel list file directly by simply using the ".pl" +extension in the output image name. + +To use pixel masks with \fBcombine\fR one must associate a pixel +mask file with an image by entering the pixel list file name in the +image header under the keyword BPM (bad pixel mask). This can be +done with \fBhedit\fR. Note that the same pixel mask may be associated +with more than one image as might be the case if the mask represents +defects in the detector used to obtain the images. + +If a pixel mask is associated with an image the mask is used when the +\fImasktype\fR parameter is set to a value other than "none". Note that +when it is set to "none" mask information is not used even if it exists for +the image. The values of \fImasktype\fR which apply masks are "goodvalue", +"badvalue", "goodbits", and "badbits". They are used in conjunction with +the \fImaskvalue\fR parameter. When the mask type is "goodvalue" the +pixels with mask values matching the specified value are included in +combining and all others are rejected. Similarly, for a mask type of +"badvalue" the pixels with mask values matching the specified value are +rejected and all others are accepted. The bit types are useful for +selecting a combination of attributes in a mask consisting of bit flags. +The mask value is still an integer but is interpreted by bitwise comparison +with the values in the mask file. + +If a mask operation is specified and an image has no mask image associated +with it then the mask values are taken as all zeros. In those cases be +careful that zero is an accepted value otherwise the entire image will be +rejected. + +In the case of combining the higher dimensions of an image into a +lower dimensional image, the "project" option, the same pixel mask +is applied to all of the data being combined; i.e. the same 2D +pixel mask is applied to every plane of a 3D image. This is because +a higher dimensional image is treated as a collection of lower +dimensional images having the same header and hence the same +bad pixel mask. It would be tempting to use a bad pixel mask with +the same dimension as the image being projected but this is not +currently how the task works. + +When the number of input images exceeds the maximum number of open files +allowed by IRAF (currently about 115 .imh or 57 .hhh images) the input +images are stacked and combined with the project option. \fBThis means +that the bad pixel mask from the first input image will be applied to all +the images.\fR + + +THRESHOLD REJECTION + +In addition to rejecting masked pixels, pixels in the unscaled input +images which are below or above the thresholds given by the parameters +\fIlthreshold\fR and \fIhthreshold\fR are rejected. Values of INDEF +mean that no threshold value is applied. Threshold rejection may be used +to exclude very bad pixel values or as an alternative way of masking +images. In the latter case one can use a task like \fBimedit\fR +or \fBimreplace\fR to set parts of the images to be excluded to some +very low or high magic value. + + +REJECTION ALGORITHMS + +The \fIreject\fR parameter selects a type of rejection operation to +be applied to pixels not masked or thresholded. If no rejection +operation is desired the value "none" is specified. + +MINMAX +.in 4 +A specified fraction of the highest and lowest pixels are rejected. +The fraction is specified as the number of high and low pixels, the +\fInhigh\fR and \fInlow\fR parameters, when data from all the input images +are used. If pixels have been rejected by offseting, masking, or +thresholding then a matching fraction of the remaining pixels, truncated +to an integer, are used. Thus, + +.nf + nl = n * nlow/nimages + 0.001 + nh = n * nhigh/nimages + 0.001 +.fi + +where n is the number of pixels surviving offseting, masking, and +thresholding, nimages is the number of input images, nlow and nhigh +are task parameters and nl and nh are the final number of low and +high pixels rejected by the algorithm. The factor of 0.001 is to +adjust for rounding of the ratio. + +As an example with 10 input images and specifying one low and two high +pixels to be rejected the fractions to be rejected are nlow=0.1 and nhigh=0.2 +and the number rejected as a function of n is: + +.nf + n 0 1 2 3 4 5 6 7 8 9 10 + nl 0 0 0 0 0 0 0 0 0 0 1 + nh 0 0 0 0 0 1 1 1 1 1 2 +.fi + +.in -4 +CCDCLIP +.in 4 +If the images are obtained using a CCD with known read out noise, gain, and +sensitivity noise parameters and they have been processed to preserve the +relation between data values and photons or electrons then the noise +characteristics of the images are well defined. In this model the sigma in +data values at a pixel with true value <I>, as approximated by the median +or average with the lowest and highest value excluded, is given by: + +.nf + sigma = ((rn / g) ** 2 + <I> / g + (s * <I>) ** 2) ** 1/2 +.fi + +where rn is the read out noise in electrons, g is the gain in +electrons per data value, s is a sensitivity noise given as a fraction, +and ** is the exponentiation operator. Often the sensitivity noise, +due to uncertainties in the pixel sensitivities (for example from the +flat field), is not known in which case a value of zero can be used. +See the task \fBstsdas.wfpc.noisemodel\fR for a way to determine +these vaues (though that task expresses the read out noise in data +numbers and the sensitivity noise parameter as a percentage). + +The read out noise is specified by the \fIrdnoise\fR parameter. The value +may be a numeric value to be applied to all the input images or a image +header keyword containing the value for each image. Similarly, the +parameter \fIgain\fR specifies the gain as either a value or image header +keyword and the parameter \fIsnoise\fR specifies the sensitivity +noise parameter as either a value or image header keyword. + +The algorithm operates on each output pixel independently. It starts by +taking the median or unweighted average (excluding the minimum and maximum) +of the unrejected pixels provided there are at least two input pixels. The +expected sigma is computed from the CCD noise parameters and pixels more +that \fIlsigma\fR times this sigma below or \fIhsigma\fR times this sigma +above the median or average are rejected. The process is then iterated +until no further pixels are rejected. If the average is used as the +estimator of the true value then after the first round of rejections the +highest and lowest values are no longer excluded. Note that it is possible +to reject all pixels if the average is used and is sufficiently skewed by +bad pixels such as cosmic rays. + +If there are different CCD noise parameters for the input images +(as might occur using the image header keyword specification) then +the sigmas are computed for each pixel from each image using the +same estimated true value. + +If the images are scaled and shifted and the \fIsigscale\fR threshold +is exceedd then a sigma is computed for each pixel based on the +image scale parameters; i.e. the median or average is scaled to that of the +original image before computing the sigma and residuals. + +After rejection the number of retained pixels is checked against the +\fInkeep\fR parameter. If there are fewer pixels retained than specified +by this parameter the pixels with the smallest residuals in absolute +value are added back. If there is more than one pixel with the same +absolute residual (for example the two pixels about an average +or median of two will have the same residuals) they are all added +back even if this means more than \fInkeep\fR pixels are retained. +Note that the \fInkeep\fR parameter only applies to the pixels used +by the clipping rejection algorithm and does not apply to threshold +or bad pixel mask rejection. + +This is the best clipping algorithm to use if the CCD noise parameters are +adequately known. The parameters affecting this algorithm are \fIreject\fR +to select this algorithm, \fImclip\fR to select the median or average for +the center of the clipping, \fInkeep\fR to limit the number of pixels +rejected, the CCD noise parameters \fIrdnoise, gain\fR and \fIsnoise\fR, +\fIlsigma\fR and \fIhsigma\fR to select the clipping thresholds, +and \fIsigscale\fR to set the threshold for making corrections to the sigma +calculation for different image scale factors. + +.in -4 +CRREJECT +.in 4 +This algorithm is identical to "ccdclip" except that only pixels above +the average are rejected based on the \fIhsigma\fR parameter. This +is appropriate for rejecting cosmic ray events and works even with +two images. + +.in -4 +SIGCLIP +.in 4 +The sigma clipping algorithm computes at each output pixel the median or +average excluding the high and low values and the sigma about this +estimate. There must be at least three input pixels, though for this method +to work well there should be at least 10 pixels. Values deviating by more +than the specified sigma threshold factors are rejected. These steps are +repeated, except that after the first time the average includes all values, +until no further pixels are rejected or there are fewer than three pixels. + +After rejection the number of retained pixels is checked against the +\fInkeep\fR parameter. If there are fewer pixels retained than specified +by this parameter the pixels with the smallest residuals in absolute +value are added back. If there is more than one pixel with the same +absolute residual (for example the two pixels about an average +or median of two will have the same residuals) they are all added +back even if this means more than \fInkeep\fR pixels are retained. +Note that the \fInkeep\fR parameter only applies to the pixels used +by the clipping rejection algorithm and does not apply to threshold +or bad pixel mask rejection. + +The parameters affecting this algorithm are \fIreject\fR to select +this algorithm, \fImclip\fR to select the median or average for the +center of the clipping, \fInkeep\fR to limit the number of pixels +rejected, \fIlsigma\fR and \fIhsigma\fR to select the +clipping thresholds, and \fIsigscale\fR to set the threshold for +making corrections to the sigma calculation for different image scale +factors. + +.in -4 +AVSIGCLIP +.in 4 +The averaged sigma clipping algorithm assumes that the sigma about the +median or mean (average excluding the low and high values) is proportional +to the square root of the median or mean at each point. This is +described by the equation: + +.nf + sigma(column,line) = sqrt (gain(line) * signal(column,line)) +.fi + +where the \fIestimated\fR signal is the mean or median (hopefully excluding +any bad pixels) and the gain is the \fIestimated\fR proportionality +constant having units of photons/data number. + +This noise model is valid for images whose values are proportional to the +number of photons recorded. In effect this algorithm estimates a +detector gain for each line with no read out noise component when +information about the detector noise parameters are not known or +available. The gain proportionality factor is computed +independently for each output line by averaging the square of the residuals +(at points having three or more input values) scaled by the median or +mean. In theory the proportionality should be the same for all rows but +because of the estimating process will vary somewhat. + +Once the proportionality factor is determined, deviant pixels exceeding the +specified thresholds are rejected at each point by estimating the sigma +from the median or mean. If any values are rejected the median or mean +(this time not excluding the extreme values) is recomputed and further +values rejected. This is repeated until there are no further pixels +rejected or the number of remaining input values falls below three. Note +that the proportionality factor is not recomputed after rejections. + +If the images are scaled differently and the sigma scaling correction +threshold is exceedd then a correction is made in the sigma +calculations for these differences, again under the assumption that +the noise in an image scales as the square root of the mean intensity. + +After rejection the number of retained pixels is checked against the +\fInkeep\fR parameter. If there are fewer pixels retained than specified +by this parameter the pixels with the smallest residuals in absolute +value are added back. If there is more than one pixel with the same +absolute residual (for example the two pixels about an average +or median of two will have the same residuals) they are all added +back even if this means more than \fInkeep\fR pixels are retained. +Note that the \fInkeep\fR parameter only applies to the pixels used +by the clipping rejection algorithm and does not apply to threshold +or bad pixel mask rejection. + +This algorithm works well for even a few input images. It works better if +the median is used though this is slower than using the average. Note that +if the images have a known read out noise and gain (the proportionality +factor above) then the "ccdclip" algorithm is superior. The two algorithms +are related in that the average sigma proportionality factor is an estimate +of the gain. + +The parameters affecting this algorithm are \fIreject\fR to select +this algorithm, \fImclip\fR to select the median or average for the +center of the clipping, \fInkeep\fR to limit the number of pixels +rejected, \fIlsigma\fR and \fIhsigma\fR to select the +clipping thresholds, and \fIsigscale\fR to set the threshold for +making corrections to the sigma calculation for different image scale +factors. + +.in -4 +PCLIP +.in 4 +The percentile clipping algorithm is similar to sigma clipping using the +median as the center of the distribution except that, instead of computing +the sigma of the pixels from the CCD noise parameters or from the data +values, the width of the distribution is characterized by the difference +between the median value and a specified "percentile" pixel value. This +width is then multipled by the scale factors \fIlsigma\fR and \fIhsigma\fR +to define the clipping thresholds above and below the median. The clipping +is not iterated. + +The pixel values at each output point are ordered in magnitude and the +median is determined. In the case of an even number of pixels the average +of the two middle values is used as the median value and the lower or upper +of the two is the median pixel when counting from the median pixel to +selecting the percentile pixel. The parameter \fIpclip\fR selects the +percentile pixel as the number (if the absolute value is greater +than unity) or fraction of the pixels from the median in the ordered set. +The direction of the percentile pixel from the median is set by the sign of +the \fIpclip\fR parameter with a negative value signifying pixels with +values less than the median. Fractional values are internally converted to +the appropriate number of pixels for the number of input images. A minimum +of one pixel and a maximum corresponding to the extreme pixels from the +median are enforced. The value used is reported in the log output. Note +that the same percentile pixel is used even if pixels have been rejected by +offseting, masking, or thresholding; for example, if the 3nd pixel below +the median is specified then the 3rd pixel will be used whether there are +10 pixels or 5 pixels remaining after the preliminary steps. + +Some examples help clarify the definition of the percentile pixel. In the +examples assume 10 pixels. The median is then the average of the +5th and 6th pixels. A \fIpclip\fR value of 2 selects the 2nd pixel +above the median (6th) pixel which is the 8th pixel. A \fIpclip\fR +value of -0.5 selects the point halfway between the median and the +lowest pixel. In this case there are 4 pixels below the median, +half of that is 2 pixels which makes the percentile pixel the 3rd pixel. + +The percentile clipping algorithm is most useful for clipping small +excursions, such as the wings of bright objects when combining +disregistered observations for a sky flat field, that are missed when using +the pixel values to compute a sigma. It is not as powerful, however, as +using the CCD noise parameters (provided they are accurately known) to clip +about the median. + +The parameters affecting this algorithm are \fIreject\fR to select this +algorithm, \fIpclip\fR to select the percentile pixel, \fInkeep\fR to limit +the number of pixels rejected, and \fIlsigma\fR and \fIhsigma\fR to select +the clipping thresholds. + +.in -4 +GROW REJECTION + +Neighbors of pixels rejected by the rejection algorithms along image lines +may also be rejected. The number of neighbors to be rejected on either +side is specified by the \fIgrow\fR parameter. The rejection only +applies to neighbors along each image line. This is because the +task operates independently on each image line and does not have the +ability to go back to previous lines or maintain a list of rejected +pixels to later lines. + +This rejection step is also checked against the \fInkeep\fR parameter +and only as many pixels as would not violate this parameter are +rejected. Unlike it's application in the rejection algorithms at +this stage there is no checking on the magnitude of the residuals +and the pixels retained which would otherwise be rejected are randomly +selected. + + +COMBINING + +After all the steps of offsetting the input images, masking pixels, +threshold rejection, scaling, and applying a rejection algorithms the +remaining pixels are combined and output. The pixels may be combined +by computing the median or by computing a weighted average. + + +SIGMA OUTPUT + +In addition to the combined image and optional sigma image may be +produced. The sigma computed is the standard deviation, corrected for a +finite population by a factor of n/(n-1), of the unrejected input pixel +values about the output combined pixel values. +.ih +EXAMPLES +1. To average and median images without any other features: + +.nf + cl> combine obj* avg combine=average reject=none + cl> combine obj* med combine=median reject=none +.fi + +2. To reject cosmic rays: + +.nf + cl> combine obs1,obs2 Obs reject=crreject rdnoise=5.1, gain=4.3 +.fi + +3. To make a grid for display purposes with 21 64x64 images: + +.nf + cl> combine @list grid offset="grid 5 65 5 65" +.fi + +4. To apply a mask image with good pixels marked with a zero value and + bad pixels marked with a value of one: + +.nf + cl> hedit ims* bpm badpix.pl add+ ver- + cl> combine ims* final combine=median masktype=goodval +.fi + +5. To scale image by the exposure time and then adjust for varying + sky brightness and make a weighted average: + +.nf + cl> combine obj* avsig combine=average reject=avsig \ + >>> scale=exp zero=mode weight=exp expname=exptime +.fi +.ih +TIME REQUIREMENTS +The following times were obtain with a Sun 4/470. The tests combine +1000x200 images consisting of Poisson noise and cosmic rays generated +with the \fBartdata\fR package. The times, especially the total time, +are approximate and depend on user loads. + +.nf +IMAGES: Number of images (1000x200) and datatype (R=real, S=short) +COMBINE: Combine option +REJECT: Rejection option with grow = 0 + minmax: nlow = 1, nhigh = 1 + ccdclip: lsigma = 3., hsigma = 3, sigscale = 0. + sigclip: lsigma = 3., hsigma = 3, sigscale = 0. + avsigclip: lsigma = 3., hsigma = 3, sigscale = 0. + pclip: lsigma = 3., hsigma = 3, pclip = -0.5 + /a: mclip = no (clip about the average) + /m: mclip = yes (clip about the median) +O M T S: Features used (Y=yes, N=no) +O: offset = "grid 5 10 2 10" +M: masktype = goodval, maskval = 0 + Pixel mask has 2 bad lines and 20 bad columns +T: lthreshold = INDEF, hthreshold = 1100. +S: scale = mode, zero = none, weight = mode +TIME: cpu time in seconds, total time in minutes and seconds + + +IMAGES COMBINE REJECT O M T S TIME + + 10R average none N N N N 1.3 0:08 + 10R average minmax N N N N 4.3 0:10 + 10R average pclip N N N N 17.9 0:32 + 10R average ccdclip/a N N N N 11.6 0:21 + 10R average crreject/a N N N N 11.4 0:21 + 10R average sigclip/a N N N N 13.6 0:29 + 10R average avsigclip/a N N N N 15.9 0:35 + 10R average ccdclip/m N N N N 16.9 0:32 + 10R average crreject/m N N N N 17.0 0:28 + 10R average sigclip/m N N N N 19.6 0:42 + 10R average avsigclip/m N N N N 20.6 0:43 + + 10R median none N N N N 6.8 0:17 + 10R median minmax N N N N 7.8 0:15 + 10R median pclip N N N N 16.9 1:00 + 10R median ccdclip/a N N N N 18.0 0:34 + 10R median crreject/a N N N N 17.7 0:30 + 10R median sigclip/a N N N N 21.1 1:13 + 10R median avsigclip/a N N N N 23.1 0:41 + 10R median ccdclip/m N N N N 16.1 0:27 + 10R median crreject/m N N N N 16.0 0:27 + 10R median sigclip/m N N N N 18.1 0:29 + 10R median avsigclip/m N N N N 19.6 0:32 + + 10R average none N N N Y 6.1 0:36 + 10R median none N N N Y 10.4 0:49 + 10R median pclip N N N Y 20.4 1:10 + 10R median ccdclip/m N N N Y 19.5 0:36 + 10R median avsigclip/m N N N Y 23.0 1:06 + + 10R average none N Y N N 3.5 0:12 + 10R median none N Y N N 8.9 0:21 + 10R median pclip N Y N N 19.9 0:45 + 10R median ccdclip/m N Y N N 18.0 0:44 + 10R median avsigclip/m N Y N N 20.9 0:28 + + 10R average none Y N N N 4.3 0:13 + 10R median none Y N N N 9.6 0:21 + 10R median pclip Y N N N 21.8 0:54 + 10R median ccdclip/m Y N N N 19.3 0:44 + 10R median avsigclip/m Y N N N 22.8 0:51 + + 10R average none Y Y Y Y 10.8 0:22 + 10R median none Y Y Y Y 16.1 0:28 + 10R median pclip Y Y Y Y 27.4 0:42 + 10R median ccdclip/m Y Y Y Y 25.5 0:39 + 10R median avsigclip/m Y Y Y Y 28.9 0:44 + + 10S average none N N N N 2.2 0:06 + 10S average minmax N N N N 4.6 0:12 + 10S average pclip N N N N 18.1 0:33 +.fi +.ih +REVISIONS +.ls COMBINE V2.11 +The limit of the number of images that may be combined has been removed. +If the number of images exceeds the maximum number of open images permitted +then the images are stacked in a single temporary image and then combined +with the project option. Note that this will double the amount of +diskspace temporarily. There is also a limitation in this case that the +bad pixel mask from the first image in the list will be applied to all the +images. + +Integer offsets may be determined from the image world coordinate system. +.le +.ls COMBINE V2.10.3 +The output pixel datatype parameter, \fIouttype\fR was previously ignored +and the package \fIpixeltype\fR was used. The task output pixel type +parameter is now used. + +The factors specified by an @file or keyword are not normalized. +.le +.ls COMBINE V2.10.2 +The weighting was changed from using the square root of the exposure time +or image statistics to using the values directly. This corresponds +to variance weighting. Other options for specifying the scaling and +weighting factors were added; namely from a file or from a different +image header keyword. The \fInkeep\fR parameter was added to allow +controlling the maximum number of pixels to be rejected by the clipping +algorithms. The \fIsnoise\fR parameter was added to include a sensitivity +or scale noise component to the noise model. Errors will now delete +the output images. +.le +.ls COMBINE V2.10 +This task was greatly revised to provide many new features. These features +are: + +.nf + o Bad pixel masks + o Combining offset and different size images + o Blank value for missing data + o Combining across the highest dimension (the project option) + o Separating threshold rejection, the rejection algorithms, + and the final combining statistic + o New CCDCLIP, CRREJECT, and PCLIP algorithms + o Rejection now may reject more than one pixel per output pixel + o Choice of a central median or average for clipping + o Choice of final combining operation + o Simultaneous multiplicative and zero point scaling +.fi +.le +.ih +LIMITATIONS +Though the previous limit on the number of images that can be combined +was removed in V2.11 the method has the limitation that only a single +bad pixel mask will be used for all images. +.ih +SEE ALSO +image.imcombine, instruments, ccdtypes, icfit, ccdred, guide, darkcombine, +flatcombine, zerocombine, onedspec.scombine wfpc.noisemodel +.endhelp diff --git a/noao/imred/ccdred/doc/contents.ms b/noao/imred/ccdred/doc/contents.ms new file mode 100644 index 00000000..8ba2624a --- /dev/null +++ b/noao/imred/ccdred/doc/contents.ms @@ -0,0 +1,34 @@ +.sp 1i +.ps +2 +.ft B +.ce +Contents +.sp 3 +.ps -2 +.ft R +.sp +1.\h'|0.4i'\fBIntroduction\fP\l'|5.6i.'\0\01 +.sp +2.\h'|0.4i'\fBGetting Started\fP\l'|5.6i.'\0\02 +.sp +3.\h'|0.4i'\fBProcessing Your Data\fP\l'|5.6i.'\0\05 +.br +\h'|0.4i'3.1.\h'|0.9i'Combining Calibration Images\l'|5.6i.'\0\06 +.br +\h'|0.4i'3.2.\h'|0.9i'Calibrations and Corrections\l'|5.6i.'\0\07 +.sp +4.\h'|0.4i'\fBSpecial Processing Operations\fP\l'|5.6i.'\0\08 +.br +\h'|0.4i'4.1.\h'|0.9i'Spectroscopic Flat Fields\l'|5.6i.'\0\08 +.br +\h'|0.4i'4.2.\h'|0.9i'Illumination Corrections\l'|5.6i.'\0\09 +.br +\h'|0.4i'4.3.\h'|0.9i'Sky Flat Fields\l'|5.6i.'\010 +.br +\h'|0.4i'4.4.\h'|0.9i'Illumination Corrected Flat Fields\l'|5.6i.'\010 +.br +\h'|0.4i'4.5.\h'|0.9i'Fringe Corrections\l'|5.6i.'\010 +.sp +5.\h'|0.4i'\fBSummary\fP\l'|5.6i.'\011 +.sp +\h'|0.4i'\fBReferences\fP\l'|5.6i.'\011 diff --git a/noao/imred/ccdred/doc/darkcombine.hlp b/noao/imred/ccdred/doc/darkcombine.hlp new file mode 100644 index 00000000..c545a13e --- /dev/null +++ b/noao/imred/ccdred/doc/darkcombine.hlp @@ -0,0 +1,120 @@ +.help darkcombine Aug91 noao.imred.ccdred +.ih +NAME +darkcombine -- Combine and process dark count images +.ih +USAGE +darkcombine input +.ih +PARAMETERS +.ls input +List of dark count images to combine. The \fIccdtype\fR parameter +may be used to select the zero level images from a list containing all +types of data. +.le +.ls output = "Dark" +Output dark count root image name. +.le +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "minmax" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation. See \fBcombine\fR for details. +.le +.ls ccdtype = "dark" +CCD image type to combine. If no image type is given then all input images +are combined. +.le +.ls process = yes +Process the input images before combining? +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le +.ls scale = "exposure" (none|mode|median|mean|exposure) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, or scale +by the exposure time given in the image header. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling. +If no section is given then the entire region of the image is +sampled (for efficiency the images are sampled if they are big enough). +.le + +.ce +Algorithm Parameters +.ls nlow = 0, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See \fBcombine\fR for further details. +.le +.ls blank = 0. +Output value to be used when there are no pixels. +.le +.ih +DESCRIPTION +The dark count images in the input image list are combined. +The input images may be processed first if desired. +The original images may be deleted automatically if desired. +The output pixel datatype will be real. + +This task is a script which applies \fBccdproc\fR and \fBcombine\fR. The +parameters and combining algorithms are described in detail in the help for +\fBcombine\fR. This script has default parameters specifically set for +dark count images and simplifies the combining parameters. There are other +combining options not included in this task. For these additional +features, such as thresholding, offseting, masking, and projecting, use +\fBcombine\fR. +.ih +EXAMPLES +1. The image data contains four dark count images. To automatically select +them and combine them as a background job using the default combining algorithm: + + cl> darkcombine ccd*.imh& +.ih +SEE ALSO +ccdproc, combine +.endhelp diff --git a/noao/imred/ccdred/doc/flatcombine.hlp b/noao/imred/ccdred/doc/flatcombine.hlp new file mode 100644 index 00000000..549c912c --- /dev/null +++ b/noao/imred/ccdred/doc/flatcombine.hlp @@ -0,0 +1,133 @@ +.help flatcombine Aug91 noao.imred.ccdred +.ih +NAME +flatcombine -- Combine and process flat field images +.ih +USAGE +flatcombine input +.ih +PARAMETERS +.ls input +List of flat field images to combine. The \fIccdtype\fR parameter +may be used to select the flat field images from a list containing all +types of data. +.le +.ls output = "Flat" +Output flat field root image name. The subset ID is appended. +.le +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "avsigclip" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation. See \fBcombine\fR for details. +.le +.ls ccdtype = "flat" +CCD image type to combine. If no image type is given then all input images +are combined. +.le +.ls process = yes +Process the input images before combining? +.le +.ls subsets = yes +Combine images by subset parameter? If yes then the input images are +grouped by subset parameter and each group combined into a separate output +image. The subset identifier is appended to the output and sigma image +names. See \fBsubsets\fR for more on the subset parameter. This is generally +used with flat field images. +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le +.ls scale = "mode" (none|mode|median|mean|exposure) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, or scale +by the exposure time given in the image header. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling. +If no section is given then the entire region of the image is +sampled (for efficiency the images are sampled if they are big enough). +.le + +.ce +Algorithm Parameters +.ls nlow = 1, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See \fBcombine\fR for further details. +.le +.ls blank = 1. +Output value to be used when there are no pixels. +.le +.ih +DESCRIPTION +The flat field images in the input image list are combined. If there +is more than one subset (such as a filter or grating) then the input +flat field images are grouped by subset and an combined separately. +The input images may be processed first if desired. However if all +zero level bias effects are linear then this is not necessary and some +processing time may be saved. The original images may be deleted +automatically if desired. The output pixel datatype will be real. + +This task is a script which applies \fBccdproc\fR and \fBcombine\fR. The +parameters and combining algorithms are described in detail in the help for +\fBcombine\fR. This script has default parameters specifically set for +flat field images and simplifies the combining parameters. There are other +combining options not included in this task. For these additional +features, such as thresholding, offseting, masking, and projecting, use +\fBcombine\fR. +.ih +EXAMPLES +1. The image data contains four flat field images for three filters. +To automatically select them and combine them as a background job +using the default combining algorithm: + + cl> flatcombine ccd*.imh& + +The final images are "FlatV", "FlatB", and "FlatR". +.ih +SEE ALSO +ccdproc, combine, subsets +.endhelp diff --git a/noao/imred/ccdred/doc/flatfields.hlp b/noao/imred/ccdred/doc/flatfields.hlp new file mode 100644 index 00000000..94766960 --- /dev/null +++ b/noao/imred/ccdred/doc/flatfields.hlp @@ -0,0 +1,177 @@ +.help flatfields Jun87 noao.imred.ccdred + +.ih +NAME +flatfields -- Discussion of CCD flat field calibrations +.ih +DESCRIPTION +This topic describes the different types of CCD flat fields and +the tasks available in the \fBccdred\fR and spectroscopy packages for +creating them. Flat field calibration is the most important operation +performed on CCD data. This operation calibrates the relative response +of the detector at each pixel. In some cases this is as simple as +taking a special type of observation called a flat field. However, in +many cases this calibration observation must be corrected for +iillumination, scanning, wavelength, and aperture effects. + +The discussion is in three sections; direct imaging, scan mode, +and spectroscopy. Though there are many similarities between these +modes of operation there are important differences in how corrections +are applied to the basic flat field observations. The application of +the flat field calibrations to the observations using \fBccdproc\fR is +the same in all cases, however. +.sh +1. Direct Imaging +The starting point for determining the flat field calibration is an +observation of something which should have uniform response at all +points on the detector. In addition the color of the light falling at +each pixel should be the same as that in an observation so the same +filter must be used when determining the flat field (the issue of the +matching the color of the objects observed at the appropriate pixels is +ignored here). The best calibration observation is of a blank sky. If +an accurate blank sky observation can be obtained then this is all that +is needed for a flat field calibration. This type of flat field might +be called a \fIsky flat\fR, though this term is more often used for a +type of flat field described below. There are two difficulties with +this type of calibration; finding a really blank sky and getting a +sufficiently accurate measurement without using all the observing +time. + +It is usually not possible to get a blank sky observation accurate +enough to calibrate the individual pixels without introducing +undesirable noise. What is generally done is to use a lamp to either +uniformly illuminate a part of the dome or directly illuminate the +field of view. The first type of observation is called a \fIdome +flat\fR and the second is called a \fIprojection flat\fR. We shall call +both of these types of observations \fBlamp flat fields\fR. If the +iillumination is truely uniform then these types of observations are +sufficient for flat field calibration. To get a very accurate flat +field many observations are made and then combined (see +\fBflatcombine\fR). + +Unfortunately, it is sometimes the case that the lamp flat fields +do not illuminate the telescope/detector in the same way as the actual +observations. Calibrating with these flat fields will introduce a +residual large scale iillumination pattern, though it will correctly +calibrate the relative pixel responses locally. There are two ways to +correct for this effect. The first is to correct the flat field +observation. The second is to apply the uncorrected flat field to the +observations and then apply an \fIiillumination\fR correction as a +separate operation. The first is more efficient since it consists of a +single correction applied to each observation but in some cases the +approximate correction is desired immediately, the observation needed +to make the correction has not been taken yet, or the residual +iillumination error is not discovered until later. + +For the two methods there are two types of correction. One is to +use a blank sky observation to correct for the residual iillumination +pattern. This is different than using the sky observation directly as +a flat field calibration in that only the large scale pattern is +needed. Determining the large scale iillumination does not require high +signal-to-noise at each pixel and faint objects in the image can be +either eliminated or ignored. The second method is to remove the large +scale shape from the lamp flat field. This is not as good as using a +blank sky observation but, if there is no such observation and the +iillumination pattern is essentially only in the lamp flat field, this +may be sufficient. + +From the above two paragraphs one sees there are four options. +There is a task in the \fBccdred\fR package for each of these options. +To correct a lamp flat field observation by a blank sky observation, +called a \fIsky flat\fR, the task is \fBmkskyflat\fR. To correct the +flat field for its own large scale gradients, called an \fIiillumination +flat\fR, the task is \fBmkillumflat\fR. To create a secondary +correction to be applied to data processed with the lamp flat field +image the tasks are \fBmkskycor\fR and \fBmkillumcor\fR which are, +respectively, based on a blank sky observation and the lamp flat field +iillumination pattern. + +With this introduction turn to the individual documentation for these +four tasks for further details. +.sh +2. Scan Mode +There are two types of scan modes supported by the \fBccdred\fR +package; \fIshortscan\fR and \fIlongscan\fR (see \fBccdproc\fR for +further details). They both affect the manner in which flat field +calibrations are handled. The shortscan mode produces images which are +the same as direct images except that the light recorded at each pixel +was collected by a number of different pixels. This improves the flat +field calibration. If the flat field images, of the same types +described in the direct imaging section, are observed in the same way +as all other observations, i.e. in scan mode, then there is no +difference from direct imaging (except in the quality of the flat +fields). There is a statistical advantage to observing the lamp or sky +flat field without scanning and then numerically averaging to simulate +the result of the scanning. This improves the accuracy of +the flat fields and might possibly allow direct blank sky observations +to be used for flat fields. The numerical scanning is done in +\fBccdproc\fR by setting the appropriate scanning parameters. + +In longscan mode the CCD detector is read out in such a way that +each output image pixel is the sum of the light falling on all pixels +along the direction of the scan. This reduces the flat field calibration +to one dimension, one response value for each point across the scan. +The one dimensional calibration is obtained from a longscan observation +by averaging all the readout lines. +This is done automatically in \fBccdproc\fR by setting the appropriate +parameters. In this case very good flat fields can be obtained from +one or more blank sky observations or an unscanned lamp observation. Other +corrections are not generally used. +.sh +3. Spectroscopy +Spectroscopic flat fields differ from direct imaging in that the +spectrum of the sky or lamp and transmission variations with wavelength +are part of the observation. Application of such images will introduce +the inverse of the spectrum and transmission into the observation. It +also distorts the observed counts making signal-to-noise estimates +invalid. This, and the low signal in the dispersed light, makes it +difficult to use blank sky observations directly as flat fields. As +with direct imaging, sky observation may be used to correct for +iillumination errors if necessary. At sufficiently high dispersion the +continuous lamp spectrum may be flat enough that the spectral signature +of the lamp is not a problem. Alternatively, flux calibrating the +spectra will also remove the flat field spectral signature. The +spectroscopic flat fields also have to be corrected for regions outside +of the slit or apertures to avoid bad response effects when applying +the flat field calibration to the observations. + +The basic scheme for removing the spectral signature is to average +all the lines or columns across the dispersion and within the aperture +to form an estimate of the spectrum. In addition to the averaging, a +smooth curve is fit to the lamp spectrum to remove noise. This smooth +shape is then divided back into each line or column to eliminate the +shape of the spectrum without changing the shape of the spectrum +in the spatial direction or the small scale response variations. +Regions outside of the apertures are replaced by unity. +This method requires that the dispersion be aligned fairly close to +either the CCD lines or columns. + +This scheme is used in both longslit and multiaperture spectra. +The latter includes echelle, slitlets, aperture masks, and fiber feeds. +For narrow apertures which do not have wider slits for the lamp +exposures there may be problems with flexure and defining a good +composite spectrum. The algorithm for longslit spectra is simpler and +is available in the task \fBresponse\fR in the \fBlongslit\fR package. +For multiaperture data there are problems of defining where the spectra +lie and avoiding regions off of the aperture where there is no signal. +The task which does this is \fBapnormalize\fR in the \fBapextract\fR +package. Note that the lamp observations must first be processed +explicitly for bias and dark count corrections. + +Longslit spectra may also suffer the same types of iillumination +problems found in direct imaging. However, in this case the iillumination +pattern is determined from sky observations (or the flat field itself) +by finding the large scale pattern across the dispersion and at a number +of wavelengths while avoiding the effects of night sky spectrum. The +task which makes this type of correction in the \fBlongslit\fR package +is \fBiillumination\fR. This produces an iillumination correction. +To make sky flats or the other types of corrections image arithmetic +is used. Note also that the sky observations must be explicitly +processed through the flat field stage before computing the iillumination. +.ih +SEE ALSO +.nf +ccdproc, guide, mkillumcor, mkillumflat, mkskycor, mkskyflat +apextract.apnormalize, longslit.response, longslit.iillumination +.fi +.endhelp diff --git a/noao/imred/ccdred/doc/guide.hlp b/noao/imred/ccdred/doc/guide.hlp new file mode 100644 index 00000000..5006a6ec --- /dev/null +++ b/noao/imred/ccdred/doc/guide.hlp @@ -0,0 +1,717 @@ +.help guide Feb88 noao.imred.ccdred +.ce +User's Guide to the CCDRED Package +.sh +1. Introduction + + This guide provides a brief description of the IRAF CCD reduction +package \fBccdred\fR and examples of reducing simple CCD data. It is a +generic guide in that it is not tied to any particular type of data. +There may be more specific guides (or "cookbooks") for your data. +Detailed descriptions of the tasks and features of the package are +provided in the help documentation for the package. + + The purpose of the CCDRED package is to provide tools for the easy +and efficient reduction of CCD images. The standard reduction +operations are replacement of bad columns and lines by interpolation +from neighboring columns and lines, subtraction of a bias level +determined from overscan or prescan columns or lines, subtraction of a +zero level using a zero length exposure calibration image, subtraction +of a dark count calibration image appropriately scaled to the dark time +exposure, division by a scaled flat field calibration image, division +by an iillumination image (derived from a blank sky image), subtraction +of a scaled fringe image (also derived from a blank sky image), and +trimming the image of unwanted lines or columns such as the overscan +strip. Any set of operations may be done simultaneously over a list of +images in a highly efficient manner. The reduction operations are +recorded in the image header and may also be logged on the terminal and +in a log file. + + The package also provides tools for combining multiple exposures +of object and calibration images to improve the statistical accuracy of +the observations and to remove transient bad pixels. The combining +operation scales images of different exposure times, adjusts for +variable sky background, statistically weights the images by their +signal-to-noise, and provides a number of useful algorithms for +detecting and rejecting transient bad pixels. + + Other tasks are provided for listing reduction information about +the images, deriving secondary calibration images (such as sky +corrected flat fields or iillumination correction images), and easily +setting the package parameters for different instruments. + + There are several important features provided by the package to +make the reduction of CCD images convenient; particularly to minimize +record keeping. One of these is the ability to recognize the different +types of CCD images. This ability allows the user to select a certain +class of images to be processed or listed and allows the processing +tasks to identify calibration images and process them differently from +object images. The standard CCD image types are \fIobject\fR, +\fIzero\fR level, \fIdark\fR count, and \fIflat\fR field. For more on +the image types see \fBccdtypes\fR. + + The tasks can also identify the different filters (or other subset +parameter) which require different flat field images. This means you don't +have to separate the images by filter and process each set separately. +This feature is discussed further in \fBsubsets\fR. + + The tasks keep track of the reduction steps completed on each +image and ignore images which have been processed. This feature, +along with recognizing the image types and subsets, makes it possible to +specify all the images to a task with a wildcard template, such as +"*.imh", rather than indicating each image by name. You will find this +extremely important with large sets of observations. + + A fundamental aspect of the package is that the processing +modifies the images. In other words, the reduction operations are +performed directly on the image. This "feature" further simplifies +record keeping, frees the user from having to form unique output image +names, and minimizes the amount of disk space required. There +are two safety features in this process. First, the modifications do +not take effect until the operation is completed on the image. This +allows you to abort the task without messing up the image data and +protects data if the computer crashes. The second feature is that +there is a package parameter which may be set to make a backup of the +input data with a particular prefix such as "orig" or "imdir$". This +backup feature may be used when there is sufficient disk space, when learning +to use the package, or just to be cautious. + + In a similar effort to efficiently manage disk space, when combining +images into a master object or calibration image there is an option to +delete the input images upon completion of the combining operation. +Generally this is desirable when there are many calibration exposures, +such as zero level or flat field images, which are not used after they +are combined into a final calibration image. + + The following sections guide you through the basic use of the +\fBccdred\fR package. Only the important parameters which you might +want to change are described. It is assumed that the support personnel +have created the necessary instrument files (see \fBinstruments\fR) +which will set the default parameters for the data you will be +reducing. If this is not the case you may need to delve more deeply +into the details of the tasks. Information about all the parameters +and how the various tasks operate are given in the help documentation +for the tasks and in additional special help topics. Some useful help +documentation is indicated in the discussion and also in the +\fBReferences\fR section. +.sh +2. Getting Started + + The first step is to load \fBccdred\fR. This is done by loading +the \fBnoao\fR package, followed by the image reduction package +\fBimred\fR, and finally the \fBccdred\fR package. Loading a +package consists of typing its name. Note that some of these packages may be +loaded automatically when you logon to IRAF. + + When you load the \fBccdred\fR package the menu of tasks or commands +is listed. This appears as follows: + +.nf + cl> ccdred + badpiximage ccdtest mkfringecor setinstrument + ccdgroups combine mkillumcor zerocombine + ccdhedit cosmicrays mkillumflat + ccdlist darkcombine mkskycor + ccdproc flatcombine mkskyflat +.fi + +A summary of the tasks and additional help topics is obtained by typing: + + cl> help + +This list and how to get additional help on specific topics is described +in the \fBReferences\fR section at the end of this guide. + + The first command to use is \fBsetinstrument\fR, which sets the package +appropriately for the CCD images to be reduced. The support personnel +should tell you the instrument identification, but if not a list +of known instruments may be listed by using '?' for the instrument name. + +.nf + cl> setinstrument + Instrument ID (type ? for a list) \fI<enter instrument id or ?>\fR + <Set ccdred package parameters using eparam> + <Set ccdproc task parameters using eparam> +.fi + +This task sets the default parameters and then allows you to modify the +package parameters and the processing parameters using the parameter +editor \fBeparam\fR. If you are not familiar with \fBeparam\fR see the +help or CL introduction documentation. For most terminals you move up +and down through the parameters with the terminal arrow keys, you +change the parameters by simply typing the desired value, and you exit +with control Z or control D. Note that you can change parameters for +any task at any time with \fBeparam\fR and you do not have to run +\fBsetinstrument\fR again, even if you logout, until you need to reduce +data from a different instrument. + + The \fBccdred\fR package parameters control general I/O functions of +the tasks in the package. The parameters you might wish to change are +the output pixel type and the verbose option. Except when the input +images are short integers, the noise is significantly greater than one +digital unit, and disk space is critical, it is probably better to +allow the processing to convert the images to real pixel datatype. The +verbose parameter simply prints the information written to the log file +on the terminal. This can be useful when little else is being done and +you are just beginning. However, when doing background processing and +other IRAF reduction tasks it is enough to simply look at the end of +the logfile with the task \fBtail\fR to see the current state of the +processing. + + The \fBccdproc\fR parameters control the CCD processing. There are +many parameters but they all may be conveniently set at this point. +Many of the parameters have default values set appropriately for the +instrument you specified. The images to be processed can be specified +later. What needs to be set are the processing operations that you +want done and the parameters required for each operation. The +processing operations are selected by entering yes or no for each one. +The following items briefly describe each of the possible processing +operations and the additional parameters required. + +.ls \fIfixpix\fR - Fix bad CCD lines and columns? +The bad pixels (cosmetic defects) in the detector are given in a file +specified by the parameter \fIfixfile\fR. This information is used +to replace the pixels by interpolating from the neighboring pixels. +A standard file for your instrument may be set by \fBsetinstrument\fR +or if the word "image" is given then the file is defined in the instrument +data file. For more on the bad pixel file see \fBinstruments\fR. +.le +.ls \fIoverscan\fR - Apply overscan strip correction? +The overscan or prescan region is specified by the parameter +\fIbiassec\fR. This is given as an IRAF image section. Only the +part of the section corresponding to the readout axis is used and +the other part is ignored. The length of the overscan region is +set by the \fItrimsec\fR parameter. The overscan +region is averaged along the readout axis, specified by the parameter +\fIreadaxis\fR, to create a one dimensional bias vector. This bias is +fit by a function to remove cosmic rays and noise. There are a number +of parameters at the end of the parameter list which control the +fitting. The default overscan bias section and fitting parameters for +your instrument should be set by \fBsetinstrument\fR. If the word +"image" is given the overscan bias section is defined in the image +header or the instrument translation file. If an overscan section is +not set you can use \fBimplot\fR to determine the columns or rows for +the bias region and define an overscan image section. If you are +unsure about image sections consult with someone or read the +introductory IRAF documentation. +.le +.ls \fItrim\fR - Trim the image? +The image is trimmed to the image section given by the parameter +\fItrimsec\fR. A default trim section for your instrument should be +set by \fBsetinstrument\fR, however, you may override this default if +desired. If the word "image" is given the data +image section is given in the image header or the instrument +translation file. As with the overscan image section it is +straightforward to specify, but if you are unsure consult someone. +.le +.ls \fIzerocor\fR - Apply zero level correction? +The zero level image to be subtracted is specified by the parameter +\fIzero\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.le +.ls \fIdarkcor\fR - Apply dark count correction? +The dark count image to be subtracted is specified by the parameter +\fIdark\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.le +.ls \fIflatcor\fR - Apply flat field correction? +The flat field images to be used are specified by the parameter +\fIflat\fR. There must be one flat field image for each filter +or subset (see \fBsubsets\fR) to be processed. If a flat field +image is not given then the calibration image will be sought +in the list of images to be processed. +.le +.ls \fIreadcor\fR - Convert zero level image to readout correction? +If a one dimensional zero level readout correction vector is to be subtracted +instead of a two dimensional zero level image then, when this parameter is set, +the zero level images will be averaged to one dimension. The readout axis +must be specified by the parameter \fIreadaxis\fR. The default for your +instrument is set by \fBsetinstrument\fR. +.le +.ls \fIscancor\fR - Convert flat field image to scan correction? +If the instrument is operated in a scan mode then a correction to the +flat field may be required. There are two types of scan modes, "shortscan" +and "longscan". In longscan mode flat field images will be averaged +to one dimension and the readout axis must be specified. Shortscan mode +is a little more complicated. The scan correction is used if the flat +field images are not observed in scan mode. The number of scan lines +must be specified by the parameter \fInscan\fR. If they are observed in +scan mode, like the object observations, then the scan correction +operations should \fInot\fR be specified. For details of scan mode operations +see \fBccdproc\fR. The scan parameters +should be set by \fBsetinstrument\fR. If in doubt consult someone +familiar with the instrument and mode of operation. +.le + + This description of the parameters is longer than the actual operation of +setting the parameters. The only parameters likely to change during processing +are the calibration image parameters. + + When processing many images using the same calibration files a modest +performance improvement can be achieved by keeping (caching) the +calibration images in memory to avoid disk accesses. This option +is available by specifying the amount of memory available for image +caching with the parameter \fImax_cache\fR. If the value is zero then +the images are accessed from disk as needed while if there is +sufficient memory the calibration images may be kept in memory during +the task execution. +.sh +3. Processing Your Data + + The processing path depends on the type of data, the type of +instrument, types of calibration images, and the observing +sequence. In this section we describe two types of operations common +in reducing most data; combining calibration images and performing the +standard calibration and correction operations. Some additional special +operations are described in the following section. + + However, the first thing you might want to try before any +processing is to get a listing of the CCD images showing the CCD image +types, subsets, and processing flags. The task for this is +\fBccdlist\fR. It has three types of output; a short one line per +image format, a longer format which shows the state of the processing, +and a format which prints the image names only (used to create files +containing lists of images of a particular CCD image type). To get a +quick listing type: + +.nf + cl> ccdlist *.imh + ccd001.imh[544,512][short][unknown][V]:FOCUS L98-193 + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s + ccd045.imh[544,512][short][flat][V]:dflat 5s + ccd066.imh[544,512][short][flat][B]:dflat 5s + ccd103.imh[544,512][short][flat][R]:dflat 5s + ccd104.imh[544,512][short][zero][]:bias + ccd105.imh[544,512][short][dark][]:dark 3600s +.fi + + The example shows only a sample of the images. The short format +listing tells you the name of the image, its size and pixel type, the +CCD image type as seen by the package, the subset identifier (in this +case the filter), and the title. If the data had been processed then +there would also be processing flags. If the CCD image types do not +seem right then there may be a problem with the instrument +specification. + + Many of the tasks in the \fBccdred\fR package have the parameter +\fIccdtype\fR which selects a particular type of image. To list +only the object images from the previous example: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s +.fi + +If no CCD image type is specified (by using the null string "") +then all image types are selected. This may be +necessary if your instrument data does not contain image type identifications. +.sh +3.1 Combining Calibration Images + + If you do not need to combine calibration images because you only +have one image of each type, you can skip this section. Calibration +images, particularly zero level and flat field images, are combined in +order to minimize the effects of noise and reject bad pixels in the +calibrations. The basic tool for combining images is the task +\fBcombine\fR. There are simple variants of this task whose default +parameters are set appropriately for each type of calibration image. +These are the ones you will use for calibration images leaving +\fBcombine\fR for combining object images. Zero level images are +combined with \fBzerocombine\fR, dark count images with +\fBdarkcombine\fR, and flat field images with \fBflatcombine\fR. + + For example, to combine flat field images the command is: + +.nf + cl> flatcombine *.imh + Jun 1 14:26 combine: maxreject + Images N Exp Mode Scale Offset Weight + ccd045.imh 1 5.0 INDEF 1.000 0. 0.048 + ccd046.imh 1 5.0 INDEF 1.000 0. 0.048 + <... list of files ...> + ccd065.imh 1 5.0 INDEF 1.000 0. 0.048 + ----------- ------ ------ + FlatV.imh 21 5.0 +.fi + +This output is printed when verbose mode is set. The same information +is recorded in the log file. In this case the flat fields are combined +by rejecting the maximum value at each point in the image (the +"maxreject" algorithm). The images are scaled by the exposure times, +which are all the same in this example. The mode is not evaluated for +exposure scaling and the relative weights are the same because the +exposure times are the same. The example only shows part of the +output; \fBflatcombine\fR automatically groups the flat field images by +filter to produce the calibration images "FlatV", "FlatB", and +"FlatR". +.sh +3.2 Calibrations and Corrections + + Processing the CCD data is easy and largely automated. +First, set the task parameters with the following command: + + cl> eparam ccdproc + +You may have already set the parameters when you ran +\fBsetinstrument\fR, though the calibration image parameters +\fIzero\fR, \fIdark\fR, and \fIflat\fR may still need to be set or +changed. Once this is done simply give the command + +.nf + cl> ccdproc *.imh + ccd003: Jun 1 15:13 Overscan section is [520:540,*] with mean=485.0 + ccd003: Jun 1 15:14 Trim data section is [3:510,3:510] + ccd003: Jun 1 15:14 Overscan section is [520:540,*] with mean=485.0 + FlatV: Jun 1 15:14 Trim data section is [3:510,3:510] + FlatV: Jun 1 15:15 Overscan section is [520:540,*] with mean=486.4 + ccd003: Jun 1 15:15 Flat field image is FlatV.imh with scale=138.2 + ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] + ccd004: Jun 1 15:16 Overscan section is [520:540,*] with mean=485.2 + ccd004: Jun 1 15:16 Flat field image is FlatV.imh with scale=138.2 + <... more ...> + ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] + ccd013: Jun 1 15:23 Overscan section is [520:540,*] with mean=482.4 + FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] + FlatB: Jun 1 15:23 Overscan section is [520:540,*] with mean=486.4 + ccd013: Jun 1 15:24 Flat field image is FlatB.imh with scale=132.3 + <... more ...> +.fi + + The output shown is with verbose mode set. It is the same as +recorded in the log file. It illustrates the principle of automatic +calibration image processing. The first object image, "ccd003", was +being processed when the flat field image was required. Since the +image was taken with the V filter the appropriate flat field was +determined to be "FlatV". Since it had not been processed, the +processing of "ccd003" was interrupted to process "FlatV". The +processed calibration image may have been cached if there was enough +memory. Once "FlatV" was processed (note that the flat field was not +flattened because the task knows this image is a flat field) the +processing of "ccd003" was completed. The next image, "ccd004", is +also a V filter image so the already processed, and possibly cached, +flat field "FlatV" is used again. The first B band image is "ccd013" +and, as before, the B filter flat field calibration image is processed +automatically. The same automatic calibration processing and image +caching occurs when using zero level and dark count calibration +images. + + Commonly the processing is done with the verbose mode turned off +and the task run as a background job. This is done with the commands + +.nf + cl> ccdred.verbose=no + cl> ccdproc *.imh & +.fi + +The already processed images in the input list are recognized as having been +processed and are not affected. To check the status of the processing we +can look at the end of the log file with: + + cl> tail logfile + +After processing we can repeat the \fBccdlist\fR command to find: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[508,508][real][object][V][OTF]:N2968 V 600s + ccd015.imh[508,508][real][object][B][OTF]:N3098 B 500s + ccd024.imh[544,512][short][object][R][OTF]:N4036 R 600s +.fi + +The processing flags indicate the images have been overscan corrected, +trimmed, and flat fielded. + + As you can see, processing images is very easy. There is one source +of minor confusion for beginning users and that is dealing with calibration +images. First, there is no reason that calibration images +may not be processed explicitly with \fBccdproc\fR, just remember to set +the \fIccdtype\fR to the calibration image type or to "". When processing +object images the calibration images to be used may be specified either +with the task parameter for the particular calibration image or by +including the calibration image in the list of input images. Calibration +images specified by parameter value take precedence and the task +does not check its CCD image type. Calibration images given in the +input list must have a valid CCD image type. In case too many +calibration images are specified, say because the calibration images +combined to make the master calibration images were not deleted and +so are part of the image list "*.imh", only the first one will be used. +Another point to know is that flat field, iillumination, and fringe images +are subset (filter) dependent and so a calibration image for each filter +must be specified. +.sh +4. Special Processing Operations + + The special processing operations are mostly concerned with the +flat field response correction. There are also special processing +operations available in \fBccdproc\fR for one dimensional readout +corrections in the zero level and flat field calibrations. These +were described briefly above and in more detail in \fBccdproc\fR +and are not discussed further in this guide. The processing +operations described in this section are for preparing flat fields +for two dimensional spectroscopic data, for correcting flat fields +for iilluminations effects, for making a separate iillumination correction, +and for applying corrections for fringe effects. For additional +discussion about flat fields and iillumination corrections see the +help topic \fBflatfields\fR. +.sh +4.1 Spectroscopic Flat Fields + + For spectroscopic data the flat fields may have to be processed to +remove the general shape of the lamp spectrum and to replace regions outside +of the aperture where there is no flat field information with values that +will not cause bad response effects when the flat field is applied to the +data. If the shape of the lamp spectrum is not important and if the +longslit spectra have the regions outside of the slit either off the +detector or trimmed then you may use the flat field without special +processing. + + First you must process the flat field images explicitly with + + cl> ccdproc *.imh ccdtype=flat + +where "*.imh" may be replaced with any list containing the flat fields. +If zero level and dark count corrections are required these calibration +images must be available at this time. + + Load the \fBtwodspec\fR package and then either the \fBlongslit\fR +package, for longslit data, or the \fBapextract\fR package, for +multiaperture data such as echelles, multifiber, or aperture mask +spectra. The task for removing the longslit quartz spectrum is +\fBresponse\fR. There is also a task for removing iillumination +effects, including the slit profile, from longslit spectra called +\fBiillumination\fR. For more about processing longslit spectra see the +help for these tasks and the paper \fIReduction of Longslit Spectra +with IRAF\fR. The cookbook \fIReduction of Longslit Spectroscopic +Data Using IRAF (KPNO ICCD and Cryogenic Camera Data)\fR also provides +a very good discussion even if your data is from a different instrument. + + For multiaperture data the task for removing the relative shapes of +the spectra is called \fBapnormalize\fR. Again, consult the help documentation +for this task for further details. Since you will probably also be +using the package for extracting the spectra you may be interested +in the document \fIThe IRAF APEXTRACT Package\fR. +.sh +4.2 Iillumination Corrections + + The flat field calibration images may not have the same iillumination +pattern as the observations of the sky due to the way the lamp illuminates the +optical system. In this case when the flat field correction is applied +to the data there will be gradients in the sky background. To remove +these gradients a blank sky calibration image is heavily smoothed +to produce an iillumination image. The iillumination image +is then divided into the images during processing to correct for the +iillumination difference between the flat field and the objects. +Like the flat fields, the iillumination corrections images may be subset +dependent so there should be an iillumination image for each subset. + +The task which makes iillumination correction images is \fBmkskycor\fR. +Some examples are + +.nf + cl> mkskycor sky004 Illum004 + cl> mkskycor sky*.imh "" +.fi + +In the first example the sky image "sky004" is used to make the iillumination +correction image "Illum004". In the second example the sky images are +converted to iillumination correction images by specifying no output image +names. Like \fBccdproc\fR if the input images have not been processed they +are first processed automatically. + +To apply the iillumination correction + +.nf + cl> ccdproc *.imh ccdtype=object illumcor+ illum=Illum004 + cl> ccdproc *.imh ccdtype=object illumcor+ illum=sky*.imh +.fi + +The iillumination images could also be set using \fBeparam\fR or given +on the command line. +.sh +4.3 Sky Flat Fields + + You will notice that when you process images with an iillumination +correction you are dividing each image by a flat field calibration and +an iillumination correction. If the iillumination corrections are not +done as a later step but at the same time as the rest of the processing +one will get the same calibration by multiplying the flat field by +the iillumination correction and using this product alone as the +flat field. Such an image is called a \fIsky flat\fR since it is +a flat field which has been corrected to yield a flat sky when applied +to the observations. This approach has the advantage of one less +calibration image and two less computations (scaling and dividing the +iillumination correction). As an added short cut, rather than compute +the iillumination image with \fBmkskycor\fR and then multiplying, the +task \fBmkskyflat\fR does all this in one step. Thus, \fBmkskyflat\fR +takes an input blank sky image, processes it if needed, determines the +appropriate flat field (sky flats are also subset dependent) from the +\fBccdproc\fR parameters or the input image list, and produces an +output sky flat. Further if no output image is specified the task +converts the input blank sky calibration image into a sky flat. + + Two examples in which a new image is created and in which the +input images are converted to sky flats are + +.nf + cl> mkskyflat sky004 Skyflat + cl> mkskyflat sky*.imh "" +.fi +.sh +4.4 Iillumination Corrected Flat Fields + + A third method to account for iillumination problems in the flat fields +is to remove the large scale pattern from the flat field itself. This is +useful if there are no reasonable blank sky calibration images and the +astronomical exposures are evenly illuminated but the flat fields are not. +This is done by smoothing the flat field images instead of blank sky +images. As with using the sky images there are two methods, creating +an iillumination correction to be applied as a separate step or fixing +the original flat field. The smoothing algorithm is +the same as that used in the other tasks. The tasks to make these types +of corrections are \fBmkillumcor\fR and \fBmkillumflat\fR. The usage +is pretty much the same as the other iillumination correction tasks +except that it is more reasonable to replace the original flat fields +by the corrected flat fields when fixing the flat field. Examples +of an iillumination correction and removing the iillumination pattern +from the flat field are + +.nf + cl> mkillumcor flat025 Illum025 + cl> mkillumflat flat*.imh "" +.fi + +As with the other tasks, the input images are processed if necessary. +.sh +4.5 Fringe Corrections + + Some CCD detectors suffer from fringing effects due to the night +sky emission lines which are not removed by the other calibration +and correction operations. To correct for the fringing you need a +really blank sky image. There is not yet a task to remove objects from +sky images because this is often done with an interactive image display +tool (which will soon be added). The blank sky image is heavily smoothed +to determine the mean sky background and then this is subtracted from the +original image. The image should then be essentially zero except for the +fringe pattern. This fringe correction image is scaled to the same +exposure time as the image to be corrected and then subtracted to remove +the fringing. Note that since the night sky lines are variable there +may need to be an additional scaling applied. Determining this scaling +requires either an interactive display tool or a very clever task. +Such tasks will also be added in the future. + + The task to make a fringe correction image is \fBmkfringecor\fR. +the sky background is determined in exactly the same way as the iillumination +pattern, in fact the same sky image may be used for both the sky +iillumination and for the fringe correction. The task works consistently +with the "mk" tasks in that the input images are processed first if needed +and then the output correction image is produced with the specified name +or replaces the input image if no output image is specified. +As examples, + +.nf + cl> mkfringecor sky004 Fringe + cl> mkfringecor sky*.imh "" +.fi +.sh +5. Demonstration + + A simple demonstration task is available. To run this demonstration +load the \fBccdtest\fR package; this is a subpackage of the main +\fBccdred\fR package. Then simply type + + cl> demo + +The demonstration will then create some artificial CCD data and reduce +them giving descriptive comments as it goes along. This demonstration uses +the "playback" facility of the command language and is actually substituting +it's own commands for terminal input. Initially you must type carriage return +or space after each comment ending with "...". If you wish to have the +demonstration run completely automatically at it's own speed then type 'g' +a the "..." prompt. Thereafter, it will simple pause long enough to give +you a chance to read the comments. When the demo is finished you will +need to remove the files created. However, feel free to examine the reduced +images, the log file, etc. \fINote that the demonstration changes the +setup parameters so be sure to run \fBsetinstrument\fI again and check +the setup parameters.\fR +.sh +6. Summary + + The \fBccdred\fR package is very easy to use. First load the package; +it is in the \fBimred\fR package which is in the \fBnoao\fR package. +If this is your first time reducing data from a particular instrument +or if you have changed instruments then run \fBsetinstrument\fR. +Set the processing parameters for the operations you want performed. +If you need to combine calibration images to form a master calibration +image use one of the combine tasks. Spectroscopic flat fields may +need to be processed first in order to remove the lamp spectrum. +Finally, just type + + cl> ccdproc *.imh& +.sh +7. References + + A general guide to using IRAF is \fIA User's Introduction to the IRAF +Command Language\fR. This document may be found in the IRAF documentation +sets and is available from the National Optical Astronomy Observatories, +Central Computer Services (NOAO-CCS). + + A more detailed description of the \fBccdred\fR package including +a discussion of the design and some of the algorithms see \fIThe IRAF +CCD Reduction Package -- CCDRED\fR by F. Valdes. This paper is available +from NOAO-CCS and appears in the proceedings of the Santa Cruz Summer +Workshop in Astronomy and Astrophysics, \fIInstrumentation for Ground-Based +Optical Astronomy: Present and Future\fR, edited by Lloyd B. Robinson and +published by Springer-Verlag. + + The task descriptions and supplementary documentation are available +in printed form in the IRAF documentation sets, a special set +containing documentation for just the \fBccdred\fR package, and on-line +through the help task by typing + + cl> help \fItopic\fR + +where \fItopic\fR is one of the following. + +.nf + badpiximage - Create a bad pixel mask image from a bad pixel file + ccdgroups - Group CCD images into image lists + ccdhedit - CCD image header editor + ccdlist - List CCD processing information + ccdproc - Process CCD images + ccdtest - CCD test and demonstration package + combine - Combine CCD images + cosmicrays - Detect and replace cosmic rays + darkcombine - Combine and process dark count images + flatcombine - Combine and process flat field images + mkfringecor - Make fringe correction images from sky images + mkillumcor - Make flat field iillumination correction images + mkillumflat - Make iillumination corrected flat fields + mkskycor - Make sky iillumination correction images + mkskyflat - Make sky corrected flat field images +setinstrument - Set instrument parameters + zerocombine - Combine and process zero level images + + ADDITIONAL HELP TOPICS + + ccdred - CCD image reduction package + ccdtypes - Description of the CCD image types + flatfields - Discussion of CCD flat field calibrations + guide - Introductory guide to using the CCDRED package + instruments - Instrument specific data files + subsets - Description of CCD subsets +.fi + +Printed copies of the on-line help documentation may be made with the +command + + cl> help topic | lprint + + In addition to the package documentation for \fBccdred\fR, +\fBlongslit\fR, and \fBapextract\fR there may be specific guides for +certain instruments. These specific guides, called "cookbooks", give +specific examples and parameter values for the CCD data. +.endhelp diff --git a/noao/imred/ccdred/doc/guide.ms b/noao/imred/ccdred/doc/guide.ms new file mode 100644 index 00000000..62d87bb9 --- /dev/null +++ b/noao/imred/ccdred/doc/guide.ms @@ -0,0 +1,794 @@ +.RP +.TL +User's Guide to the CCDRED Package +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +June 1987 +Revised February 1988 +.AB +The IRAF CCD reduction package, \fBccdred\fR, provides tools +for the easy and efficient reduction of CCD images. The standard +reduction operations are replacement of bad pixels, subtraction of an +overscan or prescan bias, subtraction of a zero level image, +subtraction of a dark count image, division by a flat field calibration +image, division by an illumination correction, subtraction of a fringe +image, and trimming unwanted lines or columns. Another common +operation provided by the package is scaling and combining images with +a number of algorithms for rejecting cosmic rays. Data in the image +header is used to make the reductions largely automated and +self-documenting though the package may still be used in the absence of +this data. Also a translation mechanism is used to relate image header +parameters to those used by the package to allow data from a variety of +observatories and instruments to be processed. This guide provides a brief +description of the IRAF CCD reduction package and examples of reducing +simple CCD data. +.AE +.NH +Introduction +.LP + This guide provides a brief description of the IRAF CCD reduction +package \fBccdred\fR and examples of reducing simple CCD data. It is a +generic guide in that it is not tied to any particular type of data. +There may be more specific guides (or "cookbooks") for your data. +Detailed descriptions of the tasks and features of the package are +provided in the help documentation for the package. + + The purpose of the CCDRED package is to provide tools for the easy +and efficient reduction of CCD images. The standard reduction +operations are replacement of bad columns and lines by interpolation +from neighboring columns and lines, subtraction of a bias level +determined from overscan or prescan columns or lines, subtraction of a +zero level using a zero length exposure calibration image, subtraction +of a dark count calibration image appropriately scaled to the dark time +exposure, division by a scaled flat field calibration image, division +by an illumination image (derived from a blank sky image), subtraction +of a scaled fringe image (also derived from a blank sky image), and +trimming the image of unwanted lines or columns such as the overscan +strip. Any set of operations may be done simultaneously over a list of +images in a highly efficient manner. The reduction operations are +recorded in the image header and may also be logged on the terminal and +in a log file. + + The package also provides tools for combining multiple exposures +of object and calibration images to improve the statistical accuracy of +the observations and to remove transient bad pixels. The combining +operation scales images of different exposure times, adjusts for +variable sky background, statistically weights the images by their +signal-to-noise, and provides a number of useful algorithms for +detecting and rejecting transient bad pixels. + + Other tasks are provided for listing reduction information about +the images, deriving secondary calibration images (such as sky +corrected flat fields or illumination correction images), and easily +setting the package parameters for different instruments. + + There are several important features provided by the package to +make the reduction of CCD images convenient; particularly to minimize +record keeping. One of these is the ability to recognize the different +types of CCD images. This ability allows the user to select a certain +class of images to be processed or listed and allows the processing +tasks to identify calibration images and process them differently from +object images. The standard CCD image types are \fIobject\fR, +\fIzero\fR level, \fIdark\fR count, and \fIflat\fR field. For more on +the image types see \fBccdtypes\fR. + + The tasks can also identify the different filters (or other subset +parameter) which require different flat field images. This means you don't +have to separate the images by filter and process each set separately. +This feature is discussed further in \fBsubsets\fR. + + The tasks keep track of the reduction steps completed on each +image and ignore images which have been processed. This feature, +along with recognizing the image types and subsets, makes it possible to +specify all the images to a task with a wildcard template, such as +"*.imh", rather than indicating each image by name. You will find this +extremely important with large sets of observations. + + A fundamental aspect of the package is that the processing +modifies the images. In other words, the reduction operations are +performed directly on the image. This "feature" further simplifies +record keeping, frees the user from having to form unique output image +names, and minimizes the amount of disk space required. There +are two safety features in this process. First, the modifications do +not take effect until the operation is completed on the image. This +allows you to abort the task without messing up the image data and +protects data if the computer crashes. The second feature is that +there is a package parameter which may be set to make a backup of the +input data with a particular prefix such as "orig" or "imdir$". This +backup feature may be used when there is sufficient disk space, when learning +to use the package, or just to be cautious. + + In a similar effort to efficiently manage disk space, when combining +images into a master object or calibration image there is an option to +delete the input images upon completion of the combining operation. +Generally this is desirable when there are many calibration exposures, +such as zero level or flat field images, which are not used after they +are combined into a final calibration image. + + The following sections guide you through the basic use of the +\fBccdred\fR package. Only the important parameters which you might +want to change are described. It is assumed that the support personnel +have created the necessary instrument files (see \fBinstruments\fR) +which will set the default parameters for the data you will be +reducing. If this is not the case you may need to delve more deeply +into the details of the tasks. Information about all the parameters +and how the various tasks operate are given in the help documentation +for the tasks and in additional special help topics. Some useful help +documentation is indicated in the discussion and also in the +\fBReferences\fR section. +.NH +Getting Started +.LP + The first step is to load \fBccdred\fR. This is done by loading +the \fBnoao\fR package, followed by the image reduction package +\fBimred\fR, and finally the \fBccdred\fR package. Loading a +package consists of typing its name. Note that some of these packages may be +loaded automatically when you logon to IRAF. + + When you load the \fBccdred\fR package the menu of tasks or commands +is listed. This appears as follows: + +.nf +.KS +.ft L + cl> ccdred + badpiximage ccdtest mkfringecor setinstrument + ccdgroups combine mkillumcor zerocombine + ccdhedit cosmicrays mkillumflat + ccdlist darkcombine mkskycor + ccdproc flatcombine mkskyflat +.ft R +.KE +.fi + +A summary of the tasks and additional help topics is obtained by typing: + +.ft L + cl> help +.ft R + +This list and how to get additional help on specific topics is described +in the \fBReferences\fR section at the end of this guide. + + The first command to use is \fBsetinstrument\fR, which sets the package +appropriately for the CCD images to be reduced. The support personnel +should tell you the instrument identification, but if not a list +of known instruments may be listed by using '?' for the instrument name. + +.nf +.ft L + cl> setinstrument + Instrument ID (type ? for a list) \fI<enter instrument id or ?> + <Set ccdred package parameters using eparam> + <Set ccdproc task parameters using eparam> +.ft R +.fi + +This task sets the default parameters and then allows you to modify the +package parameters and the processing parameters using the parameter +editor \fBeparam\fR. If you are not familiar with \fBeparam\fR see the +help or CL introduction documentation. For most terminals you move up +and down through the parameters with the terminal arrow keys, you +change the parameters by simply typing the desired value, and you exit +with control Z or control D. Note that you can change parameters for +any task at any time with \fBeparam\fR and you do not have to run +\fBsetinstrument\fR again, even if you logout, until you need to reduce +data from a different instrument. + + The \fBccdred\fR package parameters control general I/O functions of +the tasks in the package. The parameters you might wish to change are +the output pixel type and the verbose option. Except when the input +images are short integers, the noise is significantly greater than one +digital unit, and disk space is critical, it is probably better to +allow the processing to convert the images to real pixel datatype. The +verbose parameter simply prints the information written to the log file +on the terminal. This can be useful when little else is being done and +you are just beginning. However, when doing background processing and +other IRAF reduction tasks it is enough to simply look at the end of +the logfile with the task \fBtail\fR to see the current state of the +processing. + + The \fBccdproc\fR parameters control the CCD processing. There are +many parameters but they all may be conveniently set at this point. +Many of the parameters have default values set appropriately for the +instrument you specified. The images to be processed can be specified +later. What needs to be set are the processing operations that you +want done and the parameters required for each operation. The +processing operations are selected by entering yes or no for each one. +The following items briefly describe each of the possible processing +operations and the additional parameters required. + +.LP +\fIfixpix\fR - Fix bad CCD lines and columns? +.IP +The bad pixels (cosmetic defects) in the detector are given in a file +specified by the parameter \fIfixfile\fR. This information is used +to replace the pixels by interpolating from the neighboring pixels. +A standard file for your instrument may be set by \fBsetinstrument\fR +or if the word "image" is given then the file is defined in the instrument +data file. For more on the bad pixel file see \fBinstruments\fR. +.LP +\fIoverscan\fR - Apply overscan strip correction? +.IP +The overscan or prescan region is specified by the parameter +\fIbiassec\fR. This is given as an IRAF image section. The overscan +region is averaged along the readout axis, specified by the parameter +\fIreadaxis\fR, to create a one dimensional bias vector. This bias is +fit by a function to remove cosmic rays and noise. There are a number +of parameters at the end of the parameter list which control the +fitting. The default overscan bias section and fitting parameters for +your instrument should be set by \fBsetinstrument\fR. If the word +"image" is given the overscan bias section is defined in the image +header or the instrument translation file. If an overscan section is +not set you can use \fBimplot\fR to determine the columns or rows for +the bias region and define an overscan image section. If you are +unsure about image sections consult with someone or read the +introductory IRAF documentation. +.LP +\fItrim\fR - Trim the image? +.IP +The image is trimmed to the image section given by the parameter +\fItrimsec\fR. A default trim section for your instrument should be +set by \fBsetinstrument\fR, however, you may override this default if +desired. If the word "image" is given the data +image section is given in the image header or the instrument +translation file. As with the overscan image section it is +straightforward to specify, but if you are unsure consult someone. +.LP +\fIzerocor\fR - Apply zero level correction? +.IP +The zero level image to be subtracted is specified by the parameter +\fIzero\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.LP +\fIdarkcor\fR - Apply dark count correction? +.IP +The dark count image to be subtracted is specified by the parameter +\fIdark\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.LP +\fIflatcor\fR - Apply flat field correction? +.IP +The flat field images to be used are specified by the parameter +\fIflat\fR. There must be one flat field image for each filter +or subset (see \fBsubsets\fR) to be processed. If a flat field +image is not given then the calibration image will be sought +in the list of images to be processed. +.LP +\fIreadcor\fR - Convert zero level image to readout correction? +.IP +If a one dimensional zero level readout correction vector is to be subtracted +instead of a two dimensional zero level image then, when this parameter is set, +the zero level images will be averaged to one dimension. The readout axis +must be specified by the parameter \fIreadaxis\fR. The default for your +instrument is set by \fBsetinstrument\fR. +.LP +\fIscancor\fR - Convert flat field image to scan correction? +.IP +If the instrument is operated in a scan mode then a correction to the +flat field may be required. There are two types of scan modes, "shortscan" +and "longscan". In longscan mode flat field images will be averaged +to one dimension and the readout axis must be specified. Shortscan mode +is a little more complicated. The scan correction is used if the flat +field images are not observed in scan mode. The number of scan lines +must be specified by the parameter \fInscan\fR. If they are observed in +scan mode, like the object observations, then the scan correction +operations should \fInot\fR be specified. For details of scan mode operations +see \fBccdproc\fR. The scan parameters +should be set by \fBsetinstrument\fR. If in doubt consult someone +familiar with the instrument and mode of operation. +.LP + + This description of the parameters is longer than the actual operation of +setting the parameters. The only parameters likely to change during processing +are the calibration image parameters. + + When processing many images using the same calibration files a modest +performance improvement can be achieved by keeping (caching) the +calibration images in memory to avoid disk accesses. This option +is available by specifying the amount of memory available for image +caching with the parameter \fImax_cache\fR. If the value is zero then +the images are accessed from disk as needed while if there is +sufficient memory the calibration images may be kept in memory during +the task execution. +.NH +Processing Your Data +.LP + The processing path depends on the type of data, the type of +instrument, types of calibration images, and the observing +sequence. In this section we describe two types of operations common +in reducing most data; combining calibration images and performing the +standard calibration and correction operations. Some additional special +operations are described in the following section. + + However, the first thing you might want to try before any +processing is to get a listing of the CCD images showing the CCD image +types, subsets, and processing flags. The task for this is +\fBccdlist\fR. It has three types of of output; a short one line per +image format, a longer format which shows the state of the processing, +and a format which prints the image names only (used to create files +containing lists of images of a particular CCD image type). To get a +quick listing type: + +.nf +.ft L + cl> ccdlist *.imh + ccd001.imh[544,512][short][unknown][V]:FOCUS L98-193 + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s + ccd045.imh[544,512][short][flat][V]:dflat 5s + ccd066.imh[544,512][short][flat][B]:dflat 5s + ccd103.imh[544,512][short][flat][R]:dflat 5s + ccd104.imh[544,512][short][zero][]:bias + ccd105.imh[544,512][short][dark][]:dark 3600s +.ft R +.fi + + The example shows only a sample of the images. The short format +listing tells you the name of the image, its size and pixel type, the +CCD image type as seen by the package, the subset identifier (in this +case the filter), and the title. If the data had been processed then +there would also be processing flags. If the CCD image types do not +seem right then there may be a problem with the instrument +specification. + + Many of the tasks in the \fBccdred\fR package have the parameter +\fIccdtype\fR which selects a particular type of image. To list +only the object images from the previous example: + +.nf +.ft L + cl> ccdlist *.imh ccdtype=object + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s +.ft R +.fi + +If no CCD image type is specified (by using the null string "") +then all image types are selected. This may be +necessary if your instrument data does not contain image type identifications. +.NH 2 +Combining Calibration Images +.LP + If you do not need to combine calibration images because you only +have one image of each type, you can skip this section. Calibration +images, particularly zero level and flat field images, are combined in +order to minimize the effects of noise and reject bad pixels in the +calibrations. The basic tool for combining images is the task +\fBcombine\fR. There are simple variants of this task whose default +parameters are set appropriately for each type of calibration image. +These are the ones you will use for calibration images leaving +\fBcombine\fR for combining object images. Zero level images are +combined with \fBzerocombine\fR, dark count images with +\fBdarkcombine\fR, and flat field images with \fBflatcombine\fR. + + For example, to combine flat field images the command is: + +.nf +.ft L + cl> flatcombine *.imh + Jun 1 14:26 combine: maxreject + Images N Exp Mode Scale Offset Weight + ccd045.imh 1 5.0 INDEF 1.000 0. 0.048 + ccd046.imh 1 5.0 INDEF 1.000 0. 0.048 + \fI<... list of files ...>\fL + ccd065.imh 1 5.0 INDEF 1.000 0. 0.048 + ----------- ------ ------ + FlatV.imh 21 5.0 +.ft R +.fi + +This output is printed when verbose mode is set. The same information +is recorded in the log file. In this case the flat fields are combined +by rejecting the maximum value at each point in the image (the +"maxreject" algorithm). The images are scaled by the exposure times, +which are all the same in this example. The mode is not evaluated for +exposure scaling and the relative weights are the same because the +exposure times are the same. The example only shows part of the +output; \fBflatcombine\fR automatically groups the flat field images by +filter to produce the calibration images "FlatV", "FlatB", and +"FlatR". +.NH 2 +Calibrations and Corrections +.LP + Processing the CCD data is easy and largely automated. +First, set the task parameters with the following command: + +.ft L + cl> eparam ccdproc +.ft R + +You may have already set the parameters when you ran +\fBsetinstrument\fR, though the calibration image parameters +\fIzero\fR, \fIdark\fR, and \fIflat\fR may still need to be set or +changed. Once this is done simply give the command + +.nf +.ft L + cl> ccdproc *.imh + ccd003: Jun 1 15:13 Overscan section is [520:540,*] with mean=485.0 + ccd003: Jun 1 15:14 Trim data section is [3:510,3:510] + ccd003: Jun 1 15:14 Overscan section is [520:540,*] with mean=485.0 + FlatV: Jun 1 15:14 Trim data section is [3:510,3:510] + FlatV: Jun 1 15:15 Overscan section is [520:540,*] with mean=486.4 + ccd003: Jun 1 15:15 Flat field image is FlatV.imh with scale=138.2 + ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] + ccd004: Jun 1 15:16 Overscan section is [520:540,*] with mean=485.2 + ccd004: Jun 1 15:16 Flat field image is FlatV.imh with scale=138.2 + \fI<... more ...>\fL + ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] + ccd013: Jun 1 15:23 Overscan section is [520:540,*] with mean=482.4 + FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] + FlatB: Jun 1 15:23 Overscan section is [520:540,*] with mean=486.4 + ccd013: Jun 1 15:24 Flat field image is FlatB.imh with scale=132.3 + \fI<... more ...>\fL +.ft R +.fi + + The output shown is with verbose mode set. It is the same as +recorded in the log file. It illustrates the principle of automatic +calibration image processing. The first object image, "ccd003", was +being processed when the flat field image was required. Since the +image was taken with the V filter the appropriate flat field was +determined to be "FlatV". Since it had not been processed, the +processing of "ccd003" was interrupted to process "FlatV". The +processed calibration image may have been cached if there was enough +memory. Once "FlatV" was processed (note that the flat field was not +flattened because the task knows this image is a flat field) the +processing of "ccd003" was completed. The next image, "ccd004", is +also a V filter image so the already processed, and possibly cached, +flat field "FlatV" is used again. The first B band image is "ccd013" +and, as before, the B filter flat field calibration image is processed +automatically. The same automatic calibration processing and image +caching occurs when using zero level and dark count calibration +images. + + Commonly the processing is done with the verbose mode turned off +and the task run as a background job. This is done with the commands + +.nf +.ft L + cl> ccdred.verbose=no + cl> ccdproc *.imh & +.ft R +.fi + +The already processed images in the input list are recognized as having been +processed and are not affected. To check the status of the processing we +can look at the end of the log file with: + +.ft L + cl> tail logfile +.ft R + +After processing we can repeat the \fBccdlist\fR command to find: + +.nf +.ft L + cl> ccdlist *.imh ccdtype=object + ccd007.imh[508,508][real][object][V][OTF]:N2968 V 600s + ccd015.imh[508,508][real][object][B][OTF]:N3098 B 500s + ccd024.imh[544,512][short][object][R][OTF]:N4036 R 600s +.ft R +.fi + +The processing flags indicate the images have been overscan corrected, +trimmed, and flat fielded. + + As you can see, processing images is very easy. There is one source +of minor confusion for beginning users and that is dealing with calibration +images. First, there is no reason that calibration images +may not be processed explicitly with \fBccdproc\fR, just remember to set +the \fIccdtype\fR to the calibration image type or to "". When processing +object images the calibration images to be used may be specified either +with the task parameter for the particular calibration image or by +including the calibration image in the list of input images. Calibration +images specified by parameter value take precedence and the task +does not check its CCD image type. Calibration images given in the +input list must have a valid CCD image type. In case too many +calibration images are specified, say because the calibration images +combined to make the master calibration images were not deleted and +so are part of the image list "*.imh", only the first one will be used. +Another point to know is that flat field, illumination, and fringe images +are subset (filter) dependent and so a calibration image for each filter +must be specified. +.NH +Special Processing Operations +.LP + The special processing operations are mostly concerned with the +flat field response correction. There are also special processing +operations available in \fBccdproc\fR for one dimensional readout +corrections in the zero level and flat field calibrations. These +were described briefly above and in more detail in \fBccdproc\fR +and are not discussed further in this guide. The processing +operations described in this section are for preparing flat fields +for two dimensional spectroscopic data, for correcting flat fields +for illuminations effects, for making a separate illumination correction, +and for applying corrections for fringe effects. For additional +discussion about flat fields and illumination corrections see the +help topic \fBflatfields\fR. +.NH 2 +Spectroscopic Flat Fields +.LP + For spectroscopic data the flat fields may have to be processed to +remove the general shape of the lamp spectrum and to replace regions outside +of the aperture where there is no flat field information with values that +will not cause bad response effects when the flat field is applied to the +data. If the shape of the lamp spectrum is not important and if the +longslit spectra have the regions outside of the slit either off the +detector or trimmed then you may use the flat field without special +processing. + + First you must process the flat field images explicitly with + +.ft L + cl> ccdproc *.imh ccdtype=flat +.ft R + +where "*.imh" may be replaced with any list containing the flat fields. +If zero level and dark count corrections are required these calibration +images must be available at this time. + + Load the \fBtwodspec\fR package and then either the \fBlongslit\fR +package, for longslit data, or the \fBapextract\fR package, for +multiaperture data such as echelles, multifiber, or aperture mask +spectra. The task for removing the longslit quartz spectrum is +\fBresponse\fR. There is also a task for removing illumination +effects, including the slit profile, from longslit spectra called +\fBillumination\fR. For more about processing longslit spectra see the +help for these tasks and the paper \fIReduction of Longslit Spectra +with IRAF\fR. The cookbook \fIReduction of Longslit Spectroscopic +Data Using IRAF (KPNO ICCD and Cryogenic Camera Data)\fR also provides +a very good discussion even if your data is from a different instrument. + + For multiaperture data the task for removing the relative shapes of +the spectra is called \fBapnormalize\fR. Again, consult the help documentation +for this task for further details. Since you will probably also be +using the package for extracting the spectra you may be interested +in the document \fIThe IRAF APEXTRACT Package\fR. +.NH 2 +Illumination Corrections +.LP + The flat field calibration images may not have the same illumination +pattern as the observations of the sky due to the way the lamp illuminates the +optical system. In this case when the flat field correction is applied +to the data there will be gradients in the sky background. To remove +these gradients a blank sky calibration image is heavily smoothed +to produce an illumination image. The illumination image +is then divided into the images during processing to correct for the +illumination difference between the flat field and the objects. +Like the flat fields, the illumination corrections images may be subset +dependent so there should be an illumination image for each subset. + +The task which makes illumination correction images is \fBmkskycor\fR. +Some examples are + +.nf +.ft L + cl> mkskycor sky004 Illum004 + cl> mkskycor sky*.imh "" +.ft R +.fi + +In the first example the sky image "sky004" is used to make the illumination +correction image "Illum004". In the second example the sky images are +converted to illumination correction images by specifying no output image +names. Like \fBccdproc\fR if the input images have not been processed they +are first processed automatically. + +To apply the illumination correction + +.nf +.ft L + cl> ccdproc *.imh ccdtype=object illumcor+ illum=Illum004 + cl> ccdproc *.imh ccdtype=object illumcor+ illum=sky*.imh +.ft R +.fi + +The illumination images could also be set using \fBeparam\fR or given +on the command line. +.NH 2 +Sky Flat Fields +.LP + You will notice that when you process images with an illumination +correction you are dividing each image by a flat field calibration and +an illumination correction. If the illumination corrections are not +done as a later step but at the same time as the rest of the processing +one will get the same calibration by multiplying the flat field by +the illumination correction and using this product alone as the +flat field. Such an image is called a \fIsky flat\fR since it is +a flat field which has been corrected to yield a flat sky when applied +to the observations. This approach has the advantage of one less +calibration image and two less computations (scaling and dividing the +illumination correction). As an added short cut, rather than compute +the illumination image with \fBmkskycor\fR and then multiplying, the +task \fBmkskyflat\fR does all this in one step. Thus, \fBmkskyflat\fR +takes an input blank sky image, processes it if needed, determines the +appropriate flat field (sky flats are also subset dependent) from the +\fBccdproc\fR parameters or the input image list, and produces an +output sky flat. Further if no output image is specified the task +converts the input blank sky calibration image into a sky flat. + + Two examples in which a new image is created and in which the +input images are converted to sky flats are + +.nf +.ft L + cl> mkskyflat sky004 Skyflat + cl> mkskyflat sky*.imh "" +.ft R +.fi +.NH 2 +Illumination Corrected Flat Fields +.LP + A third method to account for illumination problems in the flat fields +is to remove the large scale pattern from the flat field itself. This is +useful if there are no reasonable blank sky calibration images and the +astronomical exposures are evenly illuminated but the flat fields are not. +This is done by smoothing the flat field images instead of blank sky +images. As with using the sky images there are two methods, creating +an illumination correction to be applied as a separate step or fixing +the original flat field. The smoothing algorithm is +the same as that used in the other tasks. The tasks to make these types +of corrections are \fBmkillumcor\fR and \fBmkillumflat\fR. The usage +is pretty much the same as the other illumination correction tasks +except that it is more reasonable to replace the original flat fields +by the corrected flat fields when fixing the flat field. Examples +of an illumination correction and removing the illumination pattern +from the flat field are + +.nf +.ft L + cl> mkillumcor flat025 Illum025 + cl> mkillumflat flat*.imh "" +.ft R +.fi + +As with the other tasks, the input images are processed if necessary. +.NH 2 +Fringe Corrections +.LP + Some CCD detectors suffer from fringing effects due to the night +sky emission lines which are not removed by the other calibration +and correction operations. To correct for the fringing you need a +really blank sky image. There is not yet a task to remove objects from +sky images because this is often done with an interactive image display +tool (which will soon be added). The blank sky image is heavily smoothed +to determine the mean sky background and then this is subtracted from the +original image. The image should then be essentially zero except for the +fringe pattern. This fringe correction image is scaled to the same +exposure time as the image to be corrected and then subtracted to remove +the fringing. Note that since the night sky lines are variable there +may need to be an additional scaling applied. Determining this scaling +requires either an interactive display tool or a very clever task. +Such tasks will also be added in the future. + + The task to make a fringe correction image is \fBmkfringecor\fR. +the sky background is determined in exactly the same way as the illumination +pattern, in fact the same sky image may be used for both the sky +illumination and for the fringe correction. The task works consistently +with the "mk" tasks in that the input images are processed first if needed +and then the output correction image is produced with the specified name +or replaces the input image if no output image is specified. +As examples, + +.nf +.ft L + cl> mkfringecor sky004 Fringe + cl> mkfringecor sky*.imh "" +.ft R +.fi +.NH +Demonstration +.LP + A simple demonstration task is available. To run this demonstration +load the \fBccdtest\fR package; this is a subpackage of the main +\fBccdred\fR package. Then simply type + +.ft L + cl> demo +.ft R + +The demonstration will then create some artificial CCD data and reduce +them giving descriptive comments as it goes along. This demonstration uses +the "playback" facility of the command language and is actually substituting +it's own commands for terminal input. Initially you must type carriage return +or space after each comment ending with "...". If you wish to have the +demonstration run completely automatically at it's own speed then type 'g' +a the "..." prompt. Thereafter, it will simple pause long enough to give +you a chance to read the comments. When the demo is finished you will +need to remove the files created. However, feel free to examine the reduced +images, the log file, etc. \fINote that the demonstration changes the +setup parameters so be sure to run \fBsetinstrument\fI again and check +the setup parameters.\fR +.NH +Summary +.LP + The \fBccdred\fR package is very easy to use. First load the package; +it is in the \fBimred\fR package which is in the \fBnoao\fR package. +If this is your first time reducing data from a particular instrument +or if you have changed instruments then run \fBsetinstrument\fR. +Set the processing parameters for the operations you want performed. +If you need to combine calibration images to form a master calibration +image use one of the combine tasks. Spectroscopic flat fields may +need to be processed first in order to remove the lamp spectrum. +Finally, just type + +.ft L + cl> ccdproc *.imh& +.ft R +.SH +References +.LP + A general guide to using IRAF is \fIA User's Introduction to the IRAF +Command Language\fR. This document may be found in the IRAF documentation +sets and is available from the National Optical Astronomy Observatories, +Central Computer Services (NOAO-CCS). + + A more detailed description of the \fBccdred\fR package including +a discussion of the design and some of the algorithms see \fIThe IRAF +CCD Reduction Package -- CCDRED\fR" by F. Valdes. This paper is available +from NOAO-CCS and appears in the proceedings of the Santa Cruz Summer +Workshop in Astronomy and Astrophysics, \fIInstrumentation for Ground-Based +Optical Astronomy: Present and Future\fR, edited by Lloyd B. Robinson and +published by Springer-Verlag. + + The task descriptions and supplementary documentation are available +in printed form in the IRAF documentation sets, a special set +containing documentation for just the \fBccdred\fR package, and on-line +through the help task by typing + +.ft L + cl> help \fItopic\fR +.ft R + +where \fItopic\fR is one of the following. + +.nf +.ft L + badpiximage - Create a bad pixel mask image from a bad pixel file + ccdgroups - Group CCD images into image lists + ccdhedit - CCD image header editor + ccdlist - List CCD processing information + ccdproc - Process CCD images + ccdtest - CCD test and demonstration package + combine - Combine CCD images + cosmicrays - Detect and replace cosmic rays + darkcombine - Combine and process dark count images + flatcombine - Combine and process flat field images + mkfringecor - Make fringe correction images from sky images + mkillumcor - Make flat field illumination correction images + mkillumflat - Make illumination corrected flat fields + mkskycor - Make sky illumination correction images + mkskyflat - Make sky corrected flat field images +setinstrument - Set instrument parameters + zerocombine - Combine and process zero level images + + ADDITIONAL HELP TOPICS + + ccdred - CCD image reduction package + ccdtypes - Description of the CCD image types + flatfields - Discussion of CCD flat field calibrations + guide - Introductory guide to using the CCDRED package + instruments - Instrument specific data files + subsets - Description of CCD subsets +.ft R +.fi + +Printed copies of the on-line help documentation may be made with the +command + +.ft L + cl> help \fItopic\fL | lprint +.ft R + + In addition to the package documentation for \fBccdred\fR, +\fBlongslit\fR, and \fBapextract\fR there may be specific guides for +certain instruments. These specific guides, called "cookbooks", give +specific examples and parameter values for the CCD data. diff --git a/noao/imred/ccdred/doc/instruments.hlp b/noao/imred/ccdred/doc/instruments.hlp new file mode 100644 index 00000000..95baff37 --- /dev/null +++ b/noao/imred/ccdred/doc/instruments.hlp @@ -0,0 +1,256 @@ +.help instruments Dec93 noao.imred.ccdred + +.ih +NAME +instruments -- Instrument specific data files +.ih +DESCRIPTION +The \fBccdred\fR package has been designed to accommodate many different +instruments, detectors, and observatories. This is done by having +instrument specific data files. Note that by instrument we mean a +combination of detector, instrument, application, and observatory, so +there might be several "instruments" associated with a particular CCD +detector. Creating and maintaining the instrument files is generally +the responsibility of the support staff, though the user may create or +copy and modify his/her own instrument/application specific files. The +task \fBsetinstrument\fR makes this information available to the user +and package easily. + +There are three instrument data files, all of which are optional. The +package may be used without the instrument files but much of the +convenience of the package, particularly with respect to using the CCD +image types, will be lost. The three files are an instrument image +header translation file, an initialization task which mainly sets +default task parameters, and a bad pixel file identifying the cosmic +bad pixels in the detector. These files are generally stored in a +system data directory which is a subdirectory of the logical +directory "ccddb$". Each file has a root name which identifies the +instrument. +.sh +1. Instrument Translation File +The instrument translation file translates the parameter names used by +the \fBccdred\fR pacakge into instrument specific parameters and also +supplies instrument specific default values. The package parameter +\fIccdred.instrument\fR specifies this file to the package. The task +\fBsetinstrument\fR sets this parameter, though it can be set +explicitly like any other parameter. For the standard instrument +translation file the root name is the instrument identification and the +extension is "dat" ("*.dat" files are protected from being removed in a +"stripped" system, i.e. when all nonessential files are removed). +Private instrument files may be given any name desired. + +The instrument translation proceeds as follows. When a package task needs +a parameter for an image, for example "imagetyp", it looks in the instrument +translation file. If the file is not found or none is specified then the +image header keyword that is requested has the same name. If an +instrument translation file is defined then the requested +parameter is translated to an image header keyword, provided a translation +entry is given. If no translation is given the package name is used. For +example the package parameter "imagetyp" might be translated to "data-typ" +(the old NOAO CCD keyword). If the parameter is not found then the default +value specified in the translation file, if present, is returned. For recording +parameter information in the header, such as processing flags, the +translation is also used. The default value has no meaning in this case. +For example, if the flag specifying that the image has been corrected +by a flat field is to be set then the package parameter name "flatcor" +might be translated to "ff-flag". If no translation is given then the +new image header parameter is entered as "flatcor". + +The format of the translation file are lines consisting of the package +parameter name, followed by the image header keyword, followed by the +default value. The first two fields are parameter names. The fields +are separated by whitespace (blanks and tabs). String default values +containing blanks must be quoted. An example is given below. + +.nf + # Sample translation file. + exptime itime + darktime itime + imagetyp data-typ + subset f1pos + biassec biassec [411:431,2:573] + datasec datasec [14:385,2:573] + + fixpix bp-flag 0 + overscan bt-flag 0 + zerocor bi-flag 0 + darkcor dk-flag 0 + flatcor ff-flag 0 + fringcor fr-flag 0 +.fi + +The first comment line is ignored as are blank lines. +The first two lines translate the CCD image type, and the subset parameter +without default values (see \fBccdtypes\fR and \fBsubsets\fR for more +information). The next two lines give the overscan bias strip +section and the data section with default values for the instrument. +Note that these parameters may be overridden in the task \fBccdproc\fR. + +The next set of translations requires further discussion. For processing +flags the package assumes that the absence of a keyword means that the +processing has not been done. If processing is always to be done with +the \fBCCDRED\fR package and no processing keywords are recorded in the raw data +then these parameters should be absent (unless you don't like the names +used by the package). However, for compatibility with the original NOAO +CCD images, which may be processed outside of IRAF and which use 0 as the +no processing value, the processing flags are translated and the false values +are indicated by the default values. + +If there is more than one translation for the same CCDRED parameter, +for example more than one exptime, then the last one is used. + +In addition to the parameter name translations the translation file +contains translations between the value of the image type parameter +and the image types used by the package. These lines +consist of the image header type string as the first field (with quotes +if there are blanks) and the image type as recognized by the package. The +following example will make this clearer. + +.nf + 'OBJECT (0)' object + 'DARK (1)' dark + 'PROJECTOR FLAT (2)' flat + 'SKY FLAT (3)' other + 'COMPARISON LAMP (4)' other + 'BIAS (5)' zero + 'DOME FLAT (6)' flat +.fi + +The values of the image type strings in the header contain blanks so they +are quoted. Also the case of the strings is important. Note that there +are two types of flat field images and three types of object images. + +The CCD image types recognized by the package are: + +.nf + zero - zero level image such as a bias or preflash + dark - dark count image + flat - flat field image + illum - iillumination image such as a sky image + fringe - fringe correction image + object - object image +.fi + +There may be more than one image type that maps to the same package +type. In particular other standard CCD image types, such as comparison +spectra, multiple exposure, standard star, etc., should be mapped to +object or other. There may also be more than one type of flat field, +i.e. dome flat, sky flat, and lamp flat. For more on the CCD image +types see \fBccdtypes\fR. + +The complete set of package parameters are given below. +The package parameter names are generally the same as the +standard image header keywords being adopted by NOAO. + +.nf + General Image Header and Default Parameters + ccdmean darktime exptime fixfile + imagetyp ncombine biassec subset + title datasec nscanrow + + CCDRED Processing Flags + ccdproc darkcor fixpix flatcor + fringcor illumcor overscan trim + zerocor + + CCDRED CCD Image Types + dark flat fringe illum + none object unknown zero +.fi + +The translation mechanism described here may become more +sophisticated in the future and a general IRAF system facility may be +implemented eventually. For the present the translation mechanism is +quite simple. +.sh +2. Instrument Setup Script +The task \fBsetinstrument\fR translates an instrument ID into a +CL script in the instrument directory. This script is then executed. +Generally this script simply sets the task parameters for an +instrument/application. However, it could do anything else the support +staff desires. Below are the first few lines of a typical instrument setup +script. + +.nf + ccdred.instrument = "ccddb$kpno/example.dat" + ccdred.pixeltype = "real" + ccdproc.fixpix = yes + ccdproc.overscan = yes + ccdproc.trim = yes + ccdproc.zerocor = no + ccdproc.darkcor = no + ccdproc.flatcor = yes + ccdproc.biassec = "[411:431,2:573]" + ccdproc.datasec = "[14:385,2:573]" +.fi + +The instrument parameter should always be set unless there is no +translation file for the instrument. The \fBccdproc\fR parameters +illustrate setting the appropriate processing flags for the +instrument. The overscan bias and trim data sections show an alternate +method of setting these instrument specific parameters. They may be +set in the setup script in which case they are given explicitly in the +user parameter list for \fBccdproc\fR. If the value is "image" then +the parameters may be determined either through the default value in +the instrument translation file, as illustrated in the previous +section, or from the image header itself. + +The instrument setup script for setting default task parameters may be +easily created by the support person as follows. Set the package +parameters using \fBeparam\fR or with CL statements. Setting the +parameters might involve testing. When satisfied with the way the +package is set then the parameters may be dumped to a setup script +using the task \fBdparam\fR. The final step is editing this script to +delete unimportant and query parameters. For example, + +.nf + cl> dparam ccdred >> file.cl + cl> dparam ccdproc >> file.cl + cl> dparam combine >> file.cl + ... + cl> ed file.cl +.fi +.sh +3. Instrument Bad Pixel File +The bad pixel file describes the bad pixels, columns, and lines in the +detector which are to be replaced by interpolation when processing the +images. This file is clearly detector specific. The file consists of +lines describing rectangular regions of the image. +The regions are specified by four numbers giving the starting and ending +columns followed by the starting and ending lines. The starting and +ending points may be the same to specify a single column or line. The +example below illustrates a bad pixel file. + +.nf + # RCA1 CCD untrimmed + 25 25 1 512 + 108 108 1 512 + 302 302 403 512 + 1 512 70 70 + 245 246 312 315 +.fi + +If there is a comment line in the file containing the word "untrimmed" +then the coordinates of the bad pixel regions apply to the original CCD +detector coordinates. +If the image has been trimmed and the bad pixels are replaced at a later +stage then this word indicates that the trim region be determined from the +image header and the necessary coordinate conversion made to the original +CCD pixel coordinates. Note that if a subraster readout is used the +coordinates must still refer to the original CCD coordinates and +not the raw, untrimmed readout image. If the word +"untrimmed" does not appear then the coordinates are assumed to apply to +the image directly; i.e. the trimmed coordinates if the image has been +trimmed or the original coordinates if the image has not been trimmed. +The standard bad pixel files should always refer to the original, untrimmed +coordinates. + +The first two bad pixel regions are complete bad columns (the image +is 512 x 512), the next line is a partial bad column, the next line is +a bad line, and the last line is a small bad region. These files are +easy to create, provided you have a good image to work from and a way +to measure the positions with an image or graphics display. +.ih +SEE ALSO +ccdtypes, subsets, setinstrument +.endhelp diff --git a/noao/imred/ccdred/doc/mkfringecor.hlp b/noao/imred/ccdred/doc/mkfringecor.hlp new file mode 100644 index 00000000..797f4d11 --- /dev/null +++ b/noao/imred/ccdred/doc/mkfringecor.hlp @@ -0,0 +1,90 @@ +.help mkfringecor Feb88 noao.imred.ccdred +.ih +NAME +mkfringecor -- Make fringe correction images from sky images +.ih +USAGE +mkfringecor input output +.ih +PARAMETERS +.ls input +List of input images for making fringe correction images. +.le +.ls output +List of output fringe correction images. If none is +specified or if the name is the same as the input image then the output +image replaces the input image. +.le +.ls ccdtype = "" +CCD image type to select from the input images. If none is specified +then all types are used. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed background. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +The input blank sky images are automatically processed up through the +iillumination correction before computing the fringe correction images. +The fringe corrections are subset dependent. +The slowly varying background is determined and subtracted leaving only +the fringe pattern caused by the sky emission lines. These fringe images +are then scaled and subtracted from the observations by \fBccdproc\fR. +The background is determined by heavily smoothing the image using a +moving "boxcar" average. The effects of the objects and fringes in the +image is minimized by using a sigma clipping algorithm to detect and +exclude them from the average. Note, however, that objects left in the +fringe image will affect the fringe corrected observations. Any objects +in the sky image should be removed using \fBskyreplace\fR (not yet +available). + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +To minimize the effects of the fringes and any objects in the blank sky +calibration images a sigma clipping algorithm is used to detect and +exclude features from the background. This is done by computing the +rms of the image lines relative to the smoothed background and +excluding points exceeding the specified threshold factors times the +rms. This is done before each image line is added to the moving +average, except for the first few lines where an iterative process is +used. +.ih +EXAMPLES +1. The two examples below make an fringe correction image from a blank +sky image, "sky017". In the first example a separate fringe +image is created and in the second the fringe image replaces the +sky image. + +.nf + cl> mkskycor sky017 Fringe + cl> mkskycor sky017 frg017 +.fi +.ih +SEE ALSO +ccdproc +.endhelp diff --git a/noao/imred/ccdred/doc/mkillumcor.hlp b/noao/imred/ccdred/doc/mkillumcor.hlp new file mode 100644 index 00000000..0effd7a2 --- /dev/null +++ b/noao/imred/ccdred/doc/mkillumcor.hlp @@ -0,0 +1,92 @@ +.help mkillumcor Oct88 noao.imred.ccdred +.ih +NAME +mkillumcor -- Make flat field iillumination correction images +.ih +USAGE +mkillumcor input output +.ih +PARAMETERS +.ls input +List of input images for making flat field iillumination correction images. +.le +.ls output +List of output flat field iillumination correction images. If none is +specified or if the name is the same as the input image then the output +image replaces the input image. +.le +.ls ccdtype = "flat" +CCD image type to select from the input images. If none is specified +then all types are used. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude deviant points from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed iillumination. +.le +.ls divbyzero = 1. +The iillumination correction is the inverse of the smoothed flat field. +This may produce division by zero. A warning is given if division +by zero takes place and the result (the iillumination correction value) +is replaced by the value of this parameter. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +First, the input flat field images are automatically processed if +needed. Then, the large scale iillumination pattern of the images is +determined by heavily smoothing them using a moving "boxcar" average. +The iillumination correction, the inverse of the iillumination pattern, +is applied by \fBccdproc\fR to CCD images to remove the iillumination +pattern introduced by the flat field. The combination of the flat +field calibration and the iillumination correction based on the flat +field is equivalent to removing the iillumination from the flat field +(see \fBmkillumflat\fR). This two step calibration is generally used +when the observations have been previously flat field calibrated. This +task is closely related to \fBmkskycor\fR which determines the +iillumination correction from a blank sky image; this is preferable to +using the iillumination from the flat field as it corrects for the +residual iillumination error. For a general discussion of the options +for flat fields and iillumination corrections see \fBflatfields\fR. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +To minimize the effects of bad pixels a sigma clipping algorithm is +used to detect and reject these pixels from the iillumination. This is +done by computing the rms of the image lines relative to the smoothed +iillumination and excluding points exceeding the specified threshold +factors times the rms. This is done before each image line is added to +the moving average, except for the first few lines where an iterative +process is used. +.ih +EXAMPLES +1. The example below makes an iillumination correction image from the +flat field image, "flat017". + + cl> mkillumcor flat017 Illum +.ih +SEE ALSO +ccdproc, flatfields, mkillumflat, mkskycor, mkskyflat +.endhelp diff --git a/noao/imred/ccdred/doc/mkillumflat.hlp b/noao/imred/ccdred/doc/mkillumflat.hlp new file mode 100644 index 00000000..8288fb85 --- /dev/null +++ b/noao/imred/ccdred/doc/mkillumflat.hlp @@ -0,0 +1,101 @@ +.help mkillumflat Oct88 noao.imred.ccdred +.ih +NAME +mkillumflat -- Make illumination corrected flat fields +.ih +USAGE +mkillumflat input output +.ih +PARAMETERS +.ls input +List of input flat field images to be illumination corrected. +.le +.ls output +List of output illumination corrected flat field images. +If none is specified or if the name is the same as the +input image then the output image replaces the input image. +.le +.ls ccdtype = "flat" +CCD image type to select from the input images. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed illumination. +.le +.ls divbyzero = 1. +The illumination flat field is the ratio of the flat field to a +smoothed flat field. This may produce division by zero. A warning is +given if division by zero takes place and the result (the illumination +corrected flat field value) is replaced by the value of this +parameter. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +First, the input flat field images are processed as needed. Then the +large scale illumination pattern of the images is removed. The +illumination pattern is determined by heavily smoothing the image using +a moving "boxcar" average. The output image is the ratio of the input +image to the illumination pattern. The illumination pattern is +normalized by its mean to preserve the mean level of the input image. + +When this task is applied to flat field images only the small scale +response effects are retained. This is appropriate if the flat field +images have illumination effects which differ from the astronomical +images and blank sky images are not available for creating sky +corrected flat fields. When a high quality blank sky image is +available the related task \fBmkskyflat\fR should be used. Note that +the illumination correction, whether from the flat field or a sky +image, may be applied as a separate step by using the task +\fBmkillumcor\fR or \fBmkskycor\fR and applying the illumination +correction as a separate operation in \fBccdproc\fR. However, creating +an illumination corrected flat field image before processing is more +efficient since one less operation per image processed is needed. For +more discussion about flat fields and illumination corrections see +\fBflatfields\fR. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +To minimize the effects of bad pixels a sigma clipping algorithm is +used to detect and reject these pixels from the illumination. This is +done by computing the rms of the image lines relative to the smoothed +illumination and excluding points exceeding the specified threshold +factors times the rms. This is done before each image line is added to +the moving average, except for the first few lines where an iterative +process is used. +.ih +EXAMPLES +1. Two examples in which a new image is created and in which the +input flat fields are corrected in place are: + +.nf + cl> mkllumflat flat004 FlatV + cl> mkillumflat flat* "" +.fi +.ih +SEE ALSO +ccdproc, flatfields, mkfringecor, mkillumcor, mkskycor, mkskyflat +.endhelp diff --git a/noao/imred/ccdred/doc/mkskycor.hlp b/noao/imred/ccdred/doc/mkskycor.hlp new file mode 100644 index 00000000..15cfacf6 --- /dev/null +++ b/noao/imred/ccdred/doc/mkskycor.hlp @@ -0,0 +1,103 @@ +.help mkskycor Feb88 noao.imred.ccdred +.ih +NAME +mkskycor -- Make sky iillumination correction images +.ih +USAGE +mkskycor input output +.ih +PARAMETERS +.ls input +List of input images for making sky iillumination correction images. +.le +.ls output +List of output flat field iillumination correction images. If none is +specified or if the name is the same as the input image then the output +image replaces the input image. +.le +.ls ccdtype = "" +CCD image type to select from the input images. If none is specified +then all types are used. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed iillumination. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +The large scale iillumination pattern of the input images, generally +blank sky calibration images, is determined by heavily smoothing +the image using a moving "boxcar" average. The effects of objects in +the image may be minimized by using a sigma clipping algorithm to +detect and exclude the objects from the average. This +iillumination image is applied by \fBccdproc\fR to CCD images to remove +the iillumination pattern. + +The input images are automatically processed up through flat field +calibration before computing the iillumination. The iillumination +correction is that needed to make the processed images flat +over large scales. The input images are generally blank sky calibration +images which have the same iillumination and instrumental effects +as the object observations. Object images may be used but removal +of the objects may not be very good; particularly large, bright objects. +For further discussion of flat fields and iillumination corrections +see \fBflatfields\fR. + +You will notice that when you process images with an iillumination +correction you are dividing each image by a flat field calibration and +an iillumination correction. If the iillumination corrections are not +done as a later step but at the same time as the rest of the processing +one will get the same calibration by multiplying the flat field by the +iillumination correction and using this product alone as the flat +field. This approach has the advantage of one less calibration image +and two less computations (scaling and dividing the iillumination +correction). Such an image, called a \fIsky flat\fR, may be created by +\fBmkskyflat\fR as an alternative to this task. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +Blank sky images may not be completely blank so a sigma clipping +algorithm may be used to detect and exclude objects from the +iillumination pattern. This is done by computing the rms of the image +lines relative to the smoothed background and excluding points +exceeding the specified threshold factors times the rms. This is done +before each image line is added to the moving average, except for the +first few lines where an iterative process is used. +.ih +EXAMPLES +1. The two examples below make an iillumination image from a blank sky image, +"sky017". In the first example a separate iillumination image is created +and in the second the iillumination image replaces the sky image. + +.nf + cl> mkskycor sky017 Illum + cl> mkskycor sky017 sky017 +.fi +.ih +SEE ALSO +ccdproc, flatfields, mkillumcor, mkillumflat, mkskyflat +.endhelp diff --git a/noao/imred/ccdred/doc/mkskyflat.hlp b/noao/imred/ccdred/doc/mkskyflat.hlp new file mode 100644 index 00000000..d28e2301 --- /dev/null +++ b/noao/imred/ccdred/doc/mkskyflat.hlp @@ -0,0 +1,110 @@ +.help mkskyflat Feb88 noao.imred.ccdred +.ih +NAME +mkskyflat -- Make sky corrected flat field images +.ih +USAGE +mkskyflat input output +.ih +PARAMETERS +.ls input +List of blank sky images to be used to create sky corrected flat field +calibration images. +.le +.ls output +List of output sky corrected flat field calibration images (called +sky flats). If none is specified or if the name is the same as the +input image then the output image replaces the input image. +.le +.ls ccdtype = "" +CCD image type to select from the input images. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed iillumination. +.le +.ls ccdproc (pset) +CCD processing parameter set. +.le +.ih +DESCRIPTION +A sky corrected flat field calibration image, called a sky flat, is a +flat field that when applied to observations of the sky have no large +scale gradients. Flat field images are generally obtained by exposures +to lamps either illuminating the telescope field or a surface in the dome +at which the telescope is pointed. Because the detector is not illuminated +in the same way as an observation of the sky there may be large +scale iillumination patterns introduced into the observations with such +a flat field. To correct this type of flat field a blank sky observation +(which has been divided by the original flat field) is heavily smoothed +to remove the noise leaving only the residual large scale iillumination +pattern. This iillumination pattern is divided into the original flat +field to remove this residual. + +The advantage of creating a sky flat field is that when processing +the observations no additional operations are required. However, +if the observations have already been processed with the original +flat field then the residual iillumination pattern of blank sky +calibration images may be created as an iillumination correction +to be applied by \fBccdproc\fR. Such a correction is created by the +task \fBmkskycor\fR. If a good blank sky image is not +available then it may be desirable to remove the iillumination pattern +of the flat field image using \fBmkillumflat\fR or \fBmkillumcor\fR +provided the sky observations are truly uniformly illuminated. +For more on flat fields and iillumination corrections see \fBflatfields\fR. + +The input, blank sky images are first processed, based on the +\fBccdproc\fR parameters, if needed. These parameters also determine +the flat field image to be used in making the sky flat. The residual +iillumination pattern is determined by heavily smoothing the image using +a moving "boxcar" average. The effects of objects in the input image +may be minimized by using a sigma clipping algorithm to detect and +exclude the objects from the average. The output image is ratio of the +flat field image, for the same subset as the input image, to the +residual iillumination pattern determined from the processed blank sky +input image. The iillumination pattern is normalized by its mean to +preserve the mean level of the flat field image. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +Blank sky images may not be completely blank so a sigma clipping +algorithm may be used to detect and exclude objects from the +iillumination pattern. This is done by computing the rms of the image +lines relative to the smoothed background and excluding points +exceeding the specified threshold factors times the rms. This is done +before each image line is added to the moving average, except for the +first few lines where an iterative process is used. +.ih +EXAMPLES +1. Two examples in which a new image is created and in which the +input sky images are converted to sky flats are: + +.nf + cl> mkskyflat sky004 Skyflat + cl> mkskyflat sky* "" +.fi +.ih +SEE ALSO +ccdproc, flatfields, mkfringecor, mkillumcor, mkillumflat, mkskycor +.endhelp diff --git a/noao/imred/ccdred/doc/setinstrument.hlp b/noao/imred/ccdred/doc/setinstrument.hlp new file mode 100644 index 00000000..410dd20f --- /dev/null +++ b/noao/imred/ccdred/doc/setinstrument.hlp @@ -0,0 +1,97 @@ +.help setinstrument Oct87 noao.imred.ccdred +.ih +NAME +setinstrument -- Set instrument parameters +.ih +USAGE +setinstrument instrument +.ih +PARAMETERS +.ls instrument +Instrument identification for instrument parameters to be set. If '?' +then a list of the instrument identifiers is printed. +.le +.ls site = "kpno" +Site ID. +.le +.ls directory = "ccddb$" +Instrument directory containing instrument files. The instrument files +are found in the subdirectory given by the site ID. +.le +.ls review = yes +Review the instrument parameters? If yes then \fBeparam\fR is run for +the parameters of \fBccdred\fR and \fBccdproc\fR. +.le +.ls query +Parameter query if initial instrument is not found. +.le +.ih +DESCRIPTION +The purpose of the task is to allow the user to easily set default +parameters for a new instrument. The default parameters are generally +defined by support personal in an instrument directory for a particular +site. The instrument directory is the concatenation of the specified +directory and the site. For example if the directory is "ccddb$" and +the site is "kpno" then the instrument directory is "ccddb$kpno/". +The user may have his own set of instrument files in a local directory. +The current directory is used by setting the directory and site to the +null string (""). + +The user specifies an instrument identifier. This instrument may +be specific to a particular observatory, telescope, instrument, and +detector. If the character '?' is specified or the instrument file is +not found then a list of instruments +in the instrument directory is produced by paging the file "instruments.men". +The task then performs the following operations: +.ls (1) +If an instrument translation file with the name given by the instrument +ID and the extension ".dat" is found then the instrument translation +file parameter, \fIccdred.instrument\fR, is set to this file. +If it does not exist then the user is queried again. Note that a +null instrument, "", is allowed to set no translation file. +.le +.ls (2) +If an instrument setup script with the name given by the instrument ID +and the extension ".cl" is found then the commands in the file are +executed (using the command \fIcl < script\fR. This script generally +sets default parameters. +.le +.ls (3) +If the review flag is set the task \fBeparam\fR is run to allow the user +to examine and modify the parameters for the package \fBccdred\fR and task +\fBccdproc\fR. +.le +.ih +EXAMPLES +1. To get a list of the instruments; + +.nf + cl> setinstrument ? + [List of instruments] + +2. To set the instrument and edit the processing parameters: + + cl> setinstrument ccdlink + [Edit CCDRED parameters] + [Edit CCDPROC parameters] + +3. To use your own instrument translation file and/or setup script in +your working directory. + + cl> setinst.site="" + cl> setinst.dir="" + cl> setinst myinstrument + +To make these files see help under \fBinstruments\fR. Copying and modifying +system files is also straightforward. + + cl> copy ccddb$kpno/fits.dat . + cl> edit fits.dat + cl> setinst.site="" + cl> setinst.dir="" + cl> setinst fits +.fi +.ih +SEE ALSO +instruments, ccdred, ccdproc +.endhelp diff --git a/noao/imred/ccdred/doc/subsets.hlp b/noao/imred/ccdred/doc/subsets.hlp new file mode 100644 index 00000000..78aafb01 --- /dev/null +++ b/noao/imred/ccdred/doc/subsets.hlp @@ -0,0 +1,99 @@ +.help subsets Jun87 noao.imred.ccdred +.ih +NAME +subsets -- Description of CCD subsets +.ih +DESCRIPTION +The \fBccdred\fR package groups observation into subsets. +The image header parameter used to identify the subsets is defined +in the instrument translation file (see help for \fBinstruments\fR). +For example to select subsets by the header parameter "filters" the +instrument translation file would contain the line: + + subset filters + +Observations are generally grouped into subsets based on a common +instrument configuration such as a filter, aperture mask, +grating setting, etc. This allows combining images from several +different subsets automatically and applying the appropriate +flat field image when processing the observations. For example +if the subsets are by filter then \fBflatcombine\fR will search +through all the images, find the flat field images (based on the +CCD type parameter), and combine the flat field images from +each filter separately. Then when processing the images the +flat field with the same filter as the observation is used. + +Each subset is assigned a short identifier. This is listed when +using \fBccdlist\fR and is appended to a root name when combining +images. Because the subset parameter in the image header may be +any string there must be a mapping applied to generate unique +identifiers. This mapping is defined in the file given by +the package parameter \fIccdred.ssfile\fR. The file consists of +lines with two fields (except that comment lines may be included +as a line by itself or following the second field): + + 'subset string' subset_id + +where the subset string is the image header string and the subset_id is +the identifier. A field must be quoted if it contains blanks. The +user may create this file but generally it is created by the tasks. The +tasks use the first word of the subset string as the default identifier +and a number is appended if the first word is not unique. The +following steps define the subset identifier: + +.ls (1) +Search the subset file, if present, for a matching subset string and +use the defined subset identifier. +.le +.ls (2) +If there is no matching subset string use the first word of the +image header subset string and, if it is not unique, +add successive integers until it is unique. +.le +.ls (3) +If the identifier is not in the subset file create the file and add an +entry if necessary. +.le +.ih +EXAMPLES +1. The subset file is "subsets" (the default). The subset parameter is +translated to "f1pos" in the image header (the old NOAO CCD parameter) +which is an integer filter position. After running a task, say +"ccdlist *.imh" to cause all filters to be checked, the subset file contains: + +.nf + '2' 2 + '5' 5 + '3' 3 +.fi + +The order reflects the order in which the filters were encountered. +Suppose the user wants to have more descriptive names then the subset +file can be created or edited to the form: + +.nf + # Sample translation file. + '2' U + '3' B + '4' V +.fi + +(This is only an example and does not mean these are standard filters.) + +2. As another example suppose the image header parameter is "filter" and +contains more descriptive strings. The subset file might become: + +.nf + 'GG 385 Filter' GG + 'GG 495 Filter' GG1 + 'RG 610 Filter' RG + 'H-ALPHA' H_ALPHA +.fi + +In this case use of the first word was not very good but it is unique. +It is better if the filters are encoded with the thought that the first +word will be used by \fBccdred\fR; it should be short and unique. +.ih +SEE ALSO +instruments +.endhelp diff --git a/noao/imred/ccdred/doc/zerocombine.hlp b/noao/imred/ccdred/doc/zerocombine.hlp new file mode 100644 index 00000000..1646ea9c --- /dev/null +++ b/noao/imred/ccdred/doc/zerocombine.hlp @@ -0,0 +1,121 @@ +.help zerocombine Aug91 noao.imred.ccdred +.ih +NAME +zerocombine -- Combine and process zero level images +.ih +USAGE +zerocombine input +.ih +PARAMETERS +.ls input +List of zero level images to combine. The \fIccdtype\fR parameter +may be used to select the zero level images from a list containing all +types of data. +.le +.ls output = "Zero" +Output zero level root image name. +.le +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "minmax" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation. See \fBcombine\fR for details. +.le +.ls ccdtype = "zero" +CCD image type to combine. If no image type is given then all input images +are combined. +.le +.ls process = no +Process the input images before combining? +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le +.ls scale = "none" (none|mode|median|mean|exposure) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, or scale +by the exposure time given in the image header. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling. +If no section is given then the entire region of the image is +sampled (for efficiency the images are sampled if they are big enough). +.le + +.ce +Algorithm Parameters +.ls nlow = 0, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See \fBcombine\fR for further details. +.le +.ls blank = 0. +Output value to be used when there are no pixels. +.le +.ih +DESCRIPTION +The zero level images in the input image list are combined. +The input images may be processed first if desired. +The original images may be deleted automatically if desired. +The output pixel datatype will be real. + +This task is a script which applies \fBccdproc\fR and \fBcombine\fR. The +parameters and combining algorithms are described in detail in the help for +\fBcombine\fR. This script has default parameters specifically set for +zero level images and simplifies the combining parameters. There are other +combining options not included in this task. For these additional +features, such as thresholding, offseting, masking, and projecting, use +\fBcombine\fR. +.ih +EXAMPLES +1. The image data contains four zero level images. +To automatically select them and combine them as a background job +using the default combining algorithm: + + cl> zerocombine ccd*.imh& +.ih +SEE ALSO +ccdproc, combine +.endhelp |