diff options
author | Joseph Hunkeler <jhunkeler@gmail.com> | 2015-07-08 20:46:52 -0400 |
---|---|---|
committer | Joseph Hunkeler <jhunkeler@gmail.com> | 2015-07-08 20:46:52 -0400 |
commit | fa080de7afc95aa1c19a6e6fc0e0708ced2eadc4 (patch) | |
tree | bdda434976bc09c864f2e4fa6f16ba1952b1e555 /noao/imred/quadred/src/quad/doc | |
download | iraf-linux-fa080de7afc95aa1c19a6e6fc0e0708ced2eadc4.tar.gz |
Initial commit
Diffstat (limited to 'noao/imred/quadred/src/quad/doc')
32 files changed, 9176 insertions, 0 deletions
diff --git a/noao/imred/quadred/src/quad/doc/Geometry.fig b/noao/imred/quadred/src/quad/doc/Geometry.fig new file mode 100644 index 00000000..429d987f --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/Geometry.fig @@ -0,0 +1,91 @@ +# The following diagrams show the CCD geometry for the different readout modes. +# +# Single readout: +# G +# <--------------------------------> +# A F +# <----------------------><--------> +# B C D E +# <-> <--><> <> +# ^ ^ +-----------------------+--------+ ^ ^ +# | | | | | | e | +# | c | | |--------| v | +# | v | +----------------+ || || | +# | | | | || o || | +# | | | | || v || | +# | | | | || e || | +# a | | | | || r || | f +# | | | | || s || | +# | | | | || c || | +# | | | | || a || | +# | | +----------------+ || n || | +# | ^ | |--------| | +# | b | | | | ^ | +# | | | | | | d | +# v V *-----------------------+--------+ v v +# +# Quad readout (single frame): +# G' +# <---------------------------------> +# A/2 F/2 F/2 A/2 +# <---------><---><---><-----------> +# B D 2*E D C +# <-> <> < > <> <--> +# ^ ^ *----------+----.----+------------* ^ ^ +# | | | | . | | | e | +# | c | | |----.----| | v | +# | v | +-------+| |.| |+--------+ | | +# a/2 | | | ||o |.|o || | | | f/2 +# | | | 3 ||v |.|v || 4 | | | +# | | | || |.| || | | | +# | | | ||3 |.|4 || | | | +# V ................................... v +# ^ | | ||o |.|o || | | ^ +# | | | 1 ||v |.|v || 2 | | | +# | | | || |.| || | | | +# a/2 | | +-------+|1 |.|2 |+--------+ | | f/2 +# | ^ | || |.| || | | +# | b | | |----.----| | ^ | +# | | | | . | | | d | +# v v *----------+----.----+------------* v V +# +# +# Quad readout (four frames): +# +# G" G" +# <--------------> <----------------> +# A/2 F/2 F/2 A/2 +# <---------><----> <----><----------> +# B D E E D C +# <-> <> <> <> <> <--> +# ^ ^ *----------+----+ +----+------------* ^ ^ +# | | | | | | | | | e | +# | c | | |----| |----| | v | +# a/2 | v | +-------+| || || |+--------+ | ^ | +# | | | ||o || ||o || | | | | f/2 +# | | | 3 ||v || ||v || 4 | | | | +# | | | || || || || | | | | +# V | | ||3 || ||4 || | | | | +# +---------------+ +-----------------+ v v +# +# +# +---------------+ +-----------------+ ^ ^ +# ^ | | ||o || ||o || | | | | +# | | | 1 ||v || ||v || 2 | | | | +# | | | || || || || | | | | +# | | +-------+|1 || ||2 |+--------+ | | | f/2 +# a/2 | ^ | || || || || | v | +# | b | | |----| |----| | ^ | +# | | | | | | | | | d | +# v v *----------+----+ +----+------------* v v +# +# Where +# A = xdata a = ydata +# B = txskip1 b = tyskip1 +# C = txskip2 c = tyskip2 +# D = bxskip1 d = byskip1 +# E = bxskip2 e = byskip2 +# F = xover f = yover = a +# G = A + F +# G' = A + F +# G" = (A + F) / 2 diff --git a/noao/imred/quadred/src/quad/doc/badpiximage.hlp b/noao/imred/quadred/src/quad/doc/badpiximage.hlp new file mode 100644 index 00000000..46e13160 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/badpiximage.hlp @@ -0,0 +1,51 @@ +.help badpiximage Jun87 noao.imred.ccdred +.ih +NAME +badpiximage -- Create a bad pixel mask image from a bad pixel file +.ih +USAGE +badpiximage fixfile template image +.ih +PARAMETERS +.ls fixfile +Bad pixel file. +.le +.ls template +Template image used to define the size of the bad pixel mask image. +.le +.ls image +Bad pixel mask image to be created. +.le +.ls goodvalue = 1 +Integer value assigned to the good pixels. +.le +.ls badvalue = 0 +Integer value assigned to the bad pixels. +.le +.ih +DESCRIPTION +A bad pixel mask image is created from the specified bad pixel file. +The format of the bad pixel file is that used by \fBccdproc\fR to +correct CCD defects (see instruments). The bad pixel image is of pixel type short and +has the value given by the parameter \fBgoodvalue\fR for the good +pixels and the value given by the parameter \fBbadvalue\fR for the bad pixels. +The image size and header parameters are taken from the specified +template image. The bad pixel mask image may be used to view the +location of the bad pixels and blink against an data image using an +image display, to mask or flag bad pixels later by image arithmetic, +and to propagate the positions of the bad pixels through the +reductions. +.ih +EXAMPLES +1. To make a bad pixel mask image from the bad pixel file "cryocambp.dat" +using the image "ccd005" as the template: + + cl> badpiximage cryocambp.dat ccd005 cryocambp + +2. To make the bad pixel mask image with good values of 0 and bad values of 1: + + cl> badpixim cryomapbp.dat ccd005 cryocambp good=0 bad=1 +.ih +SEE ALSO +ccdproc, instruments +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdgeometry.hlp b/noao/imred/quadred/src/quad/doc/ccdgeometry.hlp new file mode 100644 index 00000000..c01a09c8 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdgeometry.hlp @@ -0,0 +1,70 @@ +.help ccdgeometry Sep87 noao.imred.ccdred +.ih +NAME +ccdgeometry - Discussion of CCD geometry and header parameters +.ih +DESCRIPTION +The \fBccdred\fR package maintains and updates certain geometry +information about the images. This geometry is described by four image +header parameters which may be present. These are defined below by the +parameter names used in the package. Note that these names may be +different in the image header using the image header translation +feature of the package. + +.ls DATASEC +The section of the image containing the CCD data. If absent the +entire image is assumed to be data. Only the pixels within the +data section are modified during processing. Therefore, there may be +additional calibration or observation information in the image. +If after processing, the data section is the entire image it is +not recorded in the image header. +.le +.ls CCDSEC +The section of the CCD to corresponding to the data section. This +refers to the physical format, columns and lines, of the detector. This is +the coordinate system used during processing to relate calibration +data to the image data; i.e. image data pixels are calibrated by +calibration pixels at the same CCD coordinates regardless of image pixel +coordinates. This allows recording only parts of the CCD during data +taking and calibrating with calibration frames covering some or all of +the CCD. The CCD section is maintained during trimming operations. +Note that changing the format of the images by image operators outside +of the \fBccdred\fR package will invalidate this coordinate system. +The size of the CCD section must agree with that of the data section. +If a CCD section is absent then it defaults to the data section such +that the first pixel of the data section has CCD coordinate (1,1). +.le +.ls BIASSEC +The section of the image containing prescan or overscan bias information. +It consists of a strip perpendicular to the readout axis. There may be +both a prescan and overscan but the package currently only uses one. +This parameter may be overridden during processing by the parameter +\fIccdproc.biassec\fR. +.le +.ls TRIMSEC +The section of the image extracted during processing when the trim +operation is selected (\fIccdproc.trim\fR). If absent when the trim +operation is selected it defaults to the data section; i.e. the processed +image consists only of the data section. This parameter may be overridden +during processing by the parameter \fIccdproc.trimsec\fR. After trimming +this parameter, if present, is removed from the image header. The +CCD section, data section, and bias section parameters are also modified +by trimming. +.le + +The geometry is as follows. When a CCD image is recorded it consists +of a data section corresponding to part or all of the CCD detector. +Regions outside of the data section may contain additional information +which are not affected except by trimming. Most commonly this consists +of prescan and overscan bias data. When recording only part of the +full CCD detector the package maintains information about that part and +correctly applies calibrations for that part of the detector. Also any +trimming operation updates the CCD coordinate information. If the +images include the data section, bias section, trim section, and ccd +section the processing may be performed entirely automatically. + +The sections are specified using the notation [c1:c2,l1:l2] where c1 +and c2 are the first and last columns and l1 and l2 are the first and +last lines. Currently c1 and l1 must be less than c2 and l2 +respectively and no subsampling is allowed. This may be added later. +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdgroups.hlp b/noao/imred/quadred/src/quad/doc/ccdgroups.hlp new file mode 100644 index 00000000..48c29b99 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdgroups.hlp @@ -0,0 +1,163 @@ +.help ccdgroups Jun87 noao.imred.ccdred +.ih +NAME +ccdgroups -- Group CCD images into image lists +.ih +USAGE +ccdgroups images output +.ih +PARAMETERS +.ls images +List of CCD images to be grouped. +.le +.ls output +Output root group filename. The image group lists will be put in files +with this root name followed by a number. +.le +.ls group = "ccdtype" +Group type. There are currently four grouping types: +.ls ccdtype +Group by CCD image type. +.le +.ls subset +Group by subset parameter. +.le +.ls position +Group by position in right ascension (in hours) and declination (in degrees). +The groups are defined by a radius parameter (in arc seconds). +.le +.ls title +Group by identical titles. +.le +.ls date +Group by identical dates. +.le +.le +.ls radius = 60. +Grouping radius when grouping by positions. This is given in arc seconds. +.le +.ls ccdtype = "" +CCD image types to select from the input image list. If null ("") then +all image types are used. +.le +.ih +DESCRIPTION +The input images, possible restricted to a particular CCD image type, +are grouped into image lists. The "ccdtype" or "subset" groups +produce output image lists with the given root name and the CCD type +or subset as an extension (without a period). For the other group +types the +image lists have file names given by +the root output name and a numeric extension (without a period). +If the package parameter \fIccdred.verbose\fR is yes then the +image name and output group list is printed for each image. The image lists can +be used with the @ list feature for processing separate groups of observations. +Note that grouping by CCD image type and subset is often not necessary since +the \fBccdred\fR tasks automatically use this information (see +\fBccdtypes\fR and \fBsubsets\fR). + +Besides CCD image type and subsets there are currently three ways to +group images. These are by position in the sky, by title, and by +date. Further groups may be added as suggested. The title grouping is +useful if consistent titles are used when taking data. The date +grouping is useful if multiple nights of observations are not organized +by directories (it is recommended that data from separate nights be +kept in separate directories). The position grouping finds +observations within a given radius on the sky of the first member of +the group (this is not a clustering algorithm). The right ascension +and declination coordinates must be in standard units, hours and +degrees respectively. The grouping radius is in arc seconds. This +grouping type is useful for making sets of data in which separate +calibration images are taken at each position. + +The date, title, and coordinates are accessed through the instrument +translation file. The standard names used are "date-obs", "title", "ra", +and "dec". +.ih +EXAMPLES +1. For each object 5 exposures were taken to be combined in order to remove +cosmic rays. If the titles are the same then (with ccdred.verbose=yes): + +.nf + cl> ccdgroups *.imh group group=title ccdtype=object + ccd005.imh --> group1 + ccd006.imh --> group1 + ccd007.imh --> group1 + ccd008.imh --> group1 + ccd009.imh --> group1 + ccd012.imh --> group2 + ccd013.imh --> group2 + ccd014.imh --> group2 + ccd015.imh --> group2 + ccd016.imh --> group2 + [... etc ...] + cl> combine @group1 obj1 proc+ + cl> combine @group2 obj2 proc+ + [... etc ...] +.fi + +Note the numeric suffixes to the output root name "group". + +2. CCD observations were made in groups with a flat field, the object, and +a comparison spectrum at each position. To group and process this data: + +.nf + cl> ccdgroups *.imh obs group=position >> logfile + cl> ccdproc @obs1 + cl> ccdproc @obs2 + cl> ccdproc @obs3 +.fi + +Since no flat field is specified for the parameter \fIccdproc.flat\fR +the flat field is taken from the input image list. + +3. If for some reason you want to group by date and position it is possible +to use two steps. + +.nf + cl> ccdgroups *.imh date group=date + cl> ccdgroups @data1 pos1 + cl> ccdgroups @data2 pos2 +.fi + +4. To get groups by CCD image type: + +.nf + cl> ccdgroups *.imh "" group=ccdtype + ccd005.imh --> zero + ccd006.imh --> zero + ccd007.imh --> zero + ccd008.imh --> dark + ccd009.imh --> flat + ccd012.imh --> flat + ccd013.imh --> object + ccd014.imh --> object + ccd015.imh --> object + ccd016.imh --> object + [... etc ...] +.fi + +Note the use of a null root name and the extension is the standard +CCDRED types (not necessarily those used in the image header). + +5. To get groups by subset: + +.nf + cl> ccdgroups *.imh filt group=subset + ccd005.imh --> filt + ccd006.imh --> filtB + ccd007.imh --> filtB + ccd008.imh --> filtB + ccd009.imh --> filtV + ccd012.imh --> filtV + ccd013.imh --> filtV + ccd014.imh --> filtB + ccd015.imh --> filtB + ccd016.imh --> filtB + [... etc ...] +.fi + +.ih +SEE ALSO +ccdlist, ccdtypes, instruments, subsets +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdhedit.hlp b/noao/imred/quadred/src/quad/doc/ccdhedit.hlp new file mode 100644 index 00000000..1bc27d29 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdhedit.hlp @@ -0,0 +1,108 @@ +.help ccdhedit Jun87 noao.imred.ccdred +.ih +NAME +ccdhedit -- CCD image header editor +.ih +USAGE +ccdhedit images parameter value +.ih +PARAMETERS +.ls images +List of CCD images to be edited. +.le +.ls parameter +Image header parameter. The image header parameter will be translated by +the header translation file for the images. +.le +.ls value +The parameter value. If the null string ("") is specified then the +parameter is deleted from the image header, otherwise it is added or +modified. If the parameter is "imagetyp" then the value string giving +the CCD image type is translated from the package CCD type to the +instrument specific string. +.le +.ls type = "string" +The parameter type. The parameter types are "string", "real", or "integer". +.le +.ih +DESCRIPTION +The image headers of the specified CCD images are edited to add, modify, +or delete a parameter. The parameters may be those used by the \fBccdred\fR +package. The parameter name is translated to an image header parameter by the +instrument translation file (see \fBinstruments\fR) if a translation is +given. Otherwise the parameter is that in the image header. If the parameter +is "imagetyp" the parameter value for the CCD image type may be that +used by the package; i.e. dark, object, flat, etc. The value string will be +translated to the instrument image string in this case. The translation +facility allows use of this task in an instrument independent way. + +The value string is used to determine whether to delete or modify the +image parameter. If the null string, "", is given the specified parameter +is deleted. If parameters are added the header type must be specified +as a string, real, or integer parameter. The numeric types convert the +value string to a number. +.ih +EXAMPLES +The \fBccdred\fR package is usable even with little image header information. +However, if desired the header information can be added to images which +lack it. In all the examples the parameters used are those of the package +and apply equally well to any image header format provided there is an +instrument translation file. + +.nf +1. cl> ccdhedit obj* imagetyp object +2. cl> ccdhedit flat* imagetyp flat +3. cl> ccdhedit zero* imagetyp zero +4. cl> ccdhedit obj0![1-3]* subset "V filter" +5. cl> ccdhedit obj0![45]* subset "R filter" +6. cl> ccdhedit flat001 subset "R filter" +7. cl> ccdhedit obj* exptime 500 type=integer +.fi + +8. The following is an example of a CL script which sets the CCD image type, +the subset, and the exposure time simultaneously. The user may expand +on this example to include other parameters or other initialization +operations. + +.nf + cl> edit ccdheader.cl + + ---------------------------------------------------------------- + # Program to set CCD header parameters. + + procedure ccdheader (images) + + string images {prompt="CCD images"} + string imagetyp {prompt="CCD image type"} + string subset {prompt="CCD subset"} + string exptime {prompt="CCD exposure time"} + + begin + string ims + + ims = images + ccdhedit (ims, "imagetyp", imagetyp, type="string") + ccdhedit (ims, "subset", subset, type="string") + ccdhedit (ims, "exptime", exptime, type="real") + end + ---------------------------------------------------------------- + + cl> task ccdheader=ccdheader.cl + cl> ccdheader obj* imagetyp=object subset="V" exptime=500 +.fi + +9. The image header may be changed to force processing a calibration image +as an object. For example to flatten a flat field: + +.nf + cl> ccdhedit testflat imagetyp other + cl> ccdproc testflat +.fi + +10. To delete processing flags: + + cl> ccdhedit obj042 flatcor "" +.ih +SEE ALSO +hedit, instruments, ccdtypes, subsets +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdinst.hlp b/noao/imred/quadred/src/quad/doc/ccdinst.hlp new file mode 100644 index 00000000..23ebea60 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdinst.hlp @@ -0,0 +1,389 @@ +.help ccdinstrument Nov90 noao.imred.ccdred +.ih +NAME +ccdinstrument -- Setup and verify CCD instrument translation files +.ih +USAGE +ccdinstrument images +.ih +PARAMETERS +.ls images +List of images to be verified or used to setup a CCD instrument translation +file. +.le +.ls instrument = ")_.instrument" +CCD instrument translation file. The default is to use the translation +file defined in the \fBccdred\fR package parameters. Note that one would +need write permission to update this file though the task has a write +command to save any changes to a different file. +.le +.ls ssfile = ")_.ssfile" +Subset translation file. The default is to use the file defined in +the \fBccdred\fR package parameters. +.le +.ls edit = yes +Edit the instrument translation file? If "yes" an interactive +mode is entered allowing translation parameters to be modified while if +"no" the task is simply used to verify the translations noninteractively. +.le +.ls parameters = "basic" +Parameters to be displayed. The choices are "basic" to display only the +most basic parameters (those needed for the simplest automation of +\fBccdred\fR tasks), "common" to display the common parameters used +by the package (most of these are keywords to be written to the image +rather than translated), and "all" to display all the parameters +referenced by the package including the most obscure. For most uses +the "basic" set is all that is important and the other options are +included for completeness. +.le +.ih +DESCRIPTION +The purpose of this task is to provide an interface to simplify setting +up CCD instrument translation files and to verify the translations +for a set of images. Before this task was written users who needed to +set up translation files for new instruments and observatories had +to directly create the files with an editor. Many people encountered +difficulties and were prone to errors. Also there was no task that +directly verified the translations though \fBccdlist\fR provided some +clues. + +The \fBccdred\fR package was designed to make intelligent use of +information in image headers for determining things such as image +calibration or object type and exposure times. While the package may +be used without this capability it is much more convenient to be +able to use information from the image. The package was also intended +to be used with many different instruments, detectors, and observatories. +The key to providing image header access across different observatories +is the ability to translate the needs of the package to the appropriate +keywords in the image header. This is done through a file called +an "instrument translation file". For a complete description of +this file and other instrument setup features of the package see +\fBccdred.instruments\fR. + +The instrument translation file translates the parameter names used by +the \fBccdred\fR package into image specific parameters and also +supplies default values for parameters. The translation proceeds as +follows. When a package task needs a parameter for an image, for +example "imagetyp", it looks in the instrument translation file. If +the file is not found or none is specified then the image header +keyword that is requested is assumed to have the same name. If an +instrument translation file is defined then the requested parameter is +translated to an image header keyword, provided a translation entry is +given. If no translation is given the package name is used. For +example the package parameter "imagetyp" might be translated to +"data-typ" (the old NOAO CCD keyword). If the parameter is not found +then the default value specified in the translation file, if present, +is returned. + +For recording parameter information in the header, such +as processing flags, translation is also used. For example, if the +flag specifying that the image has been corrected by a flat field is to +be set then the package parameter name "flatcor" might be translated to +"ff-flag". If no translation is given then the new image header +parameter is entered as "flatcor". + +The CCD image type requires a second level of translation also defined +in the translation file. Once the image keyword which identifies the +type of CCD image, for example a flat field or object, is translated +to an imahe keyword the specific +string value must be translated to one of the CCD image types used +by the package. The translation works in the same way, the specific +string found is translated to the \fBccdred\fR type and returned to +the task. This translation is tricky in that the exact string +including all spaces and capitalizations must be correctly defined +in the translation file. The \fBccdinstrument\fR allows doing +this automatically thus minimizing typing errors. + +The basic display format of the task is a table of five columns +giving the parameter name used by the package, the image keyword +to which it is translated, the default value (if any), the value +the task will receive for the current image after translation, +and the actual keyword value in the image. A "?" is printed if +a value cannot be determined. The idea of the task is to make sure +that the value a \fBccdred\fR task sees is the correct one and if not +to modify the translation appropriately. In verify mode when the +\fBedit\fR parameter is not set the translation table is simply +printed for each input image. + +In edit mode the user interactively gives commands at the ccdinstrument +prompt to display or modify keywords. The modifications can then be +written to the instrument file or saved in a private copy. The +list of commands is shown below and may be printed using ? or help. + +.in 4 +.nf + CCDINSTRUMENT COMMANDS + +? Print command summary +help Print command summary +imheader Page image header +instrument Print current instrument translation file +next Next image +newimage Select a new image +quit Quit +read Read instrument translation file +show Show current translations +write Write instrument translation file + +translate Translate image string selected by the imagetyp + parameter to one of the CCDRED types given as an + argument or queried: + object, zero, dark, flat, comp, illum, fringe, other + +.fi +The following are CCDRED parameters which may be translated. You are +queried for the image keyword to use or it may be typed after the command. +An optional default value (returned if the image does not contain the +keyword) may be typed as the second argument of the command. +.nf + + BASIC PARAMETERS +imagetyp Image type parameter (see also translate) +subset Subset or filter parameter +exptime Exposure time +darktime Dark time (may be same as the exposure time) +.fi +.in -4 + +The commands may be followed by values such as file names for some of +the general commands or the keyword and default value for the parameters +to be translated. Note this is the only way to specify a default value. +If no arguments are given the user is prompted with the current value +which may then be changed. + +The set of parameters shown above are only those considered "basic". +In order to avoid confusion the task can limit the set of parameters +displayed. Without going into great detail, it is only the basic +parameters which are generally required to have valid translations to +allow the package to work well. However, for completeness, and if someone +wants to go wild with translations, further parameters may be displayed +and changed. The parameters displayed is controlled by the \fIparameters\fR +keyword. The additional parameters not shown above are: + +.in 4 +.nf + USEFUL DEFAULT GEOMETRY PARAMETERS +biassec Bias section (often has a default value) +trimsec Trim section (often has a default value) + + COMMON PROCESSING FLAGS +fixpix Bad pixel replacement flag +overscan Overscan correction flag +trim Trim flag +zerocor Zero level correction flag +darkcor Dark count correction flag +flatcor Flat field correction flag + + RARELY TRANSLATED PARAMETERS +ccdsec CCD section +datasec Data section +fixfile Bad pixel file + +fringcor Fringe correction flag +illumcor Ilumination correction flag +readcor One dimensional zero level read out correction +scancor Scan mode correction flag + +illumflt Ilumination flat image +mkfringe Fringe image +mkillum Iillumination image +skyflat Sky flat image + +ccdmean Mean value +fringscl Fringe scale factor +ncombine Number of images combined +date-obs Date of observations +dec Declination +ra Right Ascension +title Image title +.fi +.in -4 +.ih +EXAMPLES +1. To verify the translations for a set of images using the default +translation file: + +.nf + cl> setinst "" review- + cl> ccdinst dev$pix edit- + Image: dev$pix + Instrument file: + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + -------------------------------- + imagetyp imagetyp none ? + subset subset ? + exptime exptime ? ? + darktime darktime ? ? + + cl> setinst "" site=kpno dir=ccddb$ review- + cl> ccdinst dev$pix edit- + Image: dev$pix + + Instrument file: ccddb$kpno/camera.dat + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + -------------------------------- + imagetyp data-typ object OBJECT (0) + subset f1pos 2 2 + exptime otime 600 600 + darktime ttime 600 600 +.fi + +2. Set up an instrument translation file from scratch. + +.nf + ccdinst ech???.imh instr=myccd edit+ + Warning: OPEN: File does not exist (myccd) + Image: ech001.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp imagetyp none ? + subset subset ? + exptime exptime ? ? + darktime darktime ? ? + + ccdinstrument> imagetyp + Image keyword for image type (imagetyp): ccdtype + imagetyp ccdtype unknown BIAS + ccdinstrument> translate + CCDRED image type for 'BIAS' (unknown): zero + imagetyp ccdtype zero BIAS + ccdinstrument> subset + Image keyword for subset parameter (subset): filters + subset filters 1 1 0 + ccdinstrument> exptime integ + exptime integ 0. 0. + ccdinstrument> darktime integ + darktime integ 0. 0. + ccdinstrument> show + Image: ech001.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype zero BIAS + subset filters 1 1 0 + exptime integ 0. 0. + darktime integ 0. 0. + + ccdinstrument> next + Image: ech002.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype unknown PROJECTOR FLAT + subset filters 1 1 0 + exptime integ 20. 20. + darktime integ 20. 20. + + ccdinstrument> trans + CCDRED image type for 'PROJECTOR FLAT' (unknown): flat + imagetyp ccdtype flat PROJECTOR FLAT + ccdinstrument> next + Image: ech003.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype unknown COMPARISON + subset filters 1 1 0 + exptime integ 300 300 + darktime integ 300 300 + + ccdinstrument> translate comp + imagetyp ccdtype comp COMPARISON + ccdinstrument> next + Image: ech004.imh + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype unknown OBJECT + subset filters 1 1 0 + exptime integ 3600 3600 + darktime integ 3600 3600 + + ccdinstrument> translate object + imagetyp ccdtype object OBJECT + ccdinstrument> inst + imagetyp ccdtype + BIAS zero + subset filters + exptime integ + darktime integ + 'PROJECTOR FLAT' flat + COMPARISON comp + OBJECT object + + ccdinstrument> next + Update instrument file myccd (yes)? +.fi + +3. Set default geometry parameters. Note that to set a default the +arguments must be on the command line. + +.nf + cc> ccdinst ech001 instr=myccd param=common edit+ + Image: ech001 + Instrument file: myccd + Subset file: subsets + + CCDRED IMAGE DEFAULT CCDRED IMAGE + PARAM KEYWORD VALUE VALUE VALUE + ------------------------------------------------------ + imagetyp ccdtype zero BIAS + subset filters 1 1 0 + exptime integ 0. 0. + darktime integ 0. 0. + + biassec biassec ? ? + trimsec trimsec ? ? + + fixpix fixpix no ? + overscan overscan no ? + trim trim no ? + zerocor zerocor no ? + darkcor darkcor no ? + flatcor flatcor no ? + + ccdinstrument> biassec biassec [803:830,*] + biassec biassec [803:830,*] [803:830,*] ? + ccdinstrument> trimsec trimsec [2:798,2:798] + trimsec trimsec [2:798,2:798] [2:798,2:798] ? + ccdinstrument> instr + trimsec trimsec [2:798,2:798] + biassec biassec [803:830,*] + imagetyp ccdtype + BIAS zero + subset filters + exptime integ + darktime integ + 'PROJECTOR FLAT' flat + COMPARISON comp + OBJECT object + + ccdinstrument> q + Update instrument file myccd (yes)? +.fi +.ih +SEE ALSO +instruments, setinstrument +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdlist.hlp b/noao/imred/quadred/src/quad/doc/ccdlist.hlp new file mode 100644 index 00000000..9ce7dfdd --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdlist.hlp @@ -0,0 +1,133 @@ +.help ccdlist Jun87 noao.imred.ccdred +.ih +NAME +ccdlist -- List CCD processing information +.ih +USAGE +ccdlist images +.ih +PARAMETERS +.ls images +CCD images to be listed. A subset of the these may be selected using the +CCD image type parameter. +.le +.ls ccdtype = "" +CCD image type to be listed. If no type is specified then all the images +are listed. If an image type is specified then only images +of that type are listed. See \fBccdtypes\fR for a list of the package +image types. +.le +.ls names = no +List the image names only? Used with the CCD image type parameter to make +a list of the images of the specified type. +.le +.ls long = no +Long format listing? The images are listed in a long format containing some +image parameters and the processing history. +.le +.ls ccdproc (pset) +CCD processing parameter set. +.le +.ih +DESCRIPTION +Information from the specified input images is listed on the standard +output. A specific CCD image type may be selected from the input +images by the parameter \fIccdtype\fR. There are three list formats; +the default one line per image format, an image name only format, and a +multi-line long format. The default one line format consists of the +image name, image size, image pixel type, CCD image type, subset ID (if +defined), processing flags, and title. This format contains the same +information as that produced by \fBimheader\fR as well as CCD specific +information. The processing flags identifying the processing operations +performed on the image are given by the following single letter codes. + +.nf + B - Bad pixel replacement + O - Overscan bias subtraction + T - Trimming + Z - Zero level subtraction + D - Dark count subtraction + F - Flat field calibration + I - Iillumination correction + Q - Fringe correction +.fi + +The long format has the same first line as the default format plus additional +instrument information such as the exposure time and the full processing +history. In addition to listing the completed processing, the operations +not yet done (as specified by the \fBccdproc\fR parameters) are also +listed. + +The image name only format is intended to be used to generate lists of +images of the same CCD image type. These lists may be used as "@" file +lists in IRAF tasks. +.ih +EXAMPLES +1. To list the default format for all images: + +.nf + cl> ccdlist *.imh + ccd001.imh[544,512][short][unknown][V]:FOCUS L98-193 + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s + ccd045.imh[544,512][short][flat][V]:dflat 6v+blue 5s + ccd066.imh[544,512][short][flat][B]:dflat 6v+blue 5s + ccd103.imh[544,512][short][flat][R]:dflat 6v+blue 5s + ccd104.imh[544,512][short][zero][]:bias + ccd105.imh[544,512][short][dark][]:dark 3600s +.fi + +These images have not been processed. + +2. To restrict the listing to just the object images: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s +.fi + +3. The long list for image "ccd007" is obtained by: + +.nf + cl> ccdlist ccd007 l+ + ccd007[544,512][short][object][V]:N2968 R 600s + exptime = 200. darktime = 200. + [TO BE DONE] Overscan strip is [520:540,*] + [TO BE DONE] Trim image section is [3:510,3:510] + [TO BE DONE] Flat field correction +.fi + +4. After processing the images have the short listing: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[508,508][real][object][V][OTF]:N2968 V 600s + ccd015.imh[508,508][real][object][B][OTF]:N3098 B 500s + ccd024.imh[544,512][short][object][R][OTF]:N4036 R 600s +.fi + +The processing indicated is overscan subtraction, trimming, and flat fielding. + +5. The long listing for "ccd007" after processing is: + +.nf + cl> ccdlist ccd007 l+ + ccd007[508,508][real][object][V][OTF]:N2968 R 600s + exptime = 200. darktime = 200. + Jun 2 18:18 Overscan section is [520:540,*] with mean=481.8784 + Jun 2 18:18 Trim data section is [3:510,3:510] + Jun 2 18:19 Flat field image is FlatV.imh with scale=138.2713 +.fi + +6. To make a list file containing all the flat field images: + + cl> ccdlist *.imh ccdtype=flat name+ > flats + +This file can be used as an @ file for processing. +.ih +SEE ALSO +ccdtypes ccdgroups +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdproc.hlp b/noao/imred/quadred/src/quad/doc/ccdproc.hlp new file mode 100644 index 00000000..4be65f73 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdproc.hlp @@ -0,0 +1,720 @@ +.help ccdproc Oct90 noao.imred.ccdred +.ih +NAME +ccdproc -- Process CCD images +.ih +USAGE +ccdproc images +.ih +PARAMETERS +.ls images +List of input CCD images to process. The list may include processed +images and calibration images. +.le +.ls ccdtype = "" +CCD image type to select from the input image list. If no type is given +then all input images will be selected. The recognized types are described +in \fBccdtypes\fR. +.le +.ls max_cache = 0 +Maximum image caching memory (in Mbytes). If there is sufficient memory +the calibration images, such as zero level, dark count, and flat fields, +will be cached in memory when processing many input images. This +reduces the disk I/O and makes the task run a little faster. If the +value is zero image caching is not used. +.le +.ls noproc = no +List processing steps only? +.le + +.ce +PROCESSING SWITCHES +.ls fixpix = yes +Fix bad CCD lines and columns by linear interpolation from neighboring +lines and columns? If yes then a bad pixel file must be specified. +.le +.ls overscan = yes +Apply overscan or prescan bias correction? If yes then the overscan +image section and the readout axis must be specified. +.le +.ls trim = yes +Trim the image of the overscan region and bad edge lines and columns? +If yes then the data section must be specified. +.le +.ls zerocor = yes +Apply zero level correction? If yes a zero level image must be specified. +.le +.ls darkcor = yes +Apply dark count correction? If yes a dark count image must be specified. +.le +.ls flatcor = yes +Apply flat field correction? If yes flat field images must be specified. +.le +.ls illumcor = no +Apply iillumination correction? If yes iillumination images must be specified. +.le +.ls fringecor = no +Apply fringe correction? If yes fringe images must be specified. +.le +.ls readcor = no +Convert zero level images to readout correction images? If yes then +zero level images are averaged across the readout axis to form one +dimensional zero level readout correction images. +.le +.ls scancor = no +Convert flat field images to scan mode flat field images? If yes then the +form of scan mode correction is specified by the parameter \fIscantype\fR. +.le + +.ce +PROCESSING PARAMETERS +.ls readaxis = "line" +Read out axis specified as "line" or "column". +.le +.ls fixfile +File describing the bad lines and columns. If "image" is specified then +the file is specified in the image header or instrument translation file. +.le +.ls biassec +Overscan bias strip image section. If "image" is specified then the overscan +bias section is specified in the image header or instrument translation file. +.le +.ls trimsec +image section for trimming. If "image" is specified then the trim +image section is specified in the image header or instrument translation file. +.le +.ls zero = "" +Zero level calibration image. The zero level image may be one or two +dimensional. The CCD image type and subset are not checked for these +images and they take precedence over any zero level calibration images +given in the input list. +.le +.ls dark = "" +Dark count calibration image. The CCD image type and subset are not checked +for these images and they take precedence over any dark count calibration +images given in the input list. +.le +.ls flat = "" +Flat field calibration images. The flat field images may be one or +two dimensional. The CCD image type is not checked for these +images and they take precedence over any flat field calibration images given +in the input list. The flat field image with the same subset as the +input image being processed is selected. +.le +.ls illum = "" +Iillumination correction images. The CCD image type is not checked for these +images and they take precedence over any iillumination correction images given +in the input list. The iillumination image with the same subset as the +input image being processed is selected. +.le +.ls fringe = "" +Fringe correction images. The CCD image type is not checked for these +images and they take precedence over any fringe correction images given +in the input list. The fringe image with the same subset as the +input image being processed is selected. +.le +.ls minreplace = 1. +When processing flat fields, pixel values below this value (after +all other processing such as overscan, zero, and dark corrections) are +replaced by this value. This allows flat fields processed by \fBccdproc\fR +to be certain to avoid divide by zero problems when applied to object +images. +.le +.ls scantype = "shortscan" +Type of scan format used in creating the CCD images. The modes are: +.ls "shortscan" +The CCD is scanned over a number of lines and then read out as a regular +two dimensional image. In this mode unscanned flat fields are numerically +scanned to form scanned flat fields comparable to the observations. If +the flat field calibration images are taken in scanned mode then +\fIscancor\fR should be no and the processing performed in the same manner +as in unscanned mode. +.le +.ls "longscan" +In this mode the CCD is clocked and read out continuously to form a long +strip. Flat fields are averaged across the readout axis to +form a one dimensional flat field readout correction image. This assumes +that all recorded image lines are clocked over the entire active area of the +CCD. +.le +.le +.ls nscan +Number of scan readout lines used in short scan mode. This parameter is used +when the scan type is "shortscan". +.le + + +.ce +OVERSCAN FITTING PARAMETERS +.ls interactive = no +Fit the overscan vector interactively? If yes the overscan vector is fit +interactively using the \fBicfit\fR package. If no then the fitting parameters +given below are used. +.le +.ls function = "legendre" +Overscan fitting function. The function types are "legendre" polynomial, +"chebyshev" polynomial, "spline1" linear spline, and "spline3" cubic +spline. +.le +.ls order = 1 +Number of polynomial terms or spline pieces in the overscan fit. +.le +.ls sample = "*" +Sample points to use in the overscan fit. The string "*" specified all +points otherwise an \fBicfit\fR range string is used. +.le +.ls naverage = 1 +Number of points to average or median to form fitting points. Positive +numbers specify averages and negative numbers specify medians. +.le +.ls niterate = 1 +Number of rejection iterations to remove deviant points from the overscan fit. +If 0 then no points are rejected. +.le +.ls low_reject = 3., high_reject = 3. +Low and high sigma rejection factors for rejecting deviant points from the +overscan fit. +.le +.ls grow = 0. +One dimensional growing radius for rejection of neighbors to deviant points. +.le +.ih +DESCRIPTION +\fBCcdproc\fR processes CCD images to correct and calibrate for +detector defects, readout bias, zero level bias, dark counts, +response, iillumination, and fringing. It also trims unwanted +lines and columns and changes the pixel datatype. It is efficient +and easy to use; all one has to do is set the parameters and then +begin processing the images. The task takes care of most of the +record keeping and automatically does the prerequisite processing +of calibration images. Beneath this simplicity there is much that +is going on. In this section a simple description of the usage is +given. The following sections present more detailed discussions +on the different operations performed and the order and logic +of the processing steps. For a user's guide to the \fBccdred\fR +package see \fBguide\fR. Much of the ease of use derives from using +information in the image header. If this information is missing +see section 13. + +One begins by setting the task parameters. There are many parameters +but they may be easily reviewed and modified using the task \fBeparam\fR. +The input CCD images to be processed are given as an image list. +Previously processed images are ignored and calibration images are +recognized, provided the CCD image types are in the image header (see +\fBinstruments\fR and \fBccdtypes\fR). Therefore it is permissible to +use simple image templates such as "*.imh". The \fIccdtype\fR parameter +may be used to select only certain types of CCD images to process +(see \fBccdtypes\fR). + +The processing operations are selected by boolean (yes/no) parameters. +Because calibration images are recognized and processed appropriately, +the processing operations for object images should be set. +Any combination of operations may be specified and the operations are +performed simultaneously. While it is possible to do operations in +separate steps this is much less efficient. Two of the operation +parameters apply only to zero level and flat field images. These +are used for certain types of CCDs and modes of operation. + +The processing steps selected have related parameters which must be +set. These are things like image sections defining the overscan and +trim regions and calibration images. There are a number of parameters +used for fitting the overscan or prescan bias section. These are +parameters used by the standard IRAF curve fitting package \fBicfit\fR. +The parameters are described in more detail in the following sections. + +In addition to the task parameters there are package parameters +which affect \fBccdproc\fR. These include the instrument and subset +files, the text and plot log files, the output pixel datatype, +the amount of memory available for calibration image caching, +the verbose parameter for logging to the terminal, and the backup +prefix. These are described in \fBccdred\fR. + +Calibration images are specified by task parameters and/or in the +input image list. If more than one calibration image is specified +then the first one encountered is used and a warning is issued for the +extra images. Calibration images specified by +task parameters take precedence over calibration images in the input list. +These images also need not have a CCD image type parameter since the task +parameter identifies the type of calibration image. This method is +best if there is only one calibration image for all images +to be processed. This is almost always true for zero level and dark +count images. If no calibration image is specified by task parameter +then calibration images in the input image list are identified and +used. This requires that the images have CCD image types recognized +by the package. This method is useful if one may simply say "*.imh" +as the image list to process all images or if the images are broken +up into groups, in "@" files for example, each with their own calibration +frames. + +When an input image is processed the task first determines the processing +parameters and calibration images. If a requested operation has been +done it is skipped and if all requested operations have been completed then +no processing takes place. When it determines that a calibration image +is required it checks for the image from the task parameter and then +for a calibration image of the proper type in the input list. + +Having +selected a calibration image it checks if it has been processed by +looking for the image header flag CCDPROC. If it is not present then +the calibration image is processed. When any image has been processed +the CCDPROC flag is added. For images processed directly by \fBccdproc\fR +the individual processing flags are checked even if the CCDPROC flag is +present. However, the automatic processing of the calibration images is +only done if the CCDPROC flag is absent! This is to make the task more +efficient by not having to check every flag for every calibration image +for every input image. Thus, if additional processing +steps are added after images have been partially reduced then input images +will be processed for the new steps but calibration images will not be +processed automatically. + +After the calibration images have been identified, and processed if +necessary, the images may be cached in memory. This is done when there +are more than two input images (it is actually less efficient to +cache the calibration images for one or two input images) and the parameter +\fImax_cache\fR is greater than zero. When caching, as many calibration +images as allowed by the specified memory are read into memory and +kept there for all the input images. Cached images are, therefore, +only read once from disk which reduces the amount of disk I/O. This +makes a modest decrease in the execution time. It is not dramatic +because the actual processing is fairly CPU intensive. + +Once the processing parameters and calibration images have been determined +the input image is processed for all the desired operations in one +step; i.e. there are no intermediate results or images. This makes +the task efficient. The corrected image is output as a temporary image +until the entire image has been processed. When the image has been +completely processed then the original image is deleted (or renamed +using the specified backup prefix) and the corrected image replaces +the original image. Using a temporary image protects the data in the +event of an abort or computer failure. Keeping the original image name +eliminates much of the record keeping and the need to generate new +image names. +.sh +1. Fixpix +Regions of bad lines and columns may be replaced by linear +interpolation from neighboring lines and columns when the parameter +\fIfixpix\fR is set. The bad regions are specified in a bad pixel +file. The file consists of lines with four fields, the starting and +ending columns and the starting and ending lines. Any number of +regions may be specified. Comment lines beginning with the character +'#' may be included. If a comment line preceding the bad regions +contains the word "untrimmed" then the coordinate system refers to the +original format of the images; i.e. before trimming. If an image has +been trimmed previously then the trim region specified in the image +header is used to convert the coordinates in the bad pixel file to +those of the trimmed image. If the file does not contain the word +"untrimmed" then the coordinate system must match that of the image +being corrected; i.e. untrimmed coordinates if the image has not been +trimmed and trimmed coordinates if the image has been trimmed. +Standard bad pixel files should always be specified in terms of the +original format. + +The bad pixel file may be specified explicitly with the parameter \fIfixfile\fR +or indirectly if the parameter has the value "image". In the latter case +the instrument file must contain the name of the file. +.sh +2. Overscan +If an overscan or prescan correction is specified (\fIoverscan\fR +parameter) then the image section (\fIbiassec\fR parameter) is averaged +along the readout axis (\fIreadaxis\fR parameter) to form a +correction vector. A function is fit to this vector and for each readout +line (image line or column) the function value for that line is +subtracted from the image line. The fitting function is generally +either a constant (polynomial of 1 term) or a high order function +which fits the large scale shape of the overscan vector. Bad pixel +rejection is also used to eliminate cosmic ray events. The function +fitting may be done interactively using the standard \fBicfit\fR +iteractive graphical curve fitting tool. Regardless of whether the fit +is done interactively, the overscan vector and the fit may be recorded +for later review in a metacode plot file named by the parameter +\fIccdred.plotfile\fR. The mean value of the bias function is also recorded in +the image header and log file. +.sh +3. Trim +When the parameter \fItrim\fR is set the input image will be trimmed to +the image section given by the parameter \fItrimsec\fR. This trim +should, of course, be the same as that used for the calibration images. +.sh +4. Zerocor +After the readout bias is subtracted, as defined by the overscan or prescan +region, there may still be a zero level bias. This level may be two +dimensional or one dimensional (the same for every readout line). A +zero level calibration is obtained by taking zero length exposures; +generally many are taken and combined. To apply this zero +level calibration the parameter \fIzerocor\fR is set. In addition if +the zero level bias is only readout dependent then the parameter \fIreadcor\fR +is set to reduce two dimensional zero level images to one dimensional +images. The zero level images may be specified by the parameter \fIzero\fR +or given in the input image list (provided the CCD image type is defined). + +When the zero level image is needed to correct an input image it is checked +to see if it has been processed and, if not, it is processed automatically. +Processing of zero level images consists of bad pixel replacement, +overscan correction, trimming, and averaging to one dimension if the +readout correction is specified. +.sh +5. Darkcor +Dark counts are subtracted by scaling a dark count calibration image to +the same exposure time as the input image and subtracting. The +exposure time used is the dark time which may be different than the +actual integration or exposure time. A dark count calibration image is +obtained by taking a very long exposure with the shutter closed; i.e. +an exposure with no light reaching the detector. The dark count +correction is selected with the parameter \fIdarkcor\fR and the dark +count calibration image is specified either with the parameter +\fIdark\fR or as one of the input images. The dark count image is +automatically processed as needed. Processing of dark count images +consists of bad pixel replacement, overscan and zero level correction, +and trimming. +.sh +6. Flatcor +The relative detector pixel response is calibrated by dividing by a +scaled flat field calibration image. A flat field image is obtained by +exposure to a spatially uniform source of light such as an lamp or +twilight sky. Flat field images may be corrected for the spectral +signature in spectroscopic images (see \fBresponse\fR and +\fBapnormalize\fR), or for iillumination effects (see \fBmkillumflat\fR +or \fBmkskyflat\fR). For more on flat fields and iillumination corrections +see \fBflatfields\fR. The flat field response is dependent on the +wavelength of light so if different filters or spectroscopic wavelength +coverage are used a flat field calibration for each one is required. +The different flat fields are automatically selected by a subset +parameter (see \fBsubsets\fR). + +Flat field calibration is selected with the parameter \fBflatcor\fR +and the flat field images are specified with the parameter \fBflat\fR +or as part of the input image list. The appropriate subset is automatically +selected for each input image processed. The flat field image is +automatically processed as needed. Processing consists of bad pixel +replacement, overscan subtraction, zero level subtraction, dark count +subtraction, and trimming. Also if a scan mode is used and the +parameter \fIscancor\fR is specified then a scan mode correction is +applied (see below). The processing also computes the mean of the +flat field image which is used later to scale the flat field before +division into the input image. For scan mode flat fields the ramp +part is included in computing the mean which will affect the level +of images processed with this flat field. Note that there is no check for +division by zero in the interest of efficiency. If division by zero +does occur a fatal error will occur. The flat field can be fixed by +replacing small values using a task such as \fBimreplace\fR or +during processing using the \fIminreplace\fR parameter. Note that the +\fIminreplace\fR parameter only applies to flat fields processed by +\fBccdproc\fR. +.sh +7. Illumcor +CCD images processed through the flat field calibration may not be +completely flat (in the absence of objects). In particular, a blank +sky image may still show gradients. This residual nonflatness is called +the iillumination pattern. It may be introduced even if the detector is +uniformly illuminated by the sky because the flat field lamp +iillumination may be nonuniform. The iillumination pattern is found from a +blank sky, or even object image, by heavily smoothing and rejecting +objects using sigma clipping. The iillumination calibration image is +divided into the data being processed to remove the iillumination +pattern. The iillumination pattern is a function of the subset so there +must be an iillumination correction image for each subset to be +processed. The tasks \fBmkillumcor\fR and \fBmkskycor\fR are used to +create the iillumination correction images. For more on iillumination +corrections see \fBflatfields\fR. + +An alternative to treating the iillumination correction as a separate +operation is to combine the flat field and iillumination correction +into a corrected flat field image before processing the object +images. This will save some processing time but does require creating +the flat field first rather than correcting the images at the same +time or later. There are two methods, removing the large scale +shape of the flat field and combining a blank sky image iillumination +with the flat field. These methods are discussed further in the +tasks which create them; \fBmkillumcor\fR and \fBmkskycor\fR. +.sh +8. Fringecor +There may be a fringe pattern in the images due to the night sky lines. +To remove this fringe pattern a blank sky image is heavily smoothed +to produce an iillumination image which is then subtracted from the +original sky image. The residual fringe pattern is scaled to the +exposure time of the image to be fringe corrected and then subtracted. +Because the intensity of the night sky lines varies with time an +additional scaling factor may be given in the image header. +The fringe pattern is a function of the subset so there must be +a fringe correction image for each subset to be processed. +The task \fBmkfringecor\fR is used to create the fringe correction images. +.sh +9. Readcor +If a zero level correction is desired (\fIzerocor\fR parameter) +and the parameter \fIreadcor\fR is yes then a single zero level +correction vector is applied to each readout line or column. Use of a +readout correction rather than a two dimensional zero level image +depends on the nature of the detector or if the CCD is operated in +longscan mode (see below). The readout correction is specified by a +one dimensional image (\fIzero\fR parameter) and the readout axis +(\fIreadaxis\fR parameter). If the zero level image is two dimensional +then it is automatically processed to a one dimensional image by +averaging across the readout axis. Note that this modifies the zero +level calibration image. +.sh +10. Scancor +CCD detectors may be operated in several modes in astronomical +applications. The most common is as a direct imager where each pixel +integrates one point in the sky or spectrum. However, the design of most CCD's +allows the sky to be scanned across the CCD while shifting the +accumulating signal at the same rate. \fBCcdproc\fR provides for two +scanning modes called "shortscan" and "longscan". The type of scan +mode is set with the parameter \fIscanmode\fR. + +In "shortscan" mode the detector is scanned over a specified number of +lines (not necessarily at sideral rates). The lines that scroll off +the detector during the integration are thrown away. At the end of the +integration the detector is read out in the same way as an unscanned +observation. The advantage of this mode is that the small scale flat +field response is averaged in one dimension over the number of lines +scanned. A flat field may be observed in the same way in which case +there is no difference in the processing from unscanned imaging and the +parameter \fIscancor\fR should be no. However, one obtains an increase +in the statistical accuracy of the flat fields if they are not scanned +during the observation but digitally scanned during the processing. In +shortscan mode with \fIscancor\fR set to yes, flat field images are +digitally scanned, if needed, by the specified number of scan lines +(\fInscan\fR parameter). + +In "longscan" mode the detector is continuously read out to produce +an arbitrarily long strip. Provided data which has not passed over +the entire detector is thrown away, the flat field corrections will +be one dimensional. If \fIscancor\fR is specified and the +scan mode is "longscan" then a one dimensional flat field correction +will be applied. If the specified flat field (\fIflat\fR parameter) +is a two dimensional image then when the flat field image is processed +it will be averaged across the readout axis to form a one dimensional +correction image. +.sh +11. Processing Steps +The following describes the steps taken by the task. This detailed +outline provides the most detailed specification of the task. + +.ls 5 (1) +An image to be processed is first checked that it is of the specified +CCD image type. If it is not the desired type then go on to the next image. +.le +.ls (2) +A temporary output image is created of the specified pixel data type +(\fBccdred.pixeltype\fR). The header parameters are copied from the +input image. +.le +.ls (3) +If trimming is specified and the image has not been trimmed previously, +the trim section is determined. +.le +.ls (4) +If bad pixel replacement is specified and this has not been done +previously, the bad pixel file is determined either from the task +parameter or the instrument translation file. The bad pixel regions +are read. If the image has been trimmed previously and the bad pixel +file contains the word "untrimmed" then the bad pixel coordinates are +translated to those of the trimmed image. +.le +.ls (5) +If an overscan correction is specified and this correction has not been +applied, the overscan section is averaged along the readout axis. If +trimming is to be done the overscan section is trimmed to the same +limits. A function is fit either interactively or noninteractively to +the overscan vector. The function is used to produce the overscan +vector to be subtracted from the image. This is done in real +arithmetic. +.le +.ls (6) +If the image is a zero level image go to processing step 12. +If a zero level correction is desired and this correction has not been +performed, find the zero level calibration image. If the zero level +calibration image has not been processed it is processed at this point. +This is done by going to processing step 1 for this image. After the +calibration image has been processed, processing of the input image +continues from this point. +The processed calibration image may be +cached in memory if it has not been previously and if there is enough memory. +.le +.ls (7) +If the image is a dark count image go to processing step 12. +If a dark count correction is desired and this correction has not been +performed, find the dark count calibration image. If the dark count +calibration image has not been processed it is processed at this point. +This is done by going to processing step 1 for this image. After the +calibration image has been processed, processing of the input image +continues from this point. The ratio of the input image dark time +to the dark count image dark time is determined to be multiplied with +each pixel of the dark count image before subtracting from the input +image. +The processed calibration image may be +cached in memory if it has not been previously and if there is enough memory. +.le +.ls (8) +If the image is a flat field image go to processing step 12. If a flat +field correction is desired and this correction has not been performed, +find the flat field calibration image of the appropriate subset. If +the flat field calibration image has not been processed it is processed +at this point. This is done by going to processing step 1 for this +image. After the calibration image has been processed, processing of +the input image continues from this point. The mean of the image +is determined from the image header to be used for scaling. If no +mean is found then a unit scaling is used. +The processed calibration image may be +cached in memory if it has not been previously and if there is enough memory. +.le +.ls (9) +If the image is an iillumination image go to processing step 12. If an +iillumination correction is desired and this correction has not been performed, +find the iillumination calibration image of the appropriate subset. +The iillumination image must have the "mkillum" processing flag or the +\fBccdproc\fR will abort with an error. The mean of the image +is determined from the image header to be used for scaling. If no +mean is found then a unit scaling is used. The processed calibration +image may be +cached in memory if it has not been previously and there is enough memory. +.le +.ls (10) +If the image is a fringe image go to processing step 12. If a fringe +correction is desired and this correction has not been performed, +find the fringe calibration image of the appropriate subset. +The iillumination image must have the "mkfringe" processing flag or the +\fBccdproc\fR will abort with an error. The ratio of the input +image exposure time to the fringe image exposure time is determined. +If there is a fringe scaling in the image header then this factor +is multiplied by the exposure time ratio. This factor is used +for scaling. The processed calibration image may be +cached in memory if it has not been previously and there is enough memory. +.le +.ls (11) +If there are no processing operations flagged, delete the temporary output +image, which has been opened but not used, and go to 14. +.le +.ls (12) +The input image is processed line by line with trimmed lines ignored. +A line of the input image is read. Bad pixel replacement and trimming +is applied to the image. Image lines from the calibration images +are read from disk or the image cache. If the calibration is one +dimensional (such as a readout zero +level correction or a longscan flat field correction) then the image +vector is read only once. Note that IRAF image I/O is buffered for +efficiency and accessing a line at a time does not mean that image +lines are read from disk a line at a time. Given the input line, the +calibration images, the overscan vector, and the various scale factors +a special data path for each combination of corrections is used to +perform all the processing in the most efficient manner. If the +image is a flat field any pixels less than the \fIminreplace\fR +parameter are replaced by that minimum value. Also a mean is +computed for the flat field and stored as the CCDMEAN keyword. +.le +.ls (13) +The input image is deleted or renamed to a backup image. The temporary +output image is renamed to the input image name. +.le +.ls (14) +If the image is a zero level image and the readout correction is specified +then it is averaged to a one dimensional readout correction. +.le +.ls (15) +If the image is a flat field image and the scan mode correction is specified +then the correction is applied. For shortscan mode a +modified two dimensional image is produced while for longscan mode a +one dimensional average image is produced. +.le +.ls (16) +The processing is completed and either the next input image is processed +beginning at step 1 or, if it is a calibration image which is being +processed for an input image, control returns to the step which initiated +the calibration image processing. +.le +.sh +12. Processing Arithmetic +The \fBccdproc\fR task has two data paths, one for real image pixel datatypes +and one for short integer pixel datatype. In addition internal arithmetic +is based on the rules of FORTRAN. For efficiency there is +no checking for division by zero in the flat field calibration. +The following rules describe the processing arithmetic and data paths. + +.ls (1) +If the input, output, or any calibration image is of type real the +real data path is used. This means all image data is converted to +real on input. If all the images are of type short all input data +is kept as short integers. Thus, if all the images are of the same type +there is no datatype conversion on input resulting in greater +image I/O efficiency. +.le +.ls (2) +In the real data path the processing arithmetic is always real and, +if the output image is of short pixel datatype, the result +is truncated. +.le +.ls (3) +The overscan vector and the scale factors for dark count, flat field, +iillumination, and fringe calibrations are always of type real. Therefore, +in the short data path any processing which includes these operations +will be coerced to real arithmetic and the result truncated at the end +of the computation. +.le +.sh +13. In the Absence of Image Header Information +The tasks in the \fBccdred\fR package are most convenient to use when +the CCD image type, subset, and exposure time are contained in the +image header. The ability to redefine which header parameters contain +this information makes it possible to use the package at many different +observatories (see \fBinstruments\fR). However, in the absence of any +image header information the tasks may still be used effectively. +There are two ways to proceed. One way is to use \fBccdhedit\fR +to place the information in the image header. + +The second way is to specify the processing operations more explicitly +than is needed when the header information is present. The parameter +\fIccdtype\fR is set to "" or to "none". The calibration images are +specified explicitly by task parameter since they cannot be recognized +in the input list. Only one subset at a time may be processed. + +If dark count and fringe corrections are to be applied the exposure +times must be added to all the images. Alternatively, the dark count +and fringe images may be scaled explicitly for each input image. This +works because the exposure times default to 1 if they are not given in +the image header. +.ih +EXAMPLES +The user's \fBguide\fR presents a tutorial in the use of this task. + +1. In general all that needs to be done is to set the task parameters +and enter + + cl> ccdproc *.imh & + +This will run in the background and process all images which have not +been processed previously. +.ih +TIME REQUIREMENTS +.nf +o SUN-3, 15 MHz 68020 with 68881 floating point hardware (no FPA) +o 8 Mb RAM, 2 Fuji Eagle disks. +o Input images = 544 x 512 short +o Output image = 500 x 500 real +o Operations are overscan subtraction (O), trimming to 500x500 (T), + zero level subtraction (Z), dark count scaling and subtraction (D), + and flat field scaling and subtraction (F). +o UNIX statistics + (user, system, and clock time, and misc. memory and i/o statistics): + +[OTF] One calibration image and 9 object images: +No caching: 110.6u 25.5s 3:18 68% 28+ 40K 3093+1645io 9pf+0w +Caching: 111.2u 23.0s 2:59 74% 28+105K 2043+1618io 9pf+0w + +[OTZF] Two calibration images and 9 object images: +No caching: 119.2u 29.0s 3:45 65% 28+ 50K 4310+1660io 9pf+0w +Caching: 119.3u 23.0s 3:07 75% 28+124K 2179+1601io 9pf+0w + +[OTZDF] Three calibration images and 9 object images: +No caching: 149.4u 31.6s 4:41 64% 28+ 59K 5501+1680io 19pf+0w +Caching: 151.5u 29.0s 4:14 70% 27+227K 2346+1637io 148pf+0w + +[OTZF] 2 calibration images and 20 images processed: +No caching: 272.7u 63.8u 8:47 63% 28+ 50K 9598+3713io 12pf+0w +Caching: 271.2u 50.9s 7:00 76% 28+173K 4487+3613io 51pf+0w +.fi +.ih +SEE ALSO +.nf +instruments, ccdtypes, flatfields, icfit, ccdred, guide, mkillumcor, +mkskycor, mkfringecor +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdred.hlp b/noao/imred/quadred/src/quad/doc/ccdred.hlp new file mode 100644 index 00000000..0300bd38 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdred.hlp @@ -0,0 +1,98 @@ +.help package Jun87 noao.imred +.ih +NAME +ccdred -- CCD image reduction package +.ih +USAGE +ccdred +.ih +PARAMETERS +.ls pixeltype = "real real" +Output pixel datatype and calculation datatype. When images are processed +or created the output pixel datatype is determined by this parameter. +The allowed types are "short" for short integer, and "real" for real +floating point. Note that if short input images are processed into +real images the disk space required will generally increase. +The calculation datatypes are also short and real with a default of +real if none is specified. +.le +.ls verbose = no +Print log information to the standard output? +.le +.ls logfile = "logfile" +Text log file. If no filename is specified then no log file is kept. +.le +.ls plotfile = "" +Log metacode plot file for the overscan bias vector fits. If +no filename is specified then no metacode plot file is kept. +.le +.ls backup = "" +Backup prefix for backup images. If no prefix is specified then no backup +images are kept when processing. If specified then the backup image +has the specified prefix. +.le +.ls instrument = "" +CCD instrument translation file. This is usually set with \fBsetinstrument\fR. +.le +.ls ssfile = "subsets" +Subset translation file used to define the subset identifier. See +\fBsubsets\fR for more. +.le +.ls graphics = "stdgraph" +Interactive graphics output device when fitting the overscan bias vector. +.le +.ls cursor = "" +Graphics cursor input. The default is the standard graphics cursor. +.le +.ls version = "June 1987" +Package version. +.le +.ih +DESCRIPTION +The CCD reduction package is loaded when this command is entered. The +package contains parameters which affect the operation of the tasks +it defines. When images are processed or new image are created the +output pixel datatype is that specified by the parameter \fBpixeltype\fR. +Note that CCD processing replaces the original image by the processed +image so the pixel type of the CCD images may change during processing. +It is unlikely that real images will be processed to short images but +the reverse is quite likely. Processing images from short to real +pixel datatypes will generally increase the amount of disk space +required (a factor of 2 on most computers). + +The tasks produce log output which may be printed on the standard +output (the terminal unless redirected) and appended to a file. The +parameter \fIverbose\fR determines whether processing information +is printed. This may be desirable initially, but when using background +jobs the verbose output should be turned off. The user may look at +the end of the log file (for example with \fBtail\fR) to determine +the status of the processing. + +The package was designed to work with data from many different observatories +and instruments. In order to accomplish this an instrument translation +file is used to define a mapping between the package parameters and +the particular image header format. The instrument translation file +is specified to the package by the parameter \fIinstrument\fR. This +parameter is generally set by the task \fBsetinstrument\fR. The other +file used is a subset file. This is generally created and maintained +by the package and the user need not do anything. For more sophisticated +users see \fBinstruments\fR and \fBsubsets\fR. + +The package has very little graphics +output. The exception is the overscan bias subtraction. The bias +vector is logged in the metacode plot file if given. The plot file +may be examined with the tasks in the \fBplot\fR package such as +\fBgkimosaic\fR. When interactively fitting the overscan vector +the graphics input and output devices must be specified. The defaults +should apply in most cases. + +Because processing replaces the input image by the processed image it +may be desired to save the original image. This may be done by +specifying a backup prefix with the parameter \fIbackup\fR. For +example, if the prefix is "orig" and the image is "ccd001", the backup +image will be "origccd001". The prefix may be a directory but it must +end with '/' or '$' (for logical directories). +.ih +SEE ALSO +instruments, setinstrument, subsets +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/ccdred.ms b/noao/imred/quadred/src/quad/doc/ccdred.ms new file mode 100644 index 00000000..645514ec --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdred.ms @@ -0,0 +1,787 @@ +.RP +.TL +The IRAF CCD Reduction Package -- CCDRED +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +September 1987 +.AB +The IRAF\(dg CCD reduction package, \fBccdred\fR, provides tools +for the easy and efficient reduction of CCD images. The standard +reduction operations are replacement of bad pixels, subtraction of an +overscan or prescan bias, subtraction of a zero level image, +subtraction of a dark count image, division by a flat field calibration +image, division by an illumination correction, subtraction of a fringe +image, and trimming unwanted lines or columns. Another common +operation provided by the package is scaling and combining images with +a number of algorithms for rejecting cosmic rays. Data in the image +header is used to make the reductions largely automated and +self-documenting though the package may still be used in the absence of +this data. Also a translation mechanism is used to relate image header +parameters to those used by the package to allow data from a variety of +observatories and instruments to be processed. This paper describes +the design goals for the package and the main tasks and algorithms +which satisfy these goals. +.PP +This paper is to be published as part of the proceedings of the +Santa Cruz Summer Workshop in Astronomy and Astrophysics, +\fIInstrumentation for Ground-Based Optical Astronomy: Present and +Future\fR, edited by Lloyd B. Robinson and published by +Springer-Verlag. +.LP +\(dgImage Reduction and Analysis Facility (IRAF), a software system +distributed by the National Optical Astronomy Observatories (NOAO). +.AE +.NH +Introduction +.PP +The IRAF CCD reduction package, \fBccdred\fR, provides tools +for performing the standard instrumental corrections and calibrations +to CCD images. The major design goals were: +.IP +.nf +\(bu To be easy to use +\(bu To be largely automated +\(bu To be image header driven if the data allows +\(bu To be usable for a variety of instruments and observatories +\(bu To be efficient and capable of processing large volumes of data +.fi +.LP +This paper describes the important tasks and algorithms and shows how +these design goals were met. It is not intended to describe every +task, parameter, and usage in detail; the package has full +documentation on each task plus a user's guide. +.PP +The standard CCD correction and calibration operations performed are +replacement of bad columns and lines by interpolation from neighboring +columns and lines, subtraction of a bias level determined from overscan +or prescan columns or lines, subtraction of a zero level using a zero +length exposure calibration image, subtraction of a dark count +calibration image appropriately scaled to the dark time exposure of the +image, division by a scaled flat field calibration image, division by +an illumination image (derived from a blank sky image), subtraction of +a scaled fringe image (also derived from a blank sky image), and +trimming the image of unwanted lines or columns such as the overscan +strip. The processing may change the pixel datatype on disk (IRAF allows +seven image datatypes); usually from 16 bit integer to real format. +Two special operations are also supported for scan mode and one +dimensional zero level and flat field calibrations; i.e. the same +calibration is applied to each CCD readout line. Any set of operations +may be done simultaneously over a list of images in a highly efficient +manner. The reduction operations are recorded in the image header and +may also be logged on the terminal and in a log file. +.PP +The package also provides tools for combining multiple exposures +of object and calibration images to improve the statistical accuracy of +the observations and to remove transient bad pixels. The combining +operation scales images of different exposure times, adjusts for +variable sky background, statistically weights the images by their +signal-to-noise, and provides a number of useful algorithms for +detecting and rejecting transient bad pixels. +.PP +Other tasks are provided for listing reduction information about +the images, deriving secondary calibration images (such as sky +corrected flat fields or illumination correction images), and easily +setting the package parameters for different instruments. +.PP +This paper is organized as follows. There is a section giving an +overview of how the package is used to reduce CCD data. This gives the +user's perspective and illustrates the general ease of use. The next +section describes many of the features of the package contributing to +its ease of use, automation, and generality. The next two sections +describe the major tools and algorithms in some detail. This includes +discussions about achieving high efficiency. Finally the status of the +package and its use at NOAO is given. References to additional +documentation about IRAF and the CCD reduction package and an appendix +listing the individual tasks in the package are found at the end of +this paper. +.NH +A User's Overview +.PP +This section provides an overview of reducing data with the IRAF CCD +reduction package. There are many variations in usage depending on the +type of data, whether the image headers contain information about the +data which may be used by the tasks, and the scientific goal. Only a +brief example is given. A more complete discussion of usage and +examples is given in \fIA User's Guide to the IRAF CCDRED Package\fR. +The package was developed within the IRAF system and so makes use of +all the sophisticated features provided. These features are also +summarized here for those not familiar with IRAF since they are an +important part of using the package. +.PP +Since the IRAF system is widely distributed and runs on a wide variety +of computers, the site of the CCD reductions might be at the telescope, +a system at the observatory provided for this purpose, or at the +user's home computer. The CCD images to be processed are either +available immediately as the data is taken, transferred from the data taking +computer via a network link (the method adopted at NOAO), or transferred +to the reduction computer via a medium such as magnetic tape in FITS +format. The flexibility in reduction sites and hardware is one of the +virtues of the IRAF-based CCD reduction package. +.PP +IRAF tasks typically have a number of parameters which give the user +control over most aspects of the program. This is possible since the +parameters are kept in parameter files so that the user need not enter +a large number of parameters every time the task is run. The user may +change any of these parameters as desired in several ways, such as by +explicit assignment and using an easy to learn and use, +fill-in-the-value type of screen editor. The parameter values are +\fIlearned\fR so that once a user sets the values they are maintained +until the user changes them again; even between login sessions. +.PP +The first step in using the CCD reduction package is to set the default +processing parameters for the data to be reduced. These parameters include +a database file describing the image header keyword translations and +default values, the processing operations desired (operations +required vary with instrument and observer), the calibration image names, +and certain special parameters for special types of observations such +as scan mode. A special script task (a command procedure) is available +to automatically set the default values, given the instrument name, to standard +values defined by the support staff. Identifying the instrument in this +way may be all the novice user need do though most people quickly learn +to adjust parameters at will. +.PP +As an example suppose there is an instrument identified as \fLrca4m\fR +for an RCA CCD at the NOAO 4 meter telescope. The user gives the command + +.ft L + cl> setinstrument rca4m +.ft R + +which sets the default parameters to values suggested by the support staff +for this instrument. The user may then change these suggested values if +desired. In this example the processing switches are set to perform +overscan bias subtraction, zero level image subtraction, flat fielding, +and trimming. +.PP +The NOAO image headers contain information identifying the type of +image, such as object, zero level, and flat field, the filter used to +match flat fields with object images, the location of the overscan bias +data, the trim size for the data, and whether the image has been +processed. With this information the user need not worry about +selecting images, pairing object images with calibration images, or +inadvertently reprocessing an image. +.PP +The first step is to combine multiple zero level and flat field observations +to reduce the effects of statistical noise. This is done by the +commands + +.nf +.ft L + cl> zerocombine *.imh + cl> flatcombine *.imh +.ft R +.fi + +The "cl> " is the IRAF command language prompt. The first command says +look through all the images and combine the zero level images. The +second command says look through all the images and combine the flat +field images by filter. What could be simpler? Some \fIhidden\fR (default) +parameters the user may modify are the combined image name, whether to +process the images first, and the type of combining algorithm to use. +.PP +The next step is to process the images using the combined calibration +images. The command is + +.ft L + cl> ccdproc *.imh +.ft R + +This command says look through all the images, find the object images, +find the overscan data based on the image header and subtract the +bias, subtract the zero level calibration image, divide by the flat field +calibration image, and trim the bias data and edge lines and columns. +During this operation the task recognizes that the +zero level and flat field calibration images have not been processed +and automatically processes them when they are needed. The log output +of this task, which may be to the terminal, to a file, or both, shows +how this works. + +.nf +.ft L + ccd003: Jun 1 15:12 Trim data section is [3:510,3:510] + ccd003: Jun 1 15:12 Overscan section is [520:540,*], mean=485.0 + Dark: Jun 1 15:12 Trim data section is [3:510,3:510] + Dark: Jun 1 15:13 Overscan section is [520:540,*], mean=484.6 + ccd003: Jun 1 15:13 Dark count image is Dark.imh + FlatV: Jun 1 15:13 Trim data section is [3:510,3:510] + FlatV: Jun 1 15:14 Overscan section is [520:540,*], mean=486.4 + ccd003: Jun 1 15:15 Flat field image is FlatV.imh, scale=138.2 + ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] + ccd004: Jun 1 15:16 Overscan section is [520:540,*], mean=485.2 + ccd004: Jun 1 15:16 Dark count image is Dark.imh + ccd004: Jun 1 15:16 Flat field image is FlatV.imh, scale=138.2 + \fI<... more ...>\fL + ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] + ccd013: Jun 1 15:23 Overscan section is [520:540,*], mean=482.4 + ccd013: Jun 1 15:23 Dark count image is Dark.imh + FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] + FlatB: Jun 1 15:23 Overscan section is [520:540,*], mean=486.4 + ccd013: Jun 1 15:24 Flat field image is FlatB.imh, scale=132.3 + \fI<... more ...>\fL +.ft R +.fi + +.PP +The log gives the name of the image and a time stamp for each entry. +The first image is ccd003. It is to be trimmed to the specified +size given as an \fIimage section\fR, an array notation used commonly +in IRAF to specify subsections of images. The location of the +overscan data is also given by an image section which, in this case, +was found in the image header. The mean bias level of the overscan +is also logged though the overscan is actually a function of the +readout line with the order of the function selected by the user. +.PP +When the task comes to subtracting the zero level image it first +notes that the calibration image has not been processed and switches +to processing the zero level image. Since it knows it is a zero level +image the task does not attempt to zero level or flat field correct +this image. After the zero level image has been processed the task +returns to the object image only to find that the flat field image +also has not been processed. It determines that the object image was +obtained with a V filter and selects the flat field image having the same +filter. The flat field image is processed through the zero level correction +and then the task again returns to the object image, ccd003, which it +finishes processing. +.PP +The next image, ccd004, is also a V filter +observation. Since the zero level and V filter flat field have been +processed the object image is processed directly. This continues +for all the object images except for a detour to process the B filter flat +field when the task first encounters a B filter object image. +.PP +In summary, the basic usage of the CCD reduction package is quite simple. +First, the instrument is identified and some parameters for the data +are set. Calibration images are then combined if needed. Finally, +the processing is done with the simple command + +.ft L + cl> ccdproc *.imh& +.ft R + +where the processing is performed as a \fIbackground job\fR in this example. +This simplicity was a major goal of the package. +.NH +Features of the Package +.PP +This section describes some of the special features of the package +which contribute to its ease of use, generality, and efficiency. +The major criteria for ease of use are to minimize the user's record keeping +involving input and output image names, the types of images, subset +parameters such as filters which must be kept separate, and the state +of processing of each image. The goal is to allow input images to +be specified using simple wildcards, such as "*.imh" to specify all +images, with the knowledge that the task will only operate on images +for which it makes sense. To accomplish this the tasks must be able to +determine the type of image, subset, and the state of processing from +the image itself. This is done by making use of image header parameters. +.PP +For generality the package does not require any image header information +except the exposure time. It is really not very much more difficult to +reduce such data. Mainly, the user must be more explicit about specifying +images and setting task parameters or add the information to the image +headers. Some default header information may also be set in the image +header translation file (discussed below). +.PP +One important image header parameter is the image type. This +discriminates between object images and various types of calibration +images such as flat field, zero level, dark count, comparison arcs, +illumination, and fringe images. This information is used in two +ways. For most of the tasks the user may select that only one type of +image be considered. Thus, all the flat field images may be selected +for combining or only the processing status of the object images be +listed. The second usage is to allow the processing tasks to identify +the standard calibration images and apply only those operations which +make sense. For example, flat field images are not divided by a +flat field. This allows the user to set the processing operations +desired for the object images without fear of misprocessing the +calibration images. The image type is also used to automatically +select calibration images from a list of images to be processed instead +of explicitly identifying them. +.PP +A related parameter specifies the subset. For certain operations the +images must have a common value for this parameter. This parameter is +often the filter but it may also apply to a grating or aperture, for example. +The subset parameter is used to identify the appropriate flat field +image to apply to an image or to select common flat fields to be combined +into a higher quality flat field. This is automatic and the user need not +keep track of which image was taken with which filter or grating. +.PP +The other important image header parameters are the processing flags. +These identify when an image has been processed and also act as a history +of the operation including calibration images used and other parameter +information. The usage of these parameters is obvious; it allows the +user to include processed images in a wildcard list knowing that the +processing will not be repeated and to quickly determine the processing +status of the image. +.PP +Use of image header parameters often ties the software to the a +particular observatory. To maintain generality and usefulness for data +other than that at NOAO, the CCD reduction package was designed to +provide a translation between parameters requested by the package and +those actually found in the image header. This translation is defined +in a simple text file which maps one keyword to another and also gives +a default value to be used if the image header does not include a +value. In addition the translation file maps the arbitrary strings +which may identify image types to the standard types which the package +recognizes. This is a relatively simple scheme and does not allow for +forming combinations or for interpreting values which are not simple +such as embedding an exposure time as part of a string. A more complex +translation scheme may prove desirable as experience is gained with +other types of image header formats, but by then a general header translation +ability and/or new image database structure may be a standard IRAF +feature. +.PP +This feature has proven useful at NOAO. During the course of +developing the package the data taking system was modernized by +updating keywords and adding new information in the image headers, +generally following the lines laid out by the \fBccdred\fR package. +However, there is a period of transition and it is also desirable to +reduce preexisting data. There are several different formats for this +data. The header translation files make coping with these different +formats relatively easy. +.PP +A fundamental aspect of the package is that the processing +modifies the images. In other words, the reduction operations are +performed directly on the image. This "feature" further simplifies +record keeping, frees the user from having to form unique output image +names, and minimizes the amount of disk space required. There +are two safety features in this process. First, the modifications do +not take effect until the operation is completed on the image. This +allows the user to abort the task without leaving the image data in a +partially processed state and protects data if the computer +crashes. The second feature is that there is a parameter which may be +set to make a backup of the input data with a particular prefix; for +example "b", "orig", or "imdir$" (a logical directory prefix). This +backup feature may be used when there is sufficient disk space, when +learning to use the package, or just to be cautious. +.PP +In a similar effort to efficiently manage disk space, when combining +images into a master object or calibration image, there is an option to +delete the input images upon completion of the combining operation. +Generally this is desirable when there are many calibration exposures, +such as zero level or flat field images, which are not used after they +are combined into a final calibration image. +.PP +The goal of generality for many instruments at +different observatories inherently conflicts with the goal of ease of +use. Generality requires many parameters and options. This is +feasible in the CCD reduction package, as well as the other IRAF packages, +because of the IRAF parameter handling mechanism. In \fBccdred\fR +there still remains the problem of setting the parameters appropriately +for a particular instrument, image header format, and observatory. +.PP +To make this convenient there is a task, \fBsetinstrument\fR, that, +based on an instrument name, runs a setup script for the instrument. +An example of this task was given in the previous section. +The script may do any type of operation but mainly it sets default +parameters. The setup scripts are generally created by the support staff +for the instrument. The combination of the setup script and the +instrument translation file make the package, in a sense, programmable +and achieves the desired instrument/observatory generality with ease of use. +.NH +CCD Processing +.PP +This section describes in some detail how the CCD processing is performed. +The task which does the basic CCD processing is call \fBccdproc\fR. +From the point of view of usage the task is very simple but a great deal +is required to achieve this simplicity. The approach we take in describing +the task is to follow the flow of control as the task runs with digressions +as appropriate. +.PP +The highest level of control is a loop over the input images; all the +operations are performed successively on each image. It is common for +IRAF tasks which operate on individual images to allow the operation to +be repeated automatically over a list of input images. This is important +in the \fBccdred\fR package because data sets are often large and the +processing is generally the same for each image. It would be tedious +to have to give the processing command for each image to be processed. +If an error occurs while processing an image the error is +printed as a warning and processing continues with the next image. +This provides protection primarily against mistyped or nonexistent images. +.PP +Before the first image is processed the calibration images are +identified. There are two ways to specify calibration images; +explicitly via task parameters or implicitly as part of the list of +images to be processed. Explicitly identifying calibration images +takes precedence over calibration images in the input list. Specifying +calibration images as part of the input image list requires that the +image types can be determined from the image header. Using the input +list provides a mechanism for breaking processing up into sets of +images (possibly using files containing the image names for each set) +each having their own calibration images. One can, of course, +selectively specify input and calibration images, but whenever possible +one would like to avoid having to specify explicit images to process +since this requires record keeping by the user. +.PP +The first step in processing an image is to check that it is of the +appropriate image type. The user may select to process images of only +one type. Generally this is object images since calibration images are +automatically processed as needed. Images which are not of the desired +type are skipped and the next image is considered. +.PP +A temporary output image is created next. The output pixel datatype on +disk may be changed at this point as selected by the user. +For example it is common for the raw CCD images to be digitized as 16 +bit integers but after calibration it is sometimes desirable to have +real format pixels. If no output pixel datatype is specified the +output image takes the same pixel datatype as the input image. The +processing is done by operating on the input image and writing the +results to a temporary output image. When the processing is complete +the output image replaces the input image. This gives the effect of +processing the images in place but with certain safeguards. If the +computer crashes or the processing is interrupted the integrity of the +input image is maintained. The reasons for chosing to process the +images in this way are to avoid having to generate new image names (a +tiresome record keeping process for the user), to minimize disk +usage, and generally the unprocessed images are not used once they have +been processed. When dealing with large volumes of data these reasons +become fairly important. However, the user may specify a backup prefix +for the images in which case, once the processing is completed, the +original input image is renamed by appending it to the prefix (or with +an added digit if a previous backup image of the same name exits) +before the processed output image takes the original input name. +.PP +The next step is to determine the image geometry. Only a subsection of +the raw image may contain the CCD data. If this region is specified by +a header parameter then the processing will affect only this region. +This allows calibration and other data to be part of the image. +Normally, the only other data in a image is overscan or prescan data. +The location of this bias data is determined from the image header or +from a task parameter (which overrides the image header value). To +relate calibration images of different sizes and to allow for readout +of only a portion of the CCD detector, a header parameter may relate +the image data coordinates to the full CCD coordinates. Application of +calibration image data and identifying bad pixel regions via a bad +pixel file is done in this CCD coordinate system. The final +geometrical information is the region of the input image to be output +after processing; an operation called trimming. This is defined by an +image header parameter or a task parameter. Trimming of the image is +selected by the user. Any or all of this geometry information may be +absent from the image and appropriate defaults are used. +.PP +Each selected operation which is appropriate for the image type is then +considered. If the operation has been performed previously it will not +be repeated. If all selected operations have been performed then the +temporary output image is deleted and the input image is left +unchanged. The next image is then processed. +.PP +For each selected operation to be performed the pertinent data is +determined. This consists of such things as the name of the +calibration image, scaling factors, the overscan bias function, etc. +Note that at this point only the parameters are determined, the +operation is not yet performed. This is because operations are not +performed sequentially but simultaneously as described below. Consider +flat fielding as an example. First the input image is checked to see +if it has been flat fielded. Then the flat field calibration image is +determined. The flat field image is checked to see if it has been +processed. If it has not been processed then it is processed by +calling a procedure which is essentially a copy of the main processing +program. After the flat field image has been processed, parameters +affecting the processing, such as the flat field scale factor +(essentially the mean of the flat field image), are determined. A log +of the operation is then printed if desired. +.PP +Once all the processing operations and parameters have been defined the +actual processing begins. One of the key design goals was that the +processing be efficient. There are two primary methods used to achieve +this goal; separate processing paths for 16 bit integer data and +floating point data and simultaneous operations. If the image, the +calibration images, and the output image (as selected by the user) are +16 bit integer pixel datatypes then the image data is read and written +as integer data. This eliminates internal datatype conversions both +during I/O and during computations. However, many operations include +use of real factors such as the overscan bias, dark count exposure +scaling, and flat field scaling which causes the computation to be done +in real arithmetic before the result is stored again as an integer +value. In any case there is never any loss of precision except when +converting the output pixel to short integer. If any of the images are +not integer then a real internal data path is used in which input and +output image data are converted to real as necessary. +.PP +For each data path the processing proceeds line-by-line. For each line +in the output image data region (ignoring pixels outside the data area +and pixels which are trimmed) the appropriate input data and +calibration data are obtained. The calibration data is determined from +the CCD coordinates of the output image and are not necessarily from +the same image line or columns. The input data is copied to the output +array while applying bad pixel corrections and trimming. The line is +then processed using a specially optimized procedure. This procedure +applies all operations simultaneously for all combinations of +operations. As an example, consider subtracting an overscan bias, +subtracting a zero level, and dividing by a flat field. The basic +kernel of the task, where the bulk of the CPU time is used, is + +.nf +.ft L + do i = 1, n + out[i] = (out[i] - overscan - zero[i]) * flatscale / flat[i] +.ft R +.fi + +Here, \fIn\fR is the number of pixels in the line, \fIoverscan\fR is +the overscan bias value for the line, \fIzero\fR is the zero level data +from the zero level image, \fIflatscale\fR is the mean of the flat +field image, and \fIflat\fR is the flat field data from the flat +field image. Note the operations are not applied sequentially but +in a single statement. This is the most efficient method and there is +no need for intermediate images. +.PP +Though the processing is logically performed line-by-line in the program, +the image I/O from the disk is not done this way. The IRAF virtual +operating system image interface automatically provides multi-line +buffering for maximal I/O efficiency. +.PP +In many image processing systems it has been standard to apply operations +sequentially over an image. This requires producing intermediate images. +Since this is clearly inefficient in terms of I/O it has been the practice +to copy the images into main memory and operate upon them there until +the final image is ready to be saved. This has led to the perception +that in order to be efficient an image processing system \fImust\fR +store images in memory. This is not true and the IRAF CCD reduction +package illustrates this. The CCD processing does not use intermediate +images and does not need to keep the entire image in main memory. +Furthermore, though of lesser importance than I/O, the single statement method +illustrated above is more efficient than multiple passes through the +images even when the images are kept in main memory. Finally, as CCD +detectors increase in size and small, fast, and cheap processors become +common it is a distinct advantage to not require the large amounts of +memory needed to keep entire images in memory. +.PP +There is one area in which use of main memory can improve performance +and \fBccdproc\fR does take advantage of it if desired. The calibration +images usually are the same for many input images. By specifying the +maximum amount of memory available for storing images in memory +the calibration images may be stored in memory up to that amount. +By parameterizing the memory requirement there is no builtin dependence +on large memory! +.PP +After processing the input image the last steps are to log the operations +in the image header using processing keywords and replace the input +image by the output image as described earlier. The CCD coordinates +of the data are recorded in the header, even if not there previously, to +allow further processing on the image after the image has been trimmed. +.NH +Combining Images +.PP +The second important tool in the CCD reduction package is a task to combine +many images into a single, higher quality image. While this may also be +done with more general image processing tools (the IRAF task \fBimsum\fR +for example) the \fBccdred\fR tasks include special CCD dependent features such +as recognizing the image types and using the image header translation +file. Combining images is often done +with calibration images, which are easy to obtain in number, where it +is important to minimize the statistical noise so as to not affect the +object images. Sometimes object images also are combined. +The task is called \fBcombine\fR and there are special versions of +this task called \fBzerocombine, darkcombine\fR, and \fBflatcombine\fR +for the standard calibration images. +.PP +The task takes a list of input images to be combined. As output there +is the combined image, an optional sigma image, and optional log output either +to the terminal, to a log file, or both. A subset or subsets +of the input images may be selected based on the image type and a +subset parameter such as the filter. As with the processing task, +this allows selecting images without having to explicitly list each +image from a large data set. When combining based on a subset parameter +there is an output image, and possibly a sigma image, for each separate subset. +The output image pixel datatype may also be changed during combining; +usually from 16 bit integer input to real output. +The sigma image is the standard deviation of the input images about the +output image. +.PP +Except for summing the images together, +combining images may require correcting for variations between the images +due to differing exposure times, sky background, extinctions, and +positions. Currently, extinction corrections and registration are +not included but scaling and shifting corrections are included. +The scaling corrections may be done by exposure times or by computing +the mode in each image. Additive shifting is also done by computing +the mode in the images. The region of the image in which the mode +is computed can be specified but by default the whole image is used. +A scaling correction is used when the flux level or sensitivity is varying. +The offset correction is used when the sky brightness is varying independently +of the object brightness. If the images are not scaled then special +data paths combine the images more efficiently. +.PP +Except for medianing and summing, the images are combined by averaging. +The average may be weighted by + +.nf +.ft L + weight = (N * scale / mode) ** 2 +.ft R +.fi + +where \fIN\fR is the number of images previously combined (the task +records the number of images combined in the image header), \fIscale\fR +is the relative scale (applied by dividing) from the exposure time or +mode, and \fImode\fR is the background mode estimate used when adding a +variable offset. +.PP +The combining operation is the heart of the task. There are a number +algorithms which may be used as well as applying statistical weights. +The algorithms are used to detect and reject deviant pixels, such as +cosmic rays. +The choice of algorithm depends on the data, the number of images, +and the importance of rejecting cosmic rays. The more complex the +algorithm the more time consuming the operation. +The list below summarizes the algorithms. +Further algorithms may be added in time. + +.IP "Sum - sum the input images" +.br +The input images are combined by summing. Care must be taken +not to exceed the range of the 16 bit integer datatype when summing if the +output datatype is of this type. Summing is the only algorithm in which +scaling and weighting are not used. Also no sigma image is produced. +.IP "Average - average the input images" +.br +The input images are combined by averaging. The images may be scaled +and weighted. There is no pixel rejection. A sigma image is produced +if more than one image is combined. +.IP "Median - median the input images" +.br +The input images are combined by medianing each pixel. Unless the images +are at the same exposure level they should be scaled. The sigma image +is based on all the input images and is only an approximation to the +uncertainty in the median estimates. +.IP "Minreject, maxreject, minmaxreject - reject extreme pixels" +.br +At each pixel the minimum, maximum, or both are excluded from the +average. The images should be scaled and the average may be +weighted. The sigma image requires at least two pixels after rejection +of the extreme values. These are relatively fast algorithms and are +a good choice if there are many images (>15). +.IP "Threshold - reject pixels above and below specified thresholds" +.br +The input images are combined with pixels above and below specified +threshold values (before scaling) excluded. The images may be scaled +and the average weighted. The sigma image also has the rejected +pixels excluded. +.IP "Sigclip - apply a sigma clipping algorithm to each pixel" +.br +The input images are combined by applying a sigma clipping algorithm +at each pixel. The images should be scaled. This only rejects highly +deviant points and so +includes more of the data than the median or minimum and maximum +algorithms. It requires many images (>10-15) to work effectively. +Otherwise the bad pixels bias the sigma significantly. The mean +used to determine the sigmas is based on the "minmaxrej" algorithm +to eliminate the effects of bad pixels on the mean. Only one +iteration is performed and at most one pixel is rejected at each +point in the output image. After the deviant pixels are rejected the final +mean is computed from all the data. The sigma image excludes the +rejected pixels. +.IP "Avsigclip - apply a sigma clipping algorithm to each pixel" +.br +The input images are combined with a variant of the sigma clipping +algorithm which works well with only a few images. The images should +be scaled. For each line the mean is first estimated using the +"minmaxrej" algorithm. The sigmas at each point in the line are scaled +by the square root of the mean, that is a Poisson scaling of the noise +is assumed. These sigmas are averaged to get a line estimate of the +sigma. Then the sigma at each point in the line is estimated by +multiplying the line sigma by the square root of the mean at that point. As +with the sigma clipping algorithm only one iteration is performed and +at most one pixel is rejected at each point. After the deviant pixels +are rejected the file mean is computed from all the data. The sigma +image excludes the rejected pixels. +.RE +.PP +The "avsigclip" algorithm is the best algorithm for rejecting cosmic +rays, especially with a small number of images, but it is also the most +time consuming. With many images (>10-15) it might be advisable to use +one of the other algorithms ("maxreject", "median", "minmaxrej") because +of their greater speed. +.PP +This task also has several design features to make it efficient and +versatile. There are separate data paths for integer data and real +data; as with processing, if all input images and the output image are +of the same datatype then the I/O is done with no internal conversions. +With mixed datatypes the operations are done as real. Even in the +integer path the operations requiring real arithmetic to preserve the +accuracy of the calculation are performed in that mode. There is +effectively no limit to the number of images which may be combined. +Also, the task determines the amount of memory available and buffers +the I/O as much as possible. This is a case where operating on images +from disk rather than in memory is essential. +.NH +Status and Conclusion +.PP +The initial implementation of the IRAF \fBccdred\fR package was +completed in June 1987. It has been in use at the National Optical +Astronomy Observatories since April 1987. The package was not +distributed with Version 2.5 of IRAF (released in August 1987) but is +available as a separate installation upon request. It will be part of +future releases of IRAF. +.PP +At NOAO the CCD reduction package is available at the telescopes as the +data is obtained. This is accomplished by transferring the images from +the data taking computer to a Sun workstation (Sun Microsystems, Inc.) +initially via tape and later by a direct link. There are several +reasons for adopting this architecture. First, the data acquisition +system is well established and is dedicated to its real-time function. +The second computer was phased in without disrupting the essential +operation of the telescopes and if it fails data taking may continue +with data being stored on tape. The role of the second computer is to +provide faster and more powerful reduction and analysis capability not +required in a data acquisition system. In the future it can be more +easily updated to follow the state of the art in small computers. As +CCD detectors get larger the higher processing speeds will be essential +to keep up with the data flow. +.PP +By writing the reduction software in the high level, portable, IRAF +system the users have the capability to process their data from the +basic CCD reductions to a full analysis at the telescope. Furthermore, +the same software is widely available on a variety of computers if +later processing or reprocessing is desired; staff and visitors at NOAO +may also reduce their data at the headquarters facilities. The use of +a high level system was also essential in achieving the design goals; +it would be difficult to duplicate this complex package without +the rich programming environment provided by the IRAF system. +.NH +References +.PP +The following documentation is distributed by the National Optical +Astronomy Observatories, Central Computer Services, P.O. Box 26732, +Tucson, Arizona, 85726. A comprehensive description of the IRAF system +is given in \fIThe IRAF Data Reduction and Analysis System\fR by Doug +Tody (also appearing in \fIProceedings of the SPIE - Instrumentation in +Astronomy VI\fR, Vol. 627, 1986). A general guide to using IRAF is \fIA +User's Introduction to the IRAF Command Language\fR by Peter Shames +and Doug Tody. Both these documents are also part of the IRAF +documentation distributed with the system. +.PP +A somewhat more tutorial description of the \fBccdred\fR package is +\fIA User's Guide to the IRAF CCDRED Package\fR by the author. +Detailed task descriptions and supplementary documentation are +given in the on-line help library and are part of the user's guide. +.NH +Appendix +.PP +The current set of tasks making up the IRAF CCD Reduction Package, +\fBccdred\fR, are summarized below. + +.nf +.ft L + badpiximage - Create a bad pixel mask image from a bad pixel file + ccdgroups - Group CCD images into image lists + ccdhedit - CCD image header editor + ccdlist - List CCD processing information + ccdproc - Process CCD images + combine - Combine CCD images + darkcombine - Combine and process dark count images + flatcombine - Combine and process flat field images + mkfringecor - Make fringe correction images from sky images + mkillumcor - Make flat field illumination correction images + mkillumflat - Make illumination corrected flat fields + mkskycor - Make sky illumination correction images + mkskyflat - Make sky corrected flat field images +setinstrument - Set instrument parameters + zerocombine - Combine and process zero level images +.fi +.ft R diff --git a/noao/imred/quadred/src/quad/doc/ccdtypes.hlp b/noao/imred/quadred/src/quad/doc/ccdtypes.hlp new file mode 100644 index 00000000..2cec33ea --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/ccdtypes.hlp @@ -0,0 +1,124 @@ +.help ccdtypes Jun87 noao.imred.ccdred +.ih +NAME +ccdtypes -- Description of the CCD image types +.ih +CCDTYPES +The following CCD image types may be specified as the value of the parameter +\fIccdtype\fR: + +.nf + "" - (the null string) all image types + object - object images + zero - zero level images such as a bias or preflash + dark - dark count images + flat - flat field images + illum - iillumination images + fringe - fringe correction images + other - other image types defined in the translation file + none - images without an image type parameter + unknown - image types not defined in the translation file +.fi +.ih +DESCRIPTION +The \fBccdred\fR package recognizes certain standard CCD image types +identified in the image header. The tasks may select images of a +particular CCD image type from image lists with the parameter +\fIccdtype\fR and also recognize and take special actions for +calibration images. + +In order to make use of CCD image type information the header keyword +identifying the image type must be specified in the instrument +translation file. This entry has the form + + imagetyp keyword + +where keyword is the image header keyword. This allows the package to +access the image type string. There must also be a translation between +the image type strings and the CCD types as recognized by the package. +This information consists of lines in the instrument translation file +of the form + + header package + +where header is the exact string given in the image header and package +is one of the types recognized by the package. The image header string +can be virtually anything and if it contains blanks it must be +quoted. The package image types are those given above except for +the null string, "none", and "unknown". That is, these types may +be specified as a CCD image type in selecting images but not as a translations +of image type strings. + +There may be more than one image type that maps to the same package +type. In particular other standard CCD image types, such as comparison +spectra, multiple exposure, standard star, etc., should be mapped to +object or other. There may also be more than one type of flat field, i.e. dome +flat, sky flat, and lamp flat. For more on the instrument translation +file see the help for \fBinstruments\fR. +.ih +EXAMPLES +1. The example entries in the instrument translation file are from the 1986 +NOAO CCD image header format produced by the CAMERA format tape writer. + +.nf + imagetyp data-typ + + 'OBJECT (0)' object + 'DARK (1)' dark + 'PROJECTOR FLAT (2)' flat + 'SKY FLAT (3)' other + 'COMPARISON LAMP (4)' other + 'BIAS (5)' zero + 'DOME FLAT (6)' flat +.fi + +The image header keyword describing the image type is "data-typ". +The values of the image type strings in the header contain blanks so they +are quoted. Also the case of the strings is important. Note that there +are two types of flat field images and two types of other images. + +2. One way to check the image types is with the task \fBccdlist\fR. + +.nf + cl> ccdlist *.imh + Zero.imh[504,1][real][zero][1][OT]:FOCUS L98-193 + Flat1.imh[504,1][real][flat][1][OTZ]:dflat 6v+blue 5s + ccd002.imh[504,504][real][unknown][1][OTZF]:FOCUS L98-193 + ccd003.imh[544,512][short][object][1]:L98-193 + ccd004.imh[544,512][short][object][1]:L98-193 + ccd005.imh[544,512][short][object][1]:L98-193 + oldformat.imh[544,512][short][none][1]:M31 V +.fi + +The unknown type has a header image type of "MUL (8)". The old format +image does not have any header type. + +3. To select only images of a particular type: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd003.imh[544,512][short][object][1]:L98-193 + ccd004.imh[544,512][short][object][1]:L98-193 + ccd005.imh[544,512][short][object][1]:L98-193 + cl> ccdlist *.imh ccdtype=unknown + ccd002.imh[504,504][real][unknown][1][OTZF]:FOCUS L98-193 + cl> ccdlist *.imh ccdtype=none + oldformat.imh[544,512][short][none][1]:M31 V +.fi + +4. To process images with \fBccdproc\fR: + +.nf + cl> ccdproc *.imh + cl> ccdproc *.imh ccdtype=object +.fi + +In the first case all the images will be processed (the default value of +\fIccdtype\fR is ""). However, the task recognizes the calibration +images, such as zero level and flat fields, and processes them appropriately. +In the second case only object images are processed and all other images +are ignored (except if needed as a calibration image). +.ih +SEE ALSO +instruments +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/combine.hlp b/noao/imred/quadred/src/quad/doc/combine.hlp new file mode 100644 index 00000000..b3c0848e --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/combine.hlp @@ -0,0 +1,1030 @@ +.help combine Sep92 noao.imred.ccdred +.ih +NAME +combine -- Combine CCD images using various algorithms +.ih +USAGE +combine input output +.ih +PARAMETERS +.ls input +List of CCD images to combine. Images of a particular CCD image type may be +selected with the parameter \fIccdtype\fR with the remaining images ignored. +.le +.ls output +Output combined image or list of images. If the \fIproject\fR parameter is +no (the typical case for CCD acquisition) then there will be one output image +or, if the \fIsubsets\fR parameter is selected, one output image per subset. +If the images consist of stacks then the \fIproject\fR option allows combining +each input stack into separate output images as given by the image list. +.le +.ls plfile = "" (optional) +Output pixel list file or list of files. If no name is given or the +list ends prematurely then no file is produced. The pixel list file +is a map of the number of pixels rejected or, equivalently, +the total number of input images minus the number of pixels actually used. +The file name is also added to the output image header under the +keyword BPM. +.le +.ls sigma = "" (optional) +Output sigma image or list of images. If no name is given or the list ends +prematurely then no image is produced. The sigma is standard deviation, +corrected for a finite population, of the input pixel values (excluding +rejected pixels) about the output combined pixel values. +.le + +.ls ccdtype = "" +CCD image type to combine. If specified only input images of the specified +type are combined. See \fBccdtypes\fR for the possible image types. +.le +.ls subsets = no +Combine images by subset parameter? If yes then the input images are +grouped by subset parameter and each group combined into a separate output +image. The subset identifier is appended to the output image +name(s). See \fBsubsets\fR for more on the subset parameter. +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le + +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +offsetting, masking, thresholding, and rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "none" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation performed on the pixels remaining after offsetting, +masking and thresholding. The algorithms are discussed in the +DESCRIPTION section. The rejection choices are: + +.nf + none - No rejection + minmax - Reject the nlow and nhigh pixels + ccdclip - Reject pixels using CCD noise parameters + crreject - Reject only positive pixels using CCD noise parameters + sigclip - Reject pixels using a sigma clipping algorithm + avsigclip - Reject pixels using an averaged sigma clipping algorithm + pclip - Reject pixels using sigma based on percentiles +.fi + +.le +.ls project = no +Project (combine) across the highest dimension of the input images? If +no then all the input images are combined to a single output image. If +yes then the highest dimension elements of each input image are combined to +an output image and optional pixel list and sigma images. Each element of +the highest dimension may have a separate offset but there can only be one +mask image. +.le +.ls outtype = "real" (short|integer|long|real|double) +Output image pixel datatype. The pixel datatypes are "double", +"real", "long", "integer", and "short" with highest precedence first. +If none is specified then the highest precedence datatype of the input +images is used. The datatypes may be abbreviated to a single character. +.le +.ls offsets = "none" (none|grid|<filename>) +Input image offsets. The offsets may be specified in a file consisting +of one line per image with the offsets in each dimension forming the +columns. The special case of a grid may be specified by the string: + +.nf + grid [n1] [s1] [n2] [s2] ... +.fi + +where ni is the number of images in dimension i and si is the step +in dimension i. For example "grid 5 100 5 100" specifies a 5x5 +grid with origins offset by 100 pixels. +.le +.ls masktype = "none" (none|goodvalue|badvalue|goodbits|badbits) +Type of pixel masking to use. If "none" then no pixel masking is done +even if an image has an associated pixel mask. The other choices +are to select the value in the pixel mask to be treated as good +(goodvalue) or bad (badvalue) or the bits (specified as a value) +to be treated as good (goodbits) or bad (badbits). The pixel mask +file name comes from the image header keyword BPM. +.le +.ls maskvalue = 0 +Mask value used with the \fImasktype\fR parameter. If the mask type +selects good or bad bits the value may be specified using IRAF notation +for decimal, octal, or hexadecimal; i.e 12, 14b, 0cx to select bits 3 +and 4. +.le +.ls blank = 0. +Output value to be used when there are no pixels. +.le + +.ls scale = "none" (none|mode|median|mean|exposure|@<file>|!<keyword>) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, scale +by the exposure time in the image header, scale by the values in a specified +file, or scale by a specified image header keyword. When specified in +a file the scales must be one per line in the order of the input +images. +.le +.ls zero = "none" (none|mode|median|mean|@<file>|!<keyword>) +Additive zero level image shifts to be applied. The choices are none or +shift by the mode, median, or mean of the specified statistics section, +shift by values given in a file, or shift by values given by an image +header keyword. When specified in a file the zero values must be one +per line in the order of the input images. +.le +.ls weight = "none" (none|mode|median|mean|exposure|@<file>|!<keyword>) +Weights to be applied during the final averaging. The choices are none, +the mode, median, or mean of the specified statistics section, the exposure +time, values given in a file, or values given by an image header keyword. +When specified in a file the weights must be one per line in the order of +the input images. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling and +weighting. If no section is given then the entire region of the input is +sampled (for efficiency the images are sampled if they are big enough). +When the images are offset relative to each other one can precede the image +section with one of the modifiers "input", "output", "overlap". The first +interprets the section relative to the input image (which is equivalent to +not specifying a modifier), the second interprets the section relative to +the output image, and the last selects the common overlap and any following +section is ignored. +.le + +.ce +Algorithm Parameters +.ls lthreshold = INDEF, hthreshold = INDEF +Low and high thresholds to be applied to the input pixels. This is done +before any scaling, rejection, and combining. If INDEF the thresholds +are not used. +.le +.ls nlow = 1, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +These numbers are converted to fractions of the total number of input images +so that if no rejections have taken place the specified number of pixels +are rejected while if pixels have been rejected by masking, thresholding, +or nonoverlap, then the fraction of the remaining pixels, truncated +to an integer, is used. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls sigscale = 0.1 (ccdclip, crreject, sigclip, avsigclip) +This parameter determines when poisson corrections are made to the +computation of a sigma for images with different scale factors. If all +relative scales are within this value of unity and all relative zero level +offsets are within this fraction of the mean then no correction is made. +The idea is that if the images are all similarly though not identically +scaled, the extra computations involved in making poisson corrections for +variations in the sigmas can be skipped. A value of zero will apply the +corrections except in the case of equal images and a large value can be +used if the sigmas of pixels in the images are independent of scale and +zero level. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See the DESCRIPTION section for further details. +.le +.ls grow = 0 +Number of pixels to either side of a rejected pixel along image lines +to also be rejected. This applies only to pixels rejected by one of +the rejection algorithms and not the masked or threshold rejected pixels. +.le + +PACKAGE PARAMETERS + +The package parameters are used to specify verbose and log output and the +instrument and header definitions. +.ih +DESCRIPTION +A set of CCD images are combined by weighted averaging or medianing. Pixels +may be rejected from the combining by using pixel masks, threshold levels, +and rejection algorithms. The images may be scaled multiplicatively or +additively based on image statistics, image header keywords, or text files +before rejection. The images may be combined with integer pixel coordinate +offsets to produce an image bigger than any of the input images. +This task is a variant of the \fBimages.imcombine\fR task specialized +for CCD images. + +The input images to be combined are specified by a list. A subset or +subsets of the input list may be selected using the parameters +\fIccdtype\fR and \fIsubsets\fR. The \fIccdtype\fR parameter selects only +images of a specified standard CCD image type. The \fIsubsets\fR parameter +breaks up the input list into sublists of common subset parameter (filter, +grating, etc.). For more information see \fBccdtypes\fR and +\fBsubsets\fR. This selection process is useful with wildcard templates to +combine, for example, the flat field images for each filter in one step +(see \fBflatcombine\fR). When subsets of the input list are used the +output image and optional pixel file and sigma image are given by root names +with a subset identifier appended by the task. + +If the \fBproject\fR parameter is yes then the highest dimension elements +of each input image are combined to make an output image of one lower +dimension. There is no limit to the number of elements combined in this +case. This case is If the \fBproject\fR is no then the entire input list +is combined to form a single output image per subset. In this case the +images must all have the same dimensionality but they may have different +sizes. There is a software limit of approximately 100 images in this +case. + +The output image header is a copy of the first image in the combined set. +In addition, the number of images combined is recorded under the keyword +NCOMBINE, the exposure time is updated as the weighted average of the input +exposure times, and any pixel list file created is recorded under the +keyword BPM. The output pixel type is set by the parameter \fIouttype\fR. +If left blank then the input datatype of highest precision is used. + +In addition to one or more output combined images there may also be a pixel +list image containing the number of pixels rejected at each point in the +output image, an image containing the sigmas of the pixels combined about +the final output combined pixels, and a log file. The pixel list image is +in the compact pixel list format which can be used as an image in other +programs. The sigma computation is the standard deviation corrected for a +finite population (the n/(n-1) factor) including weights if a weighted +average is used. + +Other input/output parameters are \fIdelete\fR and \fIclobber\fR. The +\fIdelete\fR parameter may be set to "yes" to delete the input images +used in producing an output image after it has been created. This is +useful for minimizing disk space, particularly with large +sets of calibration images needed to achieve high statistical accuracy +in the final calibration image. The \fBclobber\fR parameter allows +the output image names to be existing images which are overwritten (at +the end of the operation). + +An outline of the steps taken by the program is given below and the +following sections elaborate on the steps. + +.nf +o Set the input image offsets and the final output image size. +o Set the input image scales and weights +o Write the log file output +.fi + +For each output image line: + +.nf +o Get input image lines that overlap the output image line +o Reject masked pixels +o Reject pixels outside the threshold limits +o Reject pixels using the specified algorithm +o Reject neighboring pixels along each line +o Combine remaining pixels using the weighted average or median +o Compute sigmas of remaining pixels about the combined values +o Write the output image line, rejected pixel list, and sigmas +.fi + + +OFFSETS + +The images to be combined need not be of the same size or overlap. They +do have to have the same dimensionality which will also be the dimensionality +of the output image. Any dimensional images supported by IRAF may be +used. Note that if the \fIproject\fR flag is yes then the input images +are the elements of the highest dimension; for example the planes of a +three dimensional image. + +The overlap of the images is determined by a set of integer pixel offsets +with an offset for each dimension of each input image. For example +offsets of 0, 10, and 20 in the first dimension of three images will +result in combining the three images with only the first image in the +first 10 colums, the first two images in the next 10 columns and +all three images starting in the 31st column. At the 31st output column +the 31st column of the first image will be combined with the 21st column +of the second image and the 1st column of the third image. + +The output image size is set by the maximum extent in each dimension +of any input image after applying the offsets. In the above example if +all the images have 100 columns then the output image will have 130 +columns corresponding to the 30 column offset in the third image. + +The input image offsets are set using the \fIoffset\fR parameter. There +are three ways to specify the offsets. If the word "none" or the empty +string "" are used then all offsets will be zero and all pixels with the +same coordinates will be combined. The output image size will be equal to +the biggest dimensions of the input images. + +If the input images have offsets in a regular grid or one wants to make +an output image in which the input images are "mosaiced" together in +a grid then the special offset string beginning with the word "grid" +is used. The format is + +.nf + grid [n1] [s1] [n2] [s2] ... +.fi + +where ni is the number of images in dimension i and si is the step in +dimension i. For example "grid 5 100 5 100" specifies a 5x5 grid with +origins offset by 100 pixels. Note that one must insure that the input +images are specified in the correct order. This may best be accomplished +using a "@" list. One useful application of the grid is to make a +nonoverlapping mosaic of a number of images for display purposes. Suppose +there are 16 images which are 100x100. The offset string "grid 4 101 4 +101" will produce a mosaic with a one pixel border having the value set +by \fIblank\fR parameter between the images. + +The offsets may be defined in a file by specifying the file name +in the \fIoffset\fR parameter. (Note that the special file name STDIN +may be used to type in the values terminated by the end-of-file +character). The file consists of a line for each input image. The lines +must be in the same order as the input images and so an "@" list may +be useful. The lines consist of whitespace separated offsets one for +each dimension of the images. In the first example cited above the +offset file might contain: + +.nf + 0 0 + 10 0 + 20 0 +.fi + +where we assume the second dimension has zero offsets. + +The offsets need not have zero for one of the images. The offsets may +include negative values or refer to some arbitrary common point. +When the offsets are read by the program it will find the minimum +value in each dimension and subtract it from all the other offsets +in that dimension. The above example could also be specified as: + +.nf + 225 15 + 235 15 + 245 15 +.fi + +There may be cases where one doesn't want the minimum offsets reset +to zero. If all the offsets are positive and the comment "# Absolute" +appears in the offset file then the images will be combined with +blank values between the first output pixel and the first overlapping +input pixel. Continuing with the above example, the file + +.nf + # Absolute + 10 10 + 20 10 + 30 10 +.fi + +will have the first pixel of the first image in the 11th pixel of the +output image. Note that there is no way to "pad" the other side of +the output image. + + +SCALES AND WEIGHTS + +In order to combine images with rejection of pixels based on deviations +from some average or median they must be scaled to a common level. There +are two types of scaling available, a multiplicative intensity scale and an +additive zero point shift. The intensity scaling is defined by the +\fIscale\fR parameter and the zero point shift by the \fIzero\fR +parameter. These parameters may take the values "none" for no scaling, +"mode", "median", or "mean" to scale by statistics of the image pixels, +"exposure" (for intensity scaling only) to scale by the exposure time +keyword in the image header, any other image header keyword specified by +the keyword name prefixed by the character '!', and the name of a file +containing the scale factors for the input image prefixed by the +character '@'. + +Examples of the possible parameter values are shown below where +"myval" is the name of an image header keyword and "scales.dat" is +a text file containing a list of scale factors. + +.nf + scale = none No scaling + zero = mean Intensity offset by the mean + scale = exposure Scale by the exposure time + zero = !myval Intensity offset by an image keyword + scale = @scales.dat Scales specified in a file +.fi + +The image statistics factors are computed by sampling a uniform grid +of points with the smallest grid step that yields less than 10000 +pixels; sampling is used to reduce the time need to compute the statistics. +If one wants to restrict the sampling to a region of the image the +\fIstatsec\fR parameter is used. This parameter has the following +syntax: + +.nf + [input|output|overlap] [image section] +.fi + +The initial modifier defaults to "input" if absent. The modifiers are useful +if the input images have offsets. In that case "input" specifies +that the image section refers to each input image, "output" specifies +that the image section refers to the output image coordinates, and +"overlap" specifies the mutually overlapping region of the input images. +In the latter case an image section is ignored. + +The statistics are as indicated by their names. In particular, the +mode is a true mode using a bin size which is a fraction of the +range of the pixels and is not based on a relationship between the +mode, median, and mean. Also masked pixels are excluded from the +computations as well as during the rejection and combining operations. + +The "exposure" option in the intensity scaling uses the exposure time +from the image header. If one wants to use a nonexposure time image +header keyword the !<keyword> syntax is available. + +If both an intensity scaling and zero point shift are selected the +multiplicative scaling is done first. Use of both makes sense +if the intensity scaling is the exposure time to correct for +different exposure times and then the zero point shift allows for +sky brightness changes. + +The image statistics and scale factors are recorded in the log file +unless they are all equal, which is equivalent to no scaling. The +intensity scale factors are normalized to a unit mean and the zero +point shifts are adjust to a zero mean. + +Scaling affects not only the mean values between images but also the +relative pixel uncertainties. For example scaling an image by a +factor of 0.5 will reduce the effective noise sigma of the image +at each pixel by the square root of 0.5. Changes in the zero +point also changes the noise sigma if the image noise characteristics +are Poissonian. In the various rejection algorithms based on +identifying a noise sigma and clipping large deviations relative to +the scaled median or mean, one may need to account for the scaling induced +changes in the image noise characteristics. + +In those algorithms it is possible to eliminate the "sigma correction" +while still using scaling. The reasons this might be desirable are 1) if +the scalings are similar the corrections in computing the mean or median +are important but the sigma corrections may not be important and 2) the +image statistics may not be Poissonian, either inherently or because the +images have been processed in some way that changes the statistics. In the +first case because computing square roots and making corrections to every +pixel during the iterative rejection operation may be a significant +computational speed limit the parameter \fIsigscale\fR selects how +dissimilar the scalings must be to require the sigma corrections. This +parameter is a fractional deviation which, since the scale factors are +normalized to unity, is the actual minimum deviation in the scale factors. +For the zero point shifts the shifts are normalized by the mean shift +before adjusting the shifts to a zero mean. To always use sigma scaling +corrections the parameter is set to zero and to eliminate the correction in +all cases it is set to a very large number. + +If the final combining operation is "average" then the images may be +weighted during the averaging. The weights are specified in the +same way as the scale factors. In addition +the NCOMBINE keyword, if present, will be used in the weights. +The weights, scaled to a unit sum, are printed in the log output. + +The weights are only used for the final weighted average and sigma image +output. They are not used to form averages in the various rejection +algorithms. + + +PIXEL MASKS + +A pixel mask is a type of IRAF file having the extension ".pl" which +identifies an integer value with each pixel of the images to which it is +applied. The integer values may denote regions, a weight, a good or bad +flag, or some other type of integer or integer bit flag. In the common +case where many values are the same this file is compacted to be small and +efficient to use. It is also most compact and efficient if the majority of +the pixels have a zero mask value so frequently zero is the value for good +pixels. Note that these files, while not stored as a strict pixel array, +may be treated as images in programs. This means they may be created by +programs such as \fBmkpattern\fR, edited by \fBimedit\fR, examined by +\fBimexamine\fR, operated upon by \fBimarith\fR, graphed by \fBimplot\fR, +and displayed by \fBdisplay\fR. + +At the time of introducing this task, generic tools for creating +pixel masks have yet to be written. There are two ways to create a +mask in V2.10. First if a regular integer image can be created +then it can be converted to pixel list format with \fBimcopy\fR: + +.nf + cl> imcopy template plfile.pl +.fi + +by specifically using the .pl extension on output. Other programs that +can create integer images (such \fBmkpattern\fR or \fBccdred.badpiximage\fR) +can create the pixel list file directly by simply using the ".pl" +extension in the output image name. + +To use pixel masks with \fBcombine\fR one must associate a pixel +mask file with an image by entering the pixel list file name in the +image header under the keyword BPM (bad pixel mask). This can be +done with \fBhedit\fR. Note that the same pixel mask may be associated +with more than one image as might be the case if the mask represents +defects in the detector used to obtain the images. + +If a pixel mask is associated with an image the mask is used when the +\fImasktype\fR parameter is set to a value other than "none". Note that +when it is set to "none" mask information is not used even if it exists for +the image. The values of \fImasktype\fR which apply masks are "goodvalue", +"badvalue", "goodbits", and "badbits". They are used in conjunction with +the \fImaskvalue\fR parameter. When the mask type is "goodvalue" the +pixels with mask values matching the specified value are included in +combining and all others are rejected. Similarly, for a mask type of +"badvalue" the pixels with mask values matching the specified value are +rejected and all others are accepted. The bit types are useful for +selecting a combination of attributes in a mask consisting of bit flags. +The mask value is still an integer but is interpreted by bitwise comparison +with the values in the mask file. + +If a mask operation is specified and an image has no mask image associated +with it then the mask values are taken as all zeros. In those cases be +careful that zero is an accepted value otherwise the entire image will be +rejected. + + +THRESHOLD REJECTION + +In addition to rejecting masked pixels, pixels in the unscaled input +images which are below or above the thresholds given by the parameters +\fIlthreshold\fR and \fIhthreshold\fR are rejected. Values of INDEF +mean that no threshold value is applied. Threshold rejection may be used +to exclude very bad pixel values or as an alternative way of masking +images. In the latter case one can use a task like \fBimedit\fR +or \fBimreplace\fR to set parts of the images to be excluded to some +very low or high magic value. + + +REJECTION ALGORITHMS + +The \fIreject\fR parameter selects a type of rejection operation to +be applied to pixels not masked or thresholded. If no rejection +operation is desired the value "none" is specified. + +MINMAX +.in 4 +A specified fraction of the highest and lowest pixels are rejected. +The fraction is specified as the number of high and low pixels, the +\fInhigh\fR and \fInlow\fR parameters, when data from all the input images +are used. If pixels have been rejected by offseting, masking, or +thresholding then a matching fraction of the remaining pixels, truncated +to an integer, are used. Thus, + +.nf + nl = n * nlow/nimages + 0.001 + nh = n * nhigh/nimages + 0.001 +.fi + +where n is the number of pixels surviving offseting, masking, and +thresholding, nimages is the number of input images, nlow and nhigh +are task parameters and nl and nh are the final number of low and +high pixels rejected by the algorithm. The factor of 0.001 is to +adjust for rounding of the ratio. + +As an example with 10 input images and specifying one low and two high +pixels to be rejected the fractions to be rejected are 0.1 and 0.2 +and the number rejected as a function of n is: + +.nf + n 0 1 2 3 4 5 6 7 8 9 10 + nl 0 0 0 0 0 1 1 1 1 1 2 + nh 0 0 0 0 0 0 0 0 0 0 1 +.fi + +.in -4 +CCDCLIP +.in 4 +If the images are obtained using a CCD with known read out noise, gain, and +sensitivity noise parameters and they have been processed to preserve the +relation between data values and photons or electrons then the noise +characteristics of the images are well defined. In this model the sigma in +data values at a pixel with true value <I>, as approximated by the median +or average with the lowest and highest value excluded, is given by: + +.nf + sigma = ((rn / g) ** 2 + <I> / g + (s * <I>) ** 2) ** 1/2 +.fi + +where rn is the read out noise in electrons, g is the gain in +electrons per data value, s is a sensitivity noise given as a fraction, +and ** is the exponentiation operator. Often the sensitivity noise, +due to uncertainties in the pixel sensitivities (for example from the +flat field), is not known in which case a value of zero can be used. +See the task \fBstsdas.wfpc.noisemodel\fR for a way to determine +these vaues (though that task expresses the read out noise in data +numbers and the sensitivity noise parameter as a percentage). + +The read out noise is specified by the \fIrdnoise\fR parameter. The value +may be a numeric value to be applied to all the input images or a image +header keyword containing the value for each image. Similarly, the +parameter \fIgain\fR specifies the gain as either a value or image header +keyword and the parameter \fIsnoise\fR specifies the sensitivity +noise parameter as either a value or image header keyword. + +The algorithm operates on each output pixel independently. It starts by +taking the median or unweighted average (excluding the minimum and maximum) +of the unrejected pixels provided there are at least two input pixels. The +expected sigma is computed from the CCD noise parameters and pixels more +that \fIlsigma\fR times this sigma below or \fIhsigma\fR times this sigma +above the median or average are rejected. The process is then iterated +until no further pixels are rejected. If the average is used as the +estimator of the true value then after the first round of rejections the +highest and lowest values are no longer excluded. Note that it is possible +to reject all pixels if the average is used and is sufficiently skewed by +bad pixels such as cosmic rays. + +If there are different CCD noise parameters for the input images +(as might occur using the image header keyword specification) then +the sigmas are computed for each pixel from each image using the +same estimated true value. + +If the images are scaled and shifted and the \fIsigscale\fR threshold +is exceedd then a sigma is computed for each pixel based on the +image scale parameters; i.e. the median or average is scaled to that of the +original image before computing the sigma and residuals. + +After rejection the number of retained pixels is checked against the +\fInkeep\fR parameter. If there are fewer pixels retained than specified +by this parameter the pixels with the smallest residuals in absolute +value are added back. If there is more than one pixel with the same +absolute residual (for example the two pixels about an average +or median of two will have the same residuals) they are all added +back even if this means more than \fInkeep\fR pixels are retained. +Note that the \fInkeep\fR parameter only applies to the pixels used +by the clipping rejection algorithm and does not apply to threshold +or bad pixel mask rejection. + +This is the best clipping algorithm to use if the CCD noise parameters are +adequately known. The parameters affecting this algorithm are \fIreject\fR +to select this algorithm, \fImclip\fR to select the median or average for +the center of the clipping, \fInkeep\fR to limit the number of pixels +rejected, the CCD noise parameters \fIrdnoise, gain\fR and \fIsnoise\fR, +\fIlsigma\fR and \fIhsigma\fR to select the clipping thresholds, +and \fIsigscale\fR to set the threshold for making corrections to the sigma +calculation for different image scale factors. + +.in -4 +CRREJECT +.in 4 +This algorithm is identical to "ccdclip" except that only pixels above +the average are rejected based on the \fIhsigma\fR parameter. This +is appropriate for rejecting cosmic ray events and works even with +two images. + +.in -4 +SIGCLIP +.in 4 +The sigma clipping algorithm computes at each output pixel the median or +average excluding the high and low values and the sigma about this +estimate. There must be at least three input pixels, though for this method +to work well there should be at least 10 pixels. Values deviating by more +than the specified sigma threshold factors are rejected. These steps are +repeated, except that after the first time the average includes all values, +until no further pixels are rejected or there are fewer than three pixels. + +After rejection the number of retained pixels is checked against the +\fInkeep\fR parameter. If there are fewer pixels retained than specified +by this parameter the pixels with the smallest residuals in absolute +value are added back. If there is more than one pixel with the same +absolute residual (for example the two pixels about an average +or median of two will have the same residuals) they are all added +back even if this means more than \fInkeep\fR pixels are retained. +Note that the \fInkeep\fR parameter only applies to the pixels used +by the clipping rejection algorithm and does not apply to threshold +or bad pixel mask rejection. + +The parameters affecting this algorithm are \fIreject\fR to select +this algorithm, \fImclip\fR to select the median or average for the +center of the clipping, \fInkeep\fR to limit the number of pixels +rejected, \fIlsigma\fR and \fIhsigma\fR to select the +clipping thresholds, and \fIsigscale\fR to set the threshold for +making corrections to the sigma calculation for different image scale +factors. + +.in -4 +AVSIGCLIP +.in 4 +The averaged sigma clipping algorithm assumes that the sigma about the +median or mean (average excluding the low and high values) is proportional +to the square root of the median or mean at each point. This is +described by the equation: + +.nf + sigma(column,line) = sqrt (gain(line) * signal(column,line)) +.fi + +where the \fIestimated\fR signal is the mean or median (hopefully excluding +any bad pixels) and the gain is the \fIestimated\fR proportionality +constant having units of photons/data number. + +This noise model is valid for images whose values are proportional to the +number of photons recorded. In effect this algorithm estimates a +detector gain for each line with no read out noise component when +information about the detector noise parameters are not known or +available. The gain proportionality factor is computed +independently for each output line by averaging the square of the residuals +(at points having three or more input values) scaled by the median or +mean. In theory the proportionality should be the same for all rows but +because of the estimating process will vary somewhat. + +Once the proportionality factor is determined, deviant pixels exceeding the +specified thresholds are rejected at each point by estimating the sigma +from the median or mean. If any values are rejected the median or mean +(this time not excluding the extreme values) is recomputed and further +values rejected. This is repeated until there are no further pixels +rejected or the number of remaining input values falls below three. Note +that the proportionality factor is not recomputed after rejections. + +If the images are scaled differently and the sigma scaling correction +threshold is exceedd then a correction is made in the sigma +calculations for these differences, again under the assumption that +the noise in an image scales as the square root of the mean intensity. + +After rejection the number of retained pixels is checked against the +\fInkeep\fR parameter. If there are fewer pixels retained than specified +by this parameter the pixels with the smallest residuals in absolute +value are added back. If there is more than one pixel with the same +absolute residual (for example the two pixels about an average +or median of two will have the same residuals) they are all added +back even if this means more than \fInkeep\fR pixels are retained. +Note that the \fInkeep\fR parameter only applies to the pixels used +by the clipping rejection algorithm and does not apply to threshold +or bad pixel mask rejection. + +This algorithm works well for even a few input images. It works better if +the median is used though this is slower than using the average. Note that +if the images have a known read out noise and gain (the proportionality +factor above) then the "ccdclip" algorithm is superior. The two algorithms +are related in that the average sigma proportionality factor is an estimate +of the gain. + +The parameters affecting this algorithm are \fIreject\fR to select +this algorithm, \fImclip\fR to select the median or average for the +center of the clipping, \fInkeep\fR to limit the number of pixels +rejected, \fIlsigma\fR and \fIhsigma\fR to select the +clipping thresholds, and \fIsigscale\fR to set the threshold for +making corrections to the sigma calculation for different image scale +factors. + +.in -4 +PCLIP +.in 4 +The percentile clipping algorithm is similar to sigma clipping using the +median as the center of the distribution except that, instead of computing +the sigma of the pixels from the CCD noise parameters or from the data +values, the width of the distribution is characterized by the difference +between the median value and a specified "percentile" pixel value. This +width is then multipled by the scale factors \fIlsigma\fR and \fIhsigma\fR +to define the clipping thresholds above and below the median. The clipping +is not iterated. + +The pixel values at each output point are ordered in magnitude and the +median is determined. In the case of an even number of pixels the average +of the two middle values is used as the median value and the lower or upper +of the two is the median pixel when counting from the median pixel to +selecting the percentile pixel. The parameter \fIpclip\fR selects the +percentile pixel as the number (if the absolute value is greater +than unity) or fraction of the pixels from the median in the ordered set. +The direction of the percentile pixel from the median is set by the sign of +the \fIpclip\fR parameter with a negative value signifying pixels with +values less than the median. Fractional values are internally converted to +the appropriate number of pixels for the number of input images. A minimum +of one pixel and a maximum corresponding to the extreme pixels from the +median are enforced. The value used is reported in the log output. Note +that the same percentile pixel is used even if pixels have been rejected by +offseting, masking, or thresholding; for example, if the 3nd pixel below +the median is specified then the 3rd pixel will be used whether there are +10 pixels or 5 pixels remaining after the preliminary steps. + +Some examples help clarify the definition of the percentile pixel. In the +examples assume 10 pixels. The median is then the average of the +5th and 6th pixels. A \fIpclip\fR value of 2 selects the 2nd pixel +above the median (6th) pixel which is the 8th pixel. A \fIpclip\fR +value of -0.5 selects the point halfway between the median and the +lowest pixel. In this case there are 4 pixels below the median, +half of that is 2 pixels which makes the percentile pixel the 3rd pixel. + +The percentile clipping algorithm is most useful for clipping small +excursions, such as the wings of bright objects when combining +disregistered observations for a sky flat field, that are missed when using +the pixel values to compute a sigma. It is not as powerful, however, as +using the CCD noise parameters (provided they are accurately known) to clip +about the median. + +The parameters affecting this algorithm are \fIreject\fR to select this +algorithm, \fIpclip\fR to select the percentile pixel, \fInkeep\fR to limit +the number of pixels rejected, and \fIlsigma\fR and \fIhsigma\fR to select +the clipping thresholds. + +.in -4 +GROW REJECTION + +Neighbors of pixels rejected by the rejection algorithms along image lines +may also be rejected. The number of neighbors to be rejected on either +side is specified by the \fIgrow\fR parameter. The rejection only +applies to neighbors along each image line. This is because the +task operates independently on each image line and does not have the +ability to go back to previous lines or maintain a list of rejected +pixels to later lines. + +This rejection step is also checked against the \fInkeep\fR parameter +and only as many pixels as would not violate this parameter are +rejected. Unlike it's application in the rejection algorithms at +this stage there is no checking on the magnitude of the residuals +and the pixels retained which would otherwise be rejected are randomly +selected. + + +COMBINING + +After all the steps of offsetting the input images, masking pixels, +threshold rejection, scaling, and applying a rejection algorithms the +remaining pixels are combined and output. The pixels may be combined +by computing the median or by computing a weighted average. + + +SIGMA OUTPUT + +In addition to the combined image and optional sigma image may be +produced. The sigma computed is the standard deviation, corrected for a +finite population by a factor of n/(n-1), of the unrejected input pixel +values about the output combined pixel values. +.ih +EXAMPLES +1. To average and median images without any other features: + +.nf + cl> combine obj* avg combine=average reject=none + cl> combine obj* med combine=median reject=none +.fi + +2. To reject cosmic rays: + +.nf + cl> combine obs1,obs2 Obs reject=crreject rdnoise=5.1, gain=4.3 +.fi + +3. To make a grid for display purposes with 21 64x64 images: + +.nf + cl> combine @list grid offset="grid 5 65 5 65" +.fi + +4. To apply a mask image with good pixels marked with a zero value and + bad pixels marked with a value of one: + +.nf + cl> hedit ims* bpm badpix.pl add+ ver- + cl> combine ims* final combine=median masktype=goodval +.fi + +5. To scale image by the exposure time and then adjust for varying + sky brightness and make a weighted average: + +.nf + cl> combine obj* avsig combine=average reject=avsig \ + >>> scale=exp zero=mode weight=exp expname=exptime +.fi +.ih +TIME REQUIREMENTS +The following times were obtain with a Sun 4/470. The tests combine +1000x200 images consisting of Poisson noise and cosmic rays generated +with the \fBartdata\fR package. The times, especially the total time, +are approximate and depend on user loads. + +.nf +IMAGES: Number of images (1000x200) and datatype (R=real, S=short) +COMBINE: Combine option +REJECT: Rejection option with grow = 0 + minmax: nlow = 1, nhigh = 1 + ccdclip: lsigma = 3., hsigma = 3, sigscale = 0. + sigclip: lsigma = 3., hsigma = 3, sigscale = 0. + avsigclip: lsigma = 3., hsigma = 3, sigscale = 0. + pclip: lsigma = 3., hsigma = 3, pclip = -0.5 + /a: mclip = no (clip about the average) + /m: mclip = yes (clip about the median) +O M T S: Features used (Y=yes, N=no) +O: offset = "grid 5 10 2 10" +M: masktype = goodval, maskval = 0 + Pixel mask has 2 bad lines and 20 bad columns +T: lthreshold = INDEF, hthreshold = 1100. +S: scale = mode, zero = none, weight = mode +TIME: cpu time in seconds, total time in minutes and seconds + + +IMAGES COMBINE REJECT O M T S TIME + + 10R average none N N N N 1.3 0:08 + 10R average minmax N N N N 4.3 0:10 + 10R average pclip N N N N 17.9 0:32 + 10R average ccdclip/a N N N N 11.6 0:21 + 10R average crreject/a N N N N 11.4 0:21 + 10R average sigclip/a N N N N 13.6 0:29 + 10R average avsigclip/a N N N N 15.9 0:35 + 10R average ccdclip/m N N N N 16.9 0:32 + 10R average crreject/m N N N N 17.0 0:28 + 10R average sigclip/m N N N N 19.6 0:42 + 10R average avsigclip/m N N N N 20.6 0:43 + + 10R median none N N N N 6.8 0:17 + 10R median minmax N N N N 7.8 0:15 + 10R median pclip N N N N 16.9 1:00 + 10R median ccdclip/a N N N N 18.0 0:34 + 10R median crreject/a N N N N 17.7 0:30 + 10R median sigclip/a N N N N 21.1 1:13 + 10R median avsigclip/a N N N N 23.1 0:41 + 10R median ccdclip/m N N N N 16.1 0:27 + 10R median crreject/m N N N N 16.0 0:27 + 10R median sigclip/m N N N N 18.1 0:29 + 10R median avsigclip/m N N N N 19.6 0:32 + + 10R average none N N N Y 6.1 0:36 + 10R median none N N N Y 10.4 0:49 + 10R median pclip N N N Y 20.4 1:10 + 10R median ccdclip/m N N N Y 19.5 0:36 + 10R median avsigclip/m N N N Y 23.0 1:06 + + 10R average none N Y N N 3.5 0:12 + 10R median none N Y N N 8.9 0:21 + 10R median pclip N Y N N 19.9 0:45 + 10R median ccdclip/m N Y N N 18.0 0:44 + 10R median avsigclip/m N Y N N 20.9 0:28 + + 10R average none Y N N N 4.3 0:13 + 10R median none Y N N N 9.6 0:21 + 10R median pclip Y N N N 21.8 0:54 + 10R median ccdclip/m Y N N N 19.3 0:44 + 10R median avsigclip/m Y N N N 22.8 0:51 + + 10R average none Y Y Y Y 10.8 0:22 + 10R median none Y Y Y Y 16.1 0:28 + 10R median pclip Y Y Y Y 27.4 0:42 + 10R median ccdclip/m Y Y Y Y 25.5 0:39 + 10R median avsigclip/m Y Y Y Y 28.9 0:44 + + 10S average none N N N N 2.2 0:06 + 10S average minmax N N N N 4.6 0:12 + 10S average pclip N N N N 18.1 0:33 +.fi +.ih +REVISIONS +.ls COMBINE V2.10.2 +The weighting was changed from using the square root of the exposure time +or image statistics to using the values directly. This corresponds +to variance weighting. Other options for specifying the scaling and +weighting factors were added; namely from a file or from a different +image header keyword. The \fInkeep\fR parameter was added to allow +controlling the maximum number of pixels to be rejected by the clipping +algorithms. The \fIsnoise\fR parameter was added to include a sensitivity +or scale noise component to the noise model. Errors will now delete +the output images. +.le +.ls COMBINE V2.10 +This task was greatly revised to provide many new features. These features +are: + +.nf + o Bad pixel masks + o Combining offset and different size images + o Blank value for missing data + o Combining across the highest dimension (the project option) + o Separating threshold rejection, the rejection algorithms, + and the final combining statistic + o New CCDCLIP, CRREJECT, and PCLIP algorithms + o Rejection now may reject more than one pixel per output pixel + o Choice of a central median or average for clipping + o Choice of final combining operation + o Simultaneous multiplicative and zero point scaling +.fi +.le +.ih +LIMITATIONS +Though this task is essentially not limited by the physical +limits of the host (number of open files, amount of memory) there is a +software limit in the IRAF virtual operating system of about 120 separate +images which may be combined. To combine more images one may combine smaller +groups and then combine those or one may stack the images into a higher +dimensional image using \fBimstack\fR and use the \fIproject\fR option. +.ih +SEE ALSO +image.imcombine, instruments, ccdtypes, icfit, ccdred, guide, darkcombine, +flatcombine, zerocombine, onedspec.scombine wfpc.noisemodel +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/contents.ms b/noao/imred/quadred/src/quad/doc/contents.ms new file mode 100644 index 00000000..8ba2624a --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/contents.ms @@ -0,0 +1,34 @@ +.sp 1i +.ps +2 +.ft B +.ce +Contents +.sp 3 +.ps -2 +.ft R +.sp +1.\h'|0.4i'\fBIntroduction\fP\l'|5.6i.'\0\01 +.sp +2.\h'|0.4i'\fBGetting Started\fP\l'|5.6i.'\0\02 +.sp +3.\h'|0.4i'\fBProcessing Your Data\fP\l'|5.6i.'\0\05 +.br +\h'|0.4i'3.1.\h'|0.9i'Combining Calibration Images\l'|5.6i.'\0\06 +.br +\h'|0.4i'3.2.\h'|0.9i'Calibrations and Corrections\l'|5.6i.'\0\07 +.sp +4.\h'|0.4i'\fBSpecial Processing Operations\fP\l'|5.6i.'\0\08 +.br +\h'|0.4i'4.1.\h'|0.9i'Spectroscopic Flat Fields\l'|5.6i.'\0\08 +.br +\h'|0.4i'4.2.\h'|0.9i'Illumination Corrections\l'|5.6i.'\0\09 +.br +\h'|0.4i'4.3.\h'|0.9i'Sky Flat Fields\l'|5.6i.'\010 +.br +\h'|0.4i'4.4.\h'|0.9i'Illumination Corrected Flat Fields\l'|5.6i.'\010 +.br +\h'|0.4i'4.5.\h'|0.9i'Fringe Corrections\l'|5.6i.'\010 +.sp +5.\h'|0.4i'\fBSummary\fP\l'|5.6i.'\011 +.sp +\h'|0.4i'\fBReferences\fP\l'|5.6i.'\011 diff --git a/noao/imred/quadred/src/quad/doc/cosmicrays.hlp b/noao/imred/quadred/src/quad/doc/cosmicrays.hlp new file mode 100644 index 00000000..c50a129f --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/cosmicrays.hlp @@ -0,0 +1,220 @@ +.help cosmicrays Dec87 noao.imred.ccdred +.ih +NAME +cosmicrays -- Detect and replace cosmic rays +.ih +USAGE +cosmicrays input output +.ih +PARAMETERS +.ls input +List of input images in which to detect cosmic rays. +.le +.ls output +List of output images in which the detected cosmic rays will be replaced +by an average of neighboring pixels. If the output image name differs +from the input image name then a copy of the input image is made with +the detected cosmic rays replaced. If no output images are specified +then the input images are modified in place. In place modification of +an input image also occurs when the output image name is the same as +the input image name. +.le +.ls badpix = "" +List of bad pixel files to be created, one for each input image. If no +file names are given then no bad pixel file is created. The bad pixel +file is a simple list of pixel coordinates for each replaced cosmic ray. +This file may be used in conjunction with \fBbadpixelimage\fR to create +a mask image. +.le +.ls ccdtype = "" +If specified only the input images of the desired CCD image type will be +selected. +.le +.ls threshold = 25. +Detection threshold above the mean of the surrounding pixels for cosmic +rays. The threshold will depend on the noise characteristics of the +image and how weak the cosmic rays may be for detection. A typical value +is 5 or more times the sigma of the background. +.le +.ls fluxratio = 2. +The ratio (as a percent) of the mean neighboring pixel flux to the candidate +cosmic ray pixel for rejection. The value depends on the seeing and the +characteristics of the cosmic rays. Typical values are in the range +2 to 10 percent. +.le +.ls npasses = 5 +Number of cosmic ray detection passes. Since only the locally strongest +pixel is considered a cosmic ray, multiple detection passes are needed to +detect and replace multiple pixel cosmic ray events. +.le +.ls window = 5 +Size of cosmic ray detection window. A square window of either 5 by 5 or +7 by 7 is used to detect cosmic rays. The smaller window allows detection +in the presence of greater background gradients but is less sensitive at +discriminating multiple event cosmic rays from stars. It is also marginally +faster. +.le +.ls interactive = yes +Examine parameters interactively? A plot of the mean flux within the +detection window (x100) vs the flux ratio (x100) is plotted and the user may +set the flux ratio threshold, delete and undelete specific events, and +examine specific events. This is useful for new data in which one is +uncertain of an appropriate flux ratio threshold. Once determined the +task need not be used interactively. +.le +.ls answer +This parameter is used for interactive queries when processing a list of +images. The responses may be "no", "yes", "NO", or "YES". The upper case +responses permanently enable or disable the interactive review while +the lower case reponses allow selective examination of certain input +images. +.le +.ih +OTHER PARAMETERS +There are other parameters which may be defined by the package, as is the +case with \fBccdred\fR, or as part of the task, as is the case with +standalone version in the \fBgeneric\fR package. + +.ls verbose +If yes then a time stamped log of the operation is printed on the standard +output. +.le +.ls logfile +If a log file is specified then a time stamped log of the operation is +recorded. +.le +.ls plotfile +If a plot file is specified then the graph of the flux ratio (x100) vs +the mean flux (x100) is recorded as metacode. This may be spooled or examined +later. +.le +.ls graphics = "stdgraph" +Interactive graphic output device for interactive examination of the +detection parameters. +.le +.ls cursor = "" +Interactive graphics cursor input. If null the graphics display cursor +is used, otherwise a file containing cursor input may be specified. +.le +.ls instrument +The \fBccdred\fR instrument file is used for mapping header keywords and +CCD image types. +.le +.ih +CURSOR COMMANDS + +.nf +d Mark candidate for replacement (applys to '+' points) +q Quit and replace the selected pixels +r Redraw the graph +s Make a surface plot for the candidate nearest the cursor +t Set the flux ratio threshold at the y cursor position +u Mark candidate to not be replaced (applys to 'x' points) +w Adjust the graph window (see \fBgtools\fR) +.fi + +There are no colon commands except those for the windowing options (type +:\help or see \fBgtools\fR). +.ih +DESCRIPTION +Cosmic ray events in each input image are detected and replaced by the +average of the four neighbors. The replacement may be performed +directly on the input image if no output image is specified or if the +output image name is the same as the input image name. If a new image +is created it is a copy of the input image except for the replaced +pixels. The processing keyword CRCOR is added to the output image +header. Optional output includes a log file to which a processing log +is appended, a verbose log output to the standard output (the same as +that in the log file), a plot file showing the parameters of the +detected cosmic ray candidates and the flux ratio threshold used, and a +bad pixel file containing the coordinates of the replaced pixels. The +bad pixel file may be used for plotting purposes or to create a mask +image for display and analysis using the task \fBbadpiximage\fR. This +bad pixel file will be replaced by the IRAF bad pixel facility when it +becomes available. If one wants more than a simple mask image then by +creating a different output image a difference image between the +original and the modified image may be made using \fBimarith\fR. + +This task may be applied to an image previously processed to detect +additional cosmic rays. A warning will be given (because of the +CRCOR header parameter) and the previous processing header keyword will +be overwritten. + +The cosmic ray detection algorithm consists of the following steps. +First a pixel must be the brightest pixel within the specified +detection window (either 5x5 or 7x7). The mean flux in the surrounding +pixels with the second brightest pixel excluded (which may also be a +cosmic ray event) is computed and the candidate pixel must exceed this +mean by the amount specified by the parameter \fIthreshold\fR. A plane +is fit to the border pixels of the window and the fitted background is +subtracted. The mean flux (now background subtracted) and the ratio of +this mean to the cosmic ray candidate (the brightest pixel) are +computed. The mean flux (x100) and the ratio (x100) are recorded for +interactive examination if desired. + +Once the list of cosmic ray candidates has been created and a threshold +for the flux ratio established (either by the parameter \fIfluxratio\fR +or modified interactively) the pixels with ratios below the threshold +are replaced in the output by the average of the four neighboring pixels +(with the second strongest pixel in the detection window excluded if it is +one of these pixels). Additonal pixels may then be detected and replaced +in further passes as specified by the parameter \fInpasses\fR. Note that +only pixels in the vicinity of replaced pixels need be considered in +further passes. + +The division between the peaks of real objects and cosmic rays is made +based on the flux ratio between the mean flux (excluding the center +pixel and the second strongest pixel) and the candidate pixel. This +threshold depends on the point spread function and the distribution of +multiple cosmic ray events and any additional neighboring light caused +by the events. This threshold is not strongly coupled to small changes +in the data so that once it is set for a new type of image data it may +be used for similar images. To set it initially one may examine the +scatter plot of the flux ratio as a function of the mean flux. This +may be done interactively or from the optional plot file produced. + +When the interactive flag is set the user is queried for each image. +Responses may be made for specific images or for all images by using +lower or upper case answers respectively. When the parameters are +examined interactively the user may change the flux ratio threshold +('t' key). Changes made are stored in the parameter file and, thus, +learned for further images. Pixels to be deleted are marked by crosses +and pixels which are peaks of objects are marked by pluses. The user +may explicitly delete or undelete any point if desired but this is only +for special cases near the threshold. In the future keys for +interactive display of the specific detections will be added. +Currently a surface plot of any candidate may be displayed graphically +in four 90 degree rotated views using the 's' key. Note that the +initial graph does not show all the points some of which are clearly +cosmic rays because they have negative mean flux or flux ratio. To +view all data one must rewindow the graph with the 'w' key or ":/" +commands (see \fBgtools\fR). +.ih +EXAMPLES +1. To replace cosmic rays in a set of images ccd*: + +.nf + cl> cosmicrays ccd* new//ccd* + ccd001: Examine parameters interactively? (yes): + [A scatter plot graph is made. One can adjust the threshold.] + [Looking at a few points using the 's' key can be instructive.] + [When done type 'q'.] + ccd002: Examine parameters interactively? (yes): NO + [No further interactive examination is done.] +.fi + +After cleaning one typically displays the images and possibly blinks them. +A difference image or mask image may also be created. + +2. To create a mask image a bad pixel file must be specified. In the +following we replace the cosmic rays in place and create a bad pixel +file and mask image: + +.nf + cl> cosmicrays ccd001 ccd001 badpix=ccd001.bp + cl> badpiximage ccd001.bp ccd001 ccd001bp +.fi +.ih +SEE ALSO +badpixelimage gtools +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/darkcombine.hlp b/noao/imred/quadred/src/quad/doc/darkcombine.hlp new file mode 100644 index 00000000..ef9eb81c --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/darkcombine.hlp @@ -0,0 +1,125 @@ +.help darkcombine Sept93 arcon.quad +.ih +NAME +darkcombine -- Combine and process dark count images +.ih +USAGE +darkcombine input +.ih +PARAMETERS +.ls input +List of dark count images to combine. The \fIccdtype\fR parameter +may be used to select the zero level images from a list containing all +types of data. +.le +.ls output = "Dark" +Output dark count root image name. +.le +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "avsigclip" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation. See \fBcombine\fR for details. +.le +.ls ccdtype = "zero" +CCD image type to combine. If no image type is given then all input images +are combined. +.le +.ls process = no +Process the input images before combining? +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le +.ls scale = "none" (none|mode|median|mean|exposure) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, or scale +by the exposure time given in the image header. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling. +If no section is given then the entire region of the image is +sampled (for efficiency the images are sampled if they are big enough). +.le + +.ce +Algorithm Parameters +.ls nlow = 1, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See \fBcombine\fR for further details. +.le +.ls blank = 0. +Output value to be used when there are no pixels. +.le +.ih +DESCRIPTION +The dark count images in the input image list are combined. +The input images may be processed first if desired. +The original images may be deleted automatically if desired. + +This task is a script which applies \fBquadproc\fR and \fBcombine\fR. The +parameters and combining algorithms are described in detail in the help for +\fBcombine\fR. This script has default parameters specifically set for +dark count images and simplifies the combining parameters. There are other +combining options not included in this task. For these additional +features, such as thresholding, offseting, masking, and projecting, use +\fBcombine\fR. + +The version of \fBdarkcombine\fR in the \fBquad\fR package differs from that +in \fBccdred\fR in that \fBquadproc\fR rather than \fBccdproc\fR is used to +process the images if this is requested. The \fBquad\fR version MUST be +used if process=yes and the input list contains any multi-readout images which +have not been overscan corrected and trimmed. +.ih +EXAMPLES +1. The image data contains four dark count images. To automatically select +them and combine them as a background job using the default combining algorithm: + + cl> darkcombine ccd*.imh& +.ih +SEE ALSO +quadproc, combine +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/flatcombine.hlp b/noao/imred/quadred/src/quad/doc/flatcombine.hlp new file mode 100644 index 00000000..c463655d --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/flatcombine.hlp @@ -0,0 +1,139 @@ +.help flatcombine Sept93 arcon.quad +.ih +NAME +flatcombine -- Combine and process flat field images +.ih +USAGE +flatcombine input +.ih +PARAMETERS +.ls input +List of flat field images to combine. The \fIccdtype\fR parameter +may be used to select the flat field images from a list containing all +types of data. +.le +.ls output = "Flat" +Output flat field root image name. The subset ID is appended. +.le +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "avsigclip" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation. See \fBcombine\fR for details. +.le +.ls ccdtype = "flat" +CCD image type to combine. If no image type is given then all input images +are combined. +.le +.ls process = no +Process the input images before combining? +.le +.ls subsets = yes +Combine images by subset parameter? If yes then the input images are +grouped by subset parameter and each group combined into a separate output +image. The subset identifier is appended to the output and sigma image +names. See \fBsubsets\fR for more on the subset parameter. This is generally +used with flat field images. +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le +.ls scale = "none" (none|mode|median|mean|exposure) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, or scale +by the exposure time given in the image header. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling. +If no section is given then the entire region of the image is +sampled (for efficiency the images are sampled if they are big enough). +.le + +.ce +Algorithm Parameters +.ls nlow = 1, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See \fBcombine\fR for further details. +.le +.ls blank = 1. +Output value to be used when there are no pixels. +.le +.ih +DESCRIPTION +The flat field images in the input image list are combined. If there +is more than one subset (such as a filter or grating) then the input +flat field images are grouped by subset and an combined separately. +The input images may be processed first if desired. However if all +zero level bias effects are linear then this is not necessary and some +processing time may be saved. The original images may be deleted +automatically if desired. + +This task is a script which applies \fBquadproc\fR and \fBcombine\fR. The +parameters and combining algorithms are described in detail in the help for +\fBcombine\fR. This script has default parameters specifically set for +flat field images and simplifies the combining parameters. There are other +combining options not included in this task. For these additional +features, such as thresholding, offseting, masking, and projecting, use +\fBcombine\fR. + +The version of \fBzerocombine\fR in the \fBquad\fR package differs from that +in \fBccdred\fR in that \fBquadproc\fR rather than \fBccdproc\fR is used to +process the images if this is requested. The \fBquad\fR version MUST be +used if process=yes and the input list contains any multi-readout images which +have not been overscan corrected and trimmed. +.ih +EXAMPLES +1. The image data contains four flat field images for three filters. +To automatically select them and combine them as a background job +using the default combining algorithm: + + cl> flatcombine ccd*.imh& + +The final images are "FlatV", "FlatB", and "FlatR". +.ih +SEE ALSO +quadproc, combine, subsets +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/flatfields.hlp b/noao/imred/quadred/src/quad/doc/flatfields.hlp new file mode 100644 index 00000000..94766960 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/flatfields.hlp @@ -0,0 +1,177 @@ +.help flatfields Jun87 noao.imred.ccdred + +.ih +NAME +flatfields -- Discussion of CCD flat field calibrations +.ih +DESCRIPTION +This topic describes the different types of CCD flat fields and +the tasks available in the \fBccdred\fR and spectroscopy packages for +creating them. Flat field calibration is the most important operation +performed on CCD data. This operation calibrates the relative response +of the detector at each pixel. In some cases this is as simple as +taking a special type of observation called a flat field. However, in +many cases this calibration observation must be corrected for +iillumination, scanning, wavelength, and aperture effects. + +The discussion is in three sections; direct imaging, scan mode, +and spectroscopy. Though there are many similarities between these +modes of operation there are important differences in how corrections +are applied to the basic flat field observations. The application of +the flat field calibrations to the observations using \fBccdproc\fR is +the same in all cases, however. +.sh +1. Direct Imaging +The starting point for determining the flat field calibration is an +observation of something which should have uniform response at all +points on the detector. In addition the color of the light falling at +each pixel should be the same as that in an observation so the same +filter must be used when determining the flat field (the issue of the +matching the color of the objects observed at the appropriate pixels is +ignored here). The best calibration observation is of a blank sky. If +an accurate blank sky observation can be obtained then this is all that +is needed for a flat field calibration. This type of flat field might +be called a \fIsky flat\fR, though this term is more often used for a +type of flat field described below. There are two difficulties with +this type of calibration; finding a really blank sky and getting a +sufficiently accurate measurement without using all the observing +time. + +It is usually not possible to get a blank sky observation accurate +enough to calibrate the individual pixels without introducing +undesirable noise. What is generally done is to use a lamp to either +uniformly illuminate a part of the dome or directly illuminate the +field of view. The first type of observation is called a \fIdome +flat\fR and the second is called a \fIprojection flat\fR. We shall call +both of these types of observations \fBlamp flat fields\fR. If the +iillumination is truely uniform then these types of observations are +sufficient for flat field calibration. To get a very accurate flat +field many observations are made and then combined (see +\fBflatcombine\fR). + +Unfortunately, it is sometimes the case that the lamp flat fields +do not illuminate the telescope/detector in the same way as the actual +observations. Calibrating with these flat fields will introduce a +residual large scale iillumination pattern, though it will correctly +calibrate the relative pixel responses locally. There are two ways to +correct for this effect. The first is to correct the flat field +observation. The second is to apply the uncorrected flat field to the +observations and then apply an \fIiillumination\fR correction as a +separate operation. The first is more efficient since it consists of a +single correction applied to each observation but in some cases the +approximate correction is desired immediately, the observation needed +to make the correction has not been taken yet, or the residual +iillumination error is not discovered until later. + +For the two methods there are two types of correction. One is to +use a blank sky observation to correct for the residual iillumination +pattern. This is different than using the sky observation directly as +a flat field calibration in that only the large scale pattern is +needed. Determining the large scale iillumination does not require high +signal-to-noise at each pixel and faint objects in the image can be +either eliminated or ignored. The second method is to remove the large +scale shape from the lamp flat field. This is not as good as using a +blank sky observation but, if there is no such observation and the +iillumination pattern is essentially only in the lamp flat field, this +may be sufficient. + +From the above two paragraphs one sees there are four options. +There is a task in the \fBccdred\fR package for each of these options. +To correct a lamp flat field observation by a blank sky observation, +called a \fIsky flat\fR, the task is \fBmkskyflat\fR. To correct the +flat field for its own large scale gradients, called an \fIiillumination +flat\fR, the task is \fBmkillumflat\fR. To create a secondary +correction to be applied to data processed with the lamp flat field +image the tasks are \fBmkskycor\fR and \fBmkillumcor\fR which are, +respectively, based on a blank sky observation and the lamp flat field +iillumination pattern. + +With this introduction turn to the individual documentation for these +four tasks for further details. +.sh +2. Scan Mode +There are two types of scan modes supported by the \fBccdred\fR +package; \fIshortscan\fR and \fIlongscan\fR (see \fBccdproc\fR for +further details). They both affect the manner in which flat field +calibrations are handled. The shortscan mode produces images which are +the same as direct images except that the light recorded at each pixel +was collected by a number of different pixels. This improves the flat +field calibration. If the flat field images, of the same types +described in the direct imaging section, are observed in the same way +as all other observations, i.e. in scan mode, then there is no +difference from direct imaging (except in the quality of the flat +fields). There is a statistical advantage to observing the lamp or sky +flat field without scanning and then numerically averaging to simulate +the result of the scanning. This improves the accuracy of +the flat fields and might possibly allow direct blank sky observations +to be used for flat fields. The numerical scanning is done in +\fBccdproc\fR by setting the appropriate scanning parameters. + +In longscan mode the CCD detector is read out in such a way that +each output image pixel is the sum of the light falling on all pixels +along the direction of the scan. This reduces the flat field calibration +to one dimension, one response value for each point across the scan. +The one dimensional calibration is obtained from a longscan observation +by averaging all the readout lines. +This is done automatically in \fBccdproc\fR by setting the appropriate +parameters. In this case very good flat fields can be obtained from +one or more blank sky observations or an unscanned lamp observation. Other +corrections are not generally used. +.sh +3. Spectroscopy +Spectroscopic flat fields differ from direct imaging in that the +spectrum of the sky or lamp and transmission variations with wavelength +are part of the observation. Application of such images will introduce +the inverse of the spectrum and transmission into the observation. It +also distorts the observed counts making signal-to-noise estimates +invalid. This, and the low signal in the dispersed light, makes it +difficult to use blank sky observations directly as flat fields. As +with direct imaging, sky observation may be used to correct for +iillumination errors if necessary. At sufficiently high dispersion the +continuous lamp spectrum may be flat enough that the spectral signature +of the lamp is not a problem. Alternatively, flux calibrating the +spectra will also remove the flat field spectral signature. The +spectroscopic flat fields also have to be corrected for regions outside +of the slit or apertures to avoid bad response effects when applying +the flat field calibration to the observations. + +The basic scheme for removing the spectral signature is to average +all the lines or columns across the dispersion and within the aperture +to form an estimate of the spectrum. In addition to the averaging, a +smooth curve is fit to the lamp spectrum to remove noise. This smooth +shape is then divided back into each line or column to eliminate the +shape of the spectrum without changing the shape of the spectrum +in the spatial direction or the small scale response variations. +Regions outside of the apertures are replaced by unity. +This method requires that the dispersion be aligned fairly close to +either the CCD lines or columns. + +This scheme is used in both longslit and multiaperture spectra. +The latter includes echelle, slitlets, aperture masks, and fiber feeds. +For narrow apertures which do not have wider slits for the lamp +exposures there may be problems with flexure and defining a good +composite spectrum. The algorithm for longslit spectra is simpler and +is available in the task \fBresponse\fR in the \fBlongslit\fR package. +For multiaperture data there are problems of defining where the spectra +lie and avoiding regions off of the aperture where there is no signal. +The task which does this is \fBapnormalize\fR in the \fBapextract\fR +package. Note that the lamp observations must first be processed +explicitly for bias and dark count corrections. + +Longslit spectra may also suffer the same types of iillumination +problems found in direct imaging. However, in this case the iillumination +pattern is determined from sky observations (or the flat field itself) +by finding the large scale pattern across the dispersion and at a number +of wavelengths while avoiding the effects of night sky spectrum. The +task which makes this type of correction in the \fBlongslit\fR package +is \fBiillumination\fR. This produces an iillumination correction. +To make sky flats or the other types of corrections image arithmetic +is used. Note also that the sky observations must be explicitly +processed through the flat field stage before computing the iillumination. +.ih +SEE ALSO +.nf +ccdproc, guide, mkillumcor, mkillumflat, mkskycor, mkskyflat +apextract.apnormalize, longslit.response, longslit.iillumination +.fi +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/guide.hlp b/noao/imred/quadred/src/quad/doc/guide.hlp new file mode 100644 index 00000000..d7639a6f --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/guide.hlp @@ -0,0 +1,715 @@ +.help guide Sept93 arcon.quad +.ce +User's Guide to the QUAD Package +.sh +1. Introduction + + This guide provides a brief description of the \fBquad\fR package including +examples of its use for reducing simple CCD data. The \fBquad\fR package +contains all the basic tasks necessary for the reduction of CCD data obtained +using the CTIO array controller Arcon. It is based on the IRAF CCD reduction +package \fBccdred\fR written by Frank Valdes. It incorporates a few special +tasks needed to deal with the peculiarities of Arcon data but most of the +routines are taken directly from the standard package. The way in which one +uses the routines and the basic reduction recipe is unchanged. + +This guide is generic in that it is not tied to any particular type of data. +There may be more specific guides (or "cookbooks") for your data. Detailed +descriptions of the tasks and features of the package are provided in the help +documentation for the package. + +With Arcon the CCD is often readout using four ("quad") or two ("dual") +amplifiers in order to reduce readout time. A feature of such images is that +each readout typically has a slightly different, DC bias level, gain, and +readout noise. As a result both zero frames and uniformly illuminated +exposures show a characteristic chequer board pattern, the sections of the +image read through each amplifier having different levels. In addition, there +will be a separate overscan strip, used to monitor the zero level, for each +readout. The location of these overscan strips in the raw frame depends on +which amplifiers are used (for more on this see \fBquadreadout\fR). Because +of these peculiarities the \fBquad\fR package must be used for the first +reduction steps, overscan correction and trimming, of multi-readout images; +subsequent steps can be performed using \fBquad\fR or \fBccdred\fR. Either +package can be used for the complete reduction of conventional single readout +CCD images. + + The purpose of \fBquad\fR package is to provide tools for the removal of +all the "instrumental signatures" from CCD data taken with Arcon. The standard +reduction operations are: replacement of bad columns and lines by interpolation +from neighboring columns and lines; subtraction of a bias level +determined from overscan or prescan columns or lines; subtraction of a +zero level using a zero exposure time calibration image; subtraction +of a dark count calibration image appropriately scaled to the dark time +exposure; division by a scaled flat field calibration image; division +by an iillumination image (derived from a blank sky image); subtraction +of a scaled fringe image (also derived from a blank sky image); and +trimming the image of unwanted lines or columns such as the overscan +strip. Any set of these operations may be done simultaneously over a list of +images in a highly efficient manner. The reduction operations are +recorded in the image header and may also be logged on the terminal and +in a log file. + + The package also provides tools for combining multiple exposures +of object and calibration images to improve the statistical accuracy of +the observations and to remove transient bad pixels. The combining +operation scales images of different exposure times, adjusts for +variable sky background, statistically weights the images by their +signal-to-noise, and provides a number of useful algorithms for +detecting and rejecting transient bad pixels. + + Other tasks are provided for listing reduction information about +the images, deriving secondary calibration images (such as sky +corrected flat fields or iillumination correction images), and easily +setting the package parameters for different instruments. + +The \fBquad\fR package also includes some tasks which can be used for the +examination of multi-readout images, prior to reducing them, by for instance +calculating simple image statistics, generating histograms, etc. + + There are several important features provided by the \fBccdred\fR package +underpining \fBquad\fR, which makes the reduction of CCD images convenient; +particularly to minimize record keeping. One of these is the ability to +recognize the different types of CCD images. This ability allows the user to +select a certain class of images to be processed or listed and allows the +processing tasks to identify calibration images and process them differently +from object images. The standard CCD image types are \fIobject\fR, +\fIzero\fR level, \fIdark\fR count, and \fIflat\fR field. For more on +the image types see \fBccdtypes\fR. + + The tasks can also identify the different filters (or other subset +parameter) which require different flat field images. This means you don't +have to separate the images by filter and process each set separately. +This feature is discussed further in \fBsubsets\fR. + + The tasks keep track of the reduction steps completed on each +image and ignore images which have been processed. This feature, +along with recognizing the image types and subsets, makes it possible to +specify all the images to a task with a wildcard template, such as +"*.imh", rather than indicating each image by name. You will find this +extremely important with large sets of observations. + + A fundamental aspect of the package is that the processing +modifies the images. In other words, the reduction operations are +performed directly on the image. This "feature" further simplifies +record keeping, frees the user from having to form unique output image +names, and minimizes the amount of disk space required. There +are two safety features in this process. First, the modifications do +not take effect until the operation is completed on the image. This +allows you to abort the task without messing up the image data and +protects data if the computer crashes. The second feature is that +there is a package parameter which may be set to make a backup of the +input data with a particular prefix such as "orig" or "imdir$". This +backup feature may be used when there is sufficient disk space, when learning +to use the package, or just to be cautious. + + In a similar effort to efficiently manage disk space, when combining +images into a master object or calibration image there is an option to +delete the input images upon completion of the combining operation. +Generally this is desirable when there are many calibration exposures, +such as zero level or flat field images, which are not used after they +are combined into a final calibration image. + + The following sections guide you through the basic use of the +\fBquad\fR package. Only the important parameters which you might +want to change are described. It is assumed that the support personnel +have created the necessary instrument files (see \fBinstruments\fR) +which will set the default parameters for the data you will be +reducing. If this is not the case you may need to delve more deeply +into the details of the tasks. Information about all the parameters +and how the various tasks operate are given in the help documentation +for the tasks and in additional special help topics. Some useful help +documentation is indicated in the discussion and also in the +\fBReferences\fR section. +.sh +2. Getting Started + + The first step is to load \fBquad\fR. This is done by loading +the \fBarcon\fR package, and then the \fBquad\fR package. Loading a +package consists of typing its name. + + When you load the \fBquad\fR package the menu of tasks or commands +is listed. This appears as follows: + +.nf + cl> quad + badpiximage combine mkillumcor qstatistics + ccdgroups cosmicrays mkillumflat quadproc + ccdhedit darkcombine mkskycor quadscale + ccdinstrument flatcombine mkskyflat setinstrument + ccdlist mkfringecor qhistogram zerocombine +.fi + +A summary of the tasks and additional help topics is obtained by typing: + + cl> help + +This list and how to get additional help on specific topics is described +in the \fBReferences\fR section at the end of this guide. + + The first command to use is \fBsetinstrument\fR, which sets the package +appropriately for the CCD images to be reduced. The support personnel +should tell you the instrument identification, but if not a list +of known instruments may be listed by using '?' for the instrument name. + +.nf + cl> setinstrument + Instrument ID (type ? for a list) \fI<enter instrument id or ?>\fR + <Set quad package parameters using eparam> + <Set quadproc task parameters using eparam> +.fi + +This task sets the default parameters and then allows you to modify the +package parameters and the processing parameters using the parameter +editor \fBeparam\fR. If you are not familiar with \fBeparam\fR see the +help or CL introduction documentation. For most terminals you move up +and down through the parameters with the terminal arrow keys, you +change the parameters by simply typing the desired value, and you exit +with control Z or control D. Note that you can change parameters for +any task at any time with \fBeparam\fR and you do not have to run +\fBsetinstrument\fR again, even if you logout, until you need to reduce +data from a different instrument. + + The \fBquad\fR package parameters control general I/O functions of +the tasks in the package. The parameters you might wish to change are +the output pixel type and the verbose option. Except when the input +images are short integers, the noise is significantly greater than one +digital unit, and disk space is critical, it is probably better to +allow the processing to convert the images to real pixel datatype. The +verbose parameter simply prints the information written to the log file +on the terminal. This can be useful when little else is being done and +you are just beginning. However, when doing background processing and +other IRAF reduction tasks it is enough to simply look at the end of +the logfile with the task \fBtail\fR to see the current state of the +processing. + + The \fBquadproc\fR parameters control the CCD processing. There are +many parameters but they all may be conveniently set at this point. +Many of the parameters have default values set appropriately for the +instrument you specified. The images to be processed can be specified +later. What needs to be set are the processing operations that you +want done and the parameters required for each operation. The +processing operations are selected by entering yes or no for each one. +The following items briefly describe each of the possible processing +operations and the additional parameters required. + +.ls \fIfixpix\fR - Fix bad CCD lines and columns? +The bad pixels (cosmetic defects) in the detector are given in a file +specified by the parameter \fIfixfile\fR. This information is used +to replace the pixels by interpolating from the neighboring pixels. +A standard file for your instrument may be set by \fBsetinstrument\fR +or if the word "image" is given then the file is defined in the instrument +data file. For more on the bad pixel file see \fBinstruments\fR. +.le +.ls \fIoverscan\fR - Apply overscan strip correction? +The overscan or prescan region is specified by the parameter +\fIbiassec\fR. This is given as an IRAF image section. For guidance on +seting this parameter see the help page for \fBquadproc\fR. The overscan +region is averaged along the readout axis, specified by the parameter +\fIreadaxis\fR, to create a one dimensional bias vector. This bias is +fit by a function to remove cosmic rays and noise. There are a number +of parameters at the end of the parameter list which control the +fitting. +.le +.ls \fItrim\fR - Trim the image? +The image is trimmed to the image section given by the parameter +\fItrimsec\fR. For guidance on setting this parameter see the help page +for \fBquadproc\fR. +.le +.ls \fIzerocor\fR - Apply zero level correction? +The zero level image to be subtracted is specified by the parameter +\fIzero\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.le +.ls \fIdarkcor\fR - Apply dark count correction? +The dark count image to be subtracted is specified by the parameter +\fIdark\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.le +.ls \fIflatcor\fR - Apply flat field correction? +The flat field images to be used are specified by the parameter +\fIflat\fR. There must be one flat field image for each filter +or subset (see \fBsubsets\fR) to be processed. If a flat field +image is not given then the calibration image will be sought +in the list of images to be processed. +.le +.ls \fIreadcor\fR - Convert zero level image to readout correction? +If a one dimensional zero level readout correction vector is to be subtracted +instead of a two dimensional zero level image then, when this parameter is set, +the zero level images will be averaged to one dimension. The readout axis +must be specified by the parameter \fIreadaxis\fR. The default for your +instrument is set by \fBsetinstrument\fR. +.le +.ls \fIscancor\fR - Convert flat field image to scan correction? +If the instrument is operated in a scan mode then a correction to the +flat field may be required. There are two types of scan modes, "shortscan" +and "longscan". In longscan mode flat field images will be averaged +to one dimension and the readout axis must be specified. Shortscan mode +is a little more complicated. The scan correction is used if the flat +field images are not observed in scan mode. The number of scan lines +must be specified by the parameter \fInscan\fR. If they are observed in +scan mode, like the object observations, then the scan correction +operations should \fInot\fR be specified. For details of scan mode operations +see \fBquadproc\fR. The scan parameters +should be set by \fBsetinstrument\fR. If in doubt consult someone +familiar with the instrument and mode of operation. +.le + + This description of the parameters is longer than the actual operation of +setting the parameters. The only parameters likely to change during processing +are the calibration image parameters. + + When processing many images using the same calibration files a modest +performance improvement can be achieved by keeping (caching) the +calibration images in memory to avoid disk accesses. This option +is available by specifying the amount of memory available for image +caching with the parameter \fImax_cache\fR. If the value is zero then +the images are accessed from disk as needed while if there is +sufficient memory the calibration images may be kept in memory during +the task execution. +.sh +3. Processing Your Data + + The processing path depends on the type of data, the type of +instrument, types of calibration images, and the observing +sequence. In this section we describe two types of operations common +in reducing most data; combining calibration images and performing the +standard calibration and correction operations. Some additional special +operations are described in the following section. + + However, the first thing you might want to try before any +processing is to get a listing of the CCD images showing the CCD image +types, subsets, and processing flags. The task for this is +\fBccdlist\fR. It has three types of output; a short one line per +image format, a longer format which shows the state of the processing, +and a format which prints the image names only (used to create files +containing lists of images of a particular CCD image type). To get a +quick listing type: + +.nf + cl> ccdlist *.imh + ccd001.imh[544,512][short][unknown][V]:FOCUS L98-193 + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s + ccd045.imh[544,512][short][flat][V]:dflat 5s + ccd066.imh[544,512][short][flat][B]:dflat 5s + ccd103.imh[544,512][short][flat][R]:dflat 5s + ccd104.imh[544,512][short][zero][]:bias + ccd105.imh[544,512][short][dark][]:dark 3600s +.fi + + The example shows only a sample of the images. The short format +listing tells you the name of the image, its size and pixel type, the +CCD image type as seen by the package, the subset identifier (in this +case the filter), and the title. If the data had been processed then +there would also be processing flags. If the CCD image types do not +seem right then there may be a problem with the instrument +specification. + + Many of the tasks in the \fBquad\fR package have the parameter +\fIccdtype\fR which selects a particular type of image. To list +only the object images from the previous example: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s +.fi + +If no CCD image type is specified (by using the null string "") +then all image types are selected. This may be +necessary if your instrument data does not contain image type identifications. +.sh +3.1 Combining Calibration Images + + If you do not need to combine calibration images because you only +have one image of each type, you can skip this section. Calibration +images, particularly zero level and flat field images, are combined in +order to minimize the effects of noise and reject cosmic ray hits in the +calibrations. The basic tool for combining images is the task +\fBcombine\fR. There are simple variants of this task whose default +parameters are set appropriately for each type of calibration image. +These are the ones you will use for calibration images leaving +\fBcombine\fR for combining object images. Zero level images are +combined with \fBzerocombine\fR, dark count images with +\fBdarkcombine\fR, and flat field images with \fBflatcombine\fR. + + For example, to combine flat field images the command is: + +.nf + cl> flatcombine *.imh + Jun 1 14:26 combine: maxreject + Images N Exp Mode Scale Offset Weight + ccd045.imh 1 5.0 INDEF 1.000 0. 0.048 + ccd046.imh 1 5.0 INDEF 1.000 0. 0.048 + <... list of files ...> + ccd065.imh 1 5.0 INDEF 1.000 0. 0.048 + ----------- ------ ------ + FlatV.imh 21 5.0 +.fi + +This output is printed when verbose mode is set. The same information +is recorded in the log file. In this case the flat fields are combined +by rejecting the maximum value at each point in the image (the +"maxreject" algorithm). The images are scaled by the exposure times, +which are all the same in this example. The mode is not evaluated for +exposure scaling and the relative weights are the same because the +exposure times are the same. The example only shows part of the +output; \fBflatcombine\fR automatically groups the flat field images by +filter to produce the calibration images "FlatV", "FlatB", and +"FlatR". +.sh +3.2 Calibrations and Corrections + + Processing the CCD data is easy and largely automated. +First, set the task parameters with the following command: + + cl> eparam quadproc + +You may have already set the parameters when you ran +\fBsetinstrument\fR, though the calibration image parameters +\fIzero\fR, \fIdark\fR, and \fIflat\fR may still need to be set or +changed. Once this is done simply give the command + +.nf + cl> quadproc *.imh + ccd003: Jun 1 15:13 Overscan section is [520:540,*] with mean=485.0 + ccd003: Jun 1 15:14 Trim data section is [3:510,3:510] + ccd003: Jun 1 15:14 Overscan section is [520:540,*] with mean=485.0 + FlatV: Jun 1 15:14 Trim data section is [3:510,3:510] + FlatV: Jun 1 15:15 Overscan section is [520:540,*] with mean=486.4 + ccd003: Jun 1 15:15 Flat field image is FlatV.imh with scale=138.2 + ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] + ccd004: Jun 1 15:16 Overscan section is [520:540,*] with mean=485.2 + ccd004: Jun 1 15:16 Flat field image is FlatV.imh with scale=138.2 + <... more ...> + ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] + ccd013: Jun 1 15:23 Overscan section is [520:540,*] with mean=482.4 + FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] + FlatB: Jun 1 15:23 Overscan section is [520:540,*] with mean=486.4 + ccd013: Jun 1 15:24 Flat field image is FlatB.imh with scale=132.3 + <... more ...> +.fi + + The output shown is with verbose mode set. It is the same as +recorded in the log file. It illustrates the principle of automatic +calibration image processing. The first object image, "ccd003", was +being processed when the flat field image was required. Since the +image was taken with the V filter the appropriate flat field was +determined to be "FlatV". Since it had not been processed, the +processing of "ccd003" was interrupted to process "FlatV". The +processed calibration image may have been cached if there was enough +memory. Once "FlatV" was processed (note that the flat field was not +flattened because the task knows this image is a flat field) the +processing of "ccd003" was completed. The next image, "ccd004", is +also a V filter image so the already processed, and possibly cached, +flat field "FlatV" is used again. The first B band image is "ccd013" +and, as before, the B filter flat field calibration image is processed +automatically. The same automatic calibration processing and image +caching occurs when using zero level and dark count calibration +images. + + Commonly the processing is done with the verbose mode turned off +and the task run as a background job. This is done with the commands + +.nf + cl> quad.verbose=no + cl> quadproc *.imh & +.fi + +The already processed images in the input list are recognized as having been +processed and are not affected. To check the status of the processing we +can look at the end of the log file with: + + cl> tail logfile + +After processing we can repeat the \fBccdlist\fR command to find: + +.nf + cl> ccdlist *.imh ccdtype=object + ccd007.imh[508,508][real][object][V][OTF]:N2968 V 600s + ccd015.imh[508,508][real][object][B][OTF]:N3098 B 500s + ccd024.imh[544,512][short][object][R][OTF]:N4036 R 600s +.fi + +The processing flags indicate the images have been overscan corrected, +trimmed, and flat fielded. + + As you can see, processing images is very easy. There is one source +of minor confusion for beginning users and that is dealing with calibration +images. First, there is no reason that calibration images +may not be processed explicitly with \fBquadproc\fR, just remember to set +the \fIccdtype\fR to the calibration image type or to "". When processing +object images the calibration images to be used may be specified either +with the task parameter for the particular calibration image or by +including the calibration image in the list of input images. Calibration +images specified by parameter value take precedence and the task +does not check its CCD image type. Calibration images given in the +input list must have a valid CCD image type. In case too many +calibration images are specified, say because the calibration images +combined to make the master calibration images were not deleted and +so are part of the image list "*.imh", only the first one will be used. +Another point to know is that flat field, iillumination, and fringe images +are subset (filter) dependent and so a calibration image for each filter +must be specified. +.sh +4. Special Processing Operations + + The special processing operations are mostly concerned with the +flat field response correction. There are also special processing +operations available in \fBquadproc\fR for one dimensional readout +corrections in the zero level and flat field calibrations. These +were described briefly above and in more detail in \fBquadproc\fR +and are not discussed further in this guide. The processing +operations described in this section are for preparing flat fields +for two dimensional spectroscopic data, for correcting flat fields +for iilluminations effects, for making a separate iillumination correction, +and for applying corrections for fringe effects. For additional +discussion about flat fields and iillumination corrections see the +help topic \fBflatfields\fR. +.sh +4.1 Spectroscopic Flat Fields + + For spectroscopic data the flat fields may have to be processed to +remove the general shape of the lamp spectrum and to replace regions outside +of the aperture where there is no flat field information with values that +will not cause bad response effects when the flat field is applied to the +data. If the shape of the lamp spectrum is not important and if the +longslit spectra have the regions outside of the slit either off the +detector or trimmed then you may use the flat field without special +processing. + + First you must process the flat field images explicitly with + + cl> quadproc *.imh ccdtype=flat + +where "*.imh" may be replaced with any list containing the flat fields. +If zero level and dark count corrections are required these calibration +images must be available at this time. + + Load the \fBtwodspec\fR package and then either the \fBlongslit\fR +package, for longslit data, or the \fBapextract\fR package, for +multiaperture data such as echelles, multifiber, or aperture mask +spectra. The task for removing the longslit quartz spectrum is +\fBresponse\fR. There is also a task for removing iillumination +effects, including the slit profile, from longslit spectra called +\fBiillumination\fR. For more about processing longslit spectra see the +help for these tasks and the paper \fIReduction of Longslit Spectra +with IRAF\fR. The cookbook \fIReduction of Longslit Spectroscopic +Data Using IRAF (KPNO ICCD and Cryogenic Camera Data)\fR also provides +a very good discussion even if your data is from a different instrument. + + For multiaperture data the task for removing the relative shapes of +the spectra is called \fBapnormalize\fR. Again, consult the help documentation +for this task for further details. Since you will probably also be +using the package for extracting the spectra you may be interested +in the document \fIThe IRAF APEXTRACT Package\fR. +.sh +4.2 Iillumination Corrections + + The flat field calibration images may not have the same iillumination +pattern as the observations of the sky due to the way the lamp illuminates the +optical system. In this case when the flat field correction is applied +to the data there will be gradients in the sky background. To remove +these gradients a blank sky calibration image is heavily smoothed +to produce an iillumination image. The iillumination image +is then divided into the images during processing to correct for the +iillumination difference between the flat field and the objects. +Like the flat fields, the iillumination corrections images may be subset +dependent so there should be an iillumination image for each subset. + +The task which makes iillumination correction images is \fBmkskycor\fR. +Some examples are + +.nf + cl> mkskycor sky004 Illum004 + cl> mkskycor sky*.imh "" +.fi + +In the first example the sky image "sky004" is used to make the iillumination +correction image "Illum004". In the second example the sky images are +converted to iillumination correction images by specifying no output image +names. Like \fBquadproc\fR if the input images have not been processed they +are first processed automatically. + +To apply the iillumination correction + +.nf + cl> quadproc *.imh ccdtype=object illumcor+ illum=Illum004 + cl> quadproc *.imh ccdtype=object illumcor+ illum=sky*.imh +.fi + +The iillumination images could also be set using \fBeparam\fR or given +on the command line. +.sh +4.3 Sky Flat Fields + + You will notice that when you process images with an iillumination +correction you are dividing each image by a flat field calibration and +an iillumination correction. If the iillumination corrections are not +done as a later step but at the same time as the rest of the processing +one will get the same calibration by multiplying the flat field by +the iillumination correction and using this product alone as the +flat field. Such an image is called a \fIsky flat\fR since it is +a flat field which has been corrected to yield a flat sky when applied +to the observations. This approach has the advantage of one less +calibration image and two less computations (scaling and dividing the +iillumination correction). As an added short cut, rather than compute +the iillumination image with \fBmkskycor\fR and then multiplying, the +task \fBmkskyflat\fR does all this in one step. Thus, \fBmkskyflat\fR +takes an input blank sky image, processes it if needed, determines the +appropriate flat field (sky flats are also subset dependent) from the +\fBquadproc\fR parameters or the input image list, and produces an +output sky flat. Further if no output image is specified the task +converts the input blank sky calibration image into a sky flat. + + Two examples in which a new image is created and in which the +input images are converted to sky flats are + +.nf + cl> mkskyflat sky004 Skyflat + cl> mkskyflat sky*.imh "" +.fi +.sh +4.4 Iillumination Corrected Flat Fields + + A third method to account for iillumination problems in the flat fields +is to remove the large scale pattern from the flat field itself. This is +useful if there are no reasonable blank sky calibration images and the +astronomical exposures are evenly illuminated but the flat fields are not. +This is done by smoothing the flat field images instead of blank sky +images. As with using the sky images there are two methods, creating +an iillumination correction to be applied as a separate step or fixing +the original flat field. The smoothing algorithm is +the same as that used in the other tasks. The tasks to make these types +of corrections are \fBmkillumcor\fR and \fBmkillumflat\fR. The usage +is pretty much the same as the other iillumination correction tasks +except that it is more reasonable to replace the original flat fields +by the corrected flat fields when fixing the flat field. Examples +of an iillumination correction and removing the iillumination pattern +from the flat field are + +.nf + cl> mkillumcor flat025 Illum025 + cl> mkillumflat flat*.imh "" +.fi + +As with the other tasks, the input images are processed if necessary. +.sh +4.5 Fringe Corrections + + Some CCD detectors suffer from fringing effects due to the night +sky emission lines which are not removed by the other calibration +and correction operations. To correct for the fringing you need a +really blank sky image. There is not yet a task to remove objects from +sky images because this is often done with an interactive image display +tool (which will soon be added). The blank sky image is heavily smoothed +to determine the mean sky background and then this is subtracted from the +original image. The image should then be essentially zero except for the +fringe pattern. This fringe correction image is scaled to the same +exposure time as the image to be corrected and then subtracted to remove +the fringing. Note that since the night sky lines are variable there +may need to be an additional scaling applied. Determining this scaling +requires either an interactive display tool or a very clever task. +Such tasks will also be added in the future. + + The task to make a fringe correction image is \fBmkfringecor\fR. +the sky background is determined in exactly the same way as the iillumination +pattern, in fact the same sky image may be used for both the sky +iillumination and for the fringe correction. The task works consistently +with the "mk" tasks in that the input images are processed first if needed +and then the output correction image is produced with the specified name +or replaces the input image if no output image is specified. +As examples, + +.nf + cl> mkfringecor sky004 Fringe + cl> mkfringecor sky*.imh "" +.fi +.sh +6. Summary + + The \fBquad\fR package is very easy to use. First load the package; +it is in the \fBarcon\fR package. If this is your first time reducing data +from a particular instrument or if you have changed instruments then run +\fBsetinstrument\fR. Set the processing parameters for the operations you want +performed. If you need to combine calibration images to form a master +calibration image use one of the combine tasks. Spectroscopic flat fields may +need to be processed first in order to remove the lamp spectrum. +Finally, just type + + cl> quadproc *.imh& +.sh +7. References + + A general guide to using IRAF is \fIA User's Introduction to the IRAF +Command Language\fR. This document may be found in the IRAF documentation +sets and is available from the National Optical Astronomy Observatories, +Central Computer Services (NOAO-CCS). + + A more detailed description of the \fBccdred\fR package, on which +\fBquad\fR is based, including a discussion of the design and some of the +algorithms see \fIThe IRAF CCD Reduction Package -- CCDRED\fR by F. Valdes. +This paper is available from NOAO-CCS and appears in the proceedings of the +Santa Cruz Summer Workshop in Astronomy and Astrophysics, \fIInstrumentation +for Ground-Based Optical Astronomy: Present and Future\fR, edited by +Lloyd B. Robinson and published by Springer-Verlag. + + The task descriptions and supplementary documentation are available +on-line through the help task by typing + + cl> help \fItopic\fR + +where \fItopic\fR is one of the following. + +.nf + SPECIAL TASKS FOR MULTI-READOUT CCD IMAGES + + quadproc - Process multi-readout CCD images + quadscale - Apply correction for amplifier dependent gain differences + qstatistics - Calculate image statistics for multi-readout CCD images + qhistogram - Make histogram of multi-readout CCD image + darkcombine - Combine and process dark count images + flatcombine - Combine and process flat field images + zerocombine - Combine and process zero level images + + + STANDARD CCDRED TASKS + + badpiximage - Create a bad pixel mask image from a bad pixel file + ccdgroups - Group CCD images into image lists + ccdhedit - CCD image header editor + ccdinstrument - Review and edit instrument translation files + ccdlist - List CCD processing information + combine - Combine CCD images + cosmicrays - Detect and replace cosmic rays + mkfringecor - Make fringe correction images from sky images + mkillumcor - Make flat field iillumination correction images + mkillumflat - Make iillumination corrected flat fields + mkskycor - Make sky iillumination correction images + mkskyflat - Make sky corrected flat field images + setinstrument - Set instrument parameters + + ADDITIONAL HELP TOPICS + + ccdgeometry - Discussion of CCD coordinate/geometry keywords + ccdtypes - Description of the CCD image types + flatfields - Discussion of CCD flat field calibrations + guide - Introductory guide to using the CCDRED package + instruments - Instrument specific data files + package - CCD image reduction package + quadreadout - Description of multi-readout CCD data + subsets - Description of CCD subsets +.fi + +Printed copies of the on-line help documentation may be made with the +command + + cl> help topic | lprint + + In addition to the package documentation for \fBquad\fR, +\fBlongslit\fR, and \fBapextract\fR there may be specific guides for +certain instruments. These specific guides, called "cookbooks", give +specific examples and parameter values for the CCD data. +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/guide.ms b/noao/imred/quadred/src/quad/doc/guide.ms new file mode 100644 index 00000000..62d87bb9 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/guide.ms @@ -0,0 +1,794 @@ +.RP +.TL +User's Guide to the CCDRED Package +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +June 1987 +Revised February 1988 +.AB +The IRAF CCD reduction package, \fBccdred\fR, provides tools +for the easy and efficient reduction of CCD images. The standard +reduction operations are replacement of bad pixels, subtraction of an +overscan or prescan bias, subtraction of a zero level image, +subtraction of a dark count image, division by a flat field calibration +image, division by an illumination correction, subtraction of a fringe +image, and trimming unwanted lines or columns. Another common +operation provided by the package is scaling and combining images with +a number of algorithms for rejecting cosmic rays. Data in the image +header is used to make the reductions largely automated and +self-documenting though the package may still be used in the absence of +this data. Also a translation mechanism is used to relate image header +parameters to those used by the package to allow data from a variety of +observatories and instruments to be processed. This guide provides a brief +description of the IRAF CCD reduction package and examples of reducing +simple CCD data. +.AE +.NH +Introduction +.LP + This guide provides a brief description of the IRAF CCD reduction +package \fBccdred\fR and examples of reducing simple CCD data. It is a +generic guide in that it is not tied to any particular type of data. +There may be more specific guides (or "cookbooks") for your data. +Detailed descriptions of the tasks and features of the package are +provided in the help documentation for the package. + + The purpose of the CCDRED package is to provide tools for the easy +and efficient reduction of CCD images. The standard reduction +operations are replacement of bad columns and lines by interpolation +from neighboring columns and lines, subtraction of a bias level +determined from overscan or prescan columns or lines, subtraction of a +zero level using a zero length exposure calibration image, subtraction +of a dark count calibration image appropriately scaled to the dark time +exposure, division by a scaled flat field calibration image, division +by an illumination image (derived from a blank sky image), subtraction +of a scaled fringe image (also derived from a blank sky image), and +trimming the image of unwanted lines or columns such as the overscan +strip. Any set of operations may be done simultaneously over a list of +images in a highly efficient manner. The reduction operations are +recorded in the image header and may also be logged on the terminal and +in a log file. + + The package also provides tools for combining multiple exposures +of object and calibration images to improve the statistical accuracy of +the observations and to remove transient bad pixels. The combining +operation scales images of different exposure times, adjusts for +variable sky background, statistically weights the images by their +signal-to-noise, and provides a number of useful algorithms for +detecting and rejecting transient bad pixels. + + Other tasks are provided for listing reduction information about +the images, deriving secondary calibration images (such as sky +corrected flat fields or illumination correction images), and easily +setting the package parameters for different instruments. + + There are several important features provided by the package to +make the reduction of CCD images convenient; particularly to minimize +record keeping. One of these is the ability to recognize the different +types of CCD images. This ability allows the user to select a certain +class of images to be processed or listed and allows the processing +tasks to identify calibration images and process them differently from +object images. The standard CCD image types are \fIobject\fR, +\fIzero\fR level, \fIdark\fR count, and \fIflat\fR field. For more on +the image types see \fBccdtypes\fR. + + The tasks can also identify the different filters (or other subset +parameter) which require different flat field images. This means you don't +have to separate the images by filter and process each set separately. +This feature is discussed further in \fBsubsets\fR. + + The tasks keep track of the reduction steps completed on each +image and ignore images which have been processed. This feature, +along with recognizing the image types and subsets, makes it possible to +specify all the images to a task with a wildcard template, such as +"*.imh", rather than indicating each image by name. You will find this +extremely important with large sets of observations. + + A fundamental aspect of the package is that the processing +modifies the images. In other words, the reduction operations are +performed directly on the image. This "feature" further simplifies +record keeping, frees the user from having to form unique output image +names, and minimizes the amount of disk space required. There +are two safety features in this process. First, the modifications do +not take effect until the operation is completed on the image. This +allows you to abort the task without messing up the image data and +protects data if the computer crashes. The second feature is that +there is a package parameter which may be set to make a backup of the +input data with a particular prefix such as "orig" or "imdir$". This +backup feature may be used when there is sufficient disk space, when learning +to use the package, or just to be cautious. + + In a similar effort to efficiently manage disk space, when combining +images into a master object or calibration image there is an option to +delete the input images upon completion of the combining operation. +Generally this is desirable when there are many calibration exposures, +such as zero level or flat field images, which are not used after they +are combined into a final calibration image. + + The following sections guide you through the basic use of the +\fBccdred\fR package. Only the important parameters which you might +want to change are described. It is assumed that the support personnel +have created the necessary instrument files (see \fBinstruments\fR) +which will set the default parameters for the data you will be +reducing. If this is not the case you may need to delve more deeply +into the details of the tasks. Information about all the parameters +and how the various tasks operate are given in the help documentation +for the tasks and in additional special help topics. Some useful help +documentation is indicated in the discussion and also in the +\fBReferences\fR section. +.NH +Getting Started +.LP + The first step is to load \fBccdred\fR. This is done by loading +the \fBnoao\fR package, followed by the image reduction package +\fBimred\fR, and finally the \fBccdred\fR package. Loading a +package consists of typing its name. Note that some of these packages may be +loaded automatically when you logon to IRAF. + + When you load the \fBccdred\fR package the menu of tasks or commands +is listed. This appears as follows: + +.nf +.KS +.ft L + cl> ccdred + badpiximage ccdtest mkfringecor setinstrument + ccdgroups combine mkillumcor zerocombine + ccdhedit cosmicrays mkillumflat + ccdlist darkcombine mkskycor + ccdproc flatcombine mkskyflat +.ft R +.KE +.fi + +A summary of the tasks and additional help topics is obtained by typing: + +.ft L + cl> help +.ft R + +This list and how to get additional help on specific topics is described +in the \fBReferences\fR section at the end of this guide. + + The first command to use is \fBsetinstrument\fR, which sets the package +appropriately for the CCD images to be reduced. The support personnel +should tell you the instrument identification, but if not a list +of known instruments may be listed by using '?' for the instrument name. + +.nf +.ft L + cl> setinstrument + Instrument ID (type ? for a list) \fI<enter instrument id or ?> + <Set ccdred package parameters using eparam> + <Set ccdproc task parameters using eparam> +.ft R +.fi + +This task sets the default parameters and then allows you to modify the +package parameters and the processing parameters using the parameter +editor \fBeparam\fR. If you are not familiar with \fBeparam\fR see the +help or CL introduction documentation. For most terminals you move up +and down through the parameters with the terminal arrow keys, you +change the parameters by simply typing the desired value, and you exit +with control Z or control D. Note that you can change parameters for +any task at any time with \fBeparam\fR and you do not have to run +\fBsetinstrument\fR again, even if you logout, until you need to reduce +data from a different instrument. + + The \fBccdred\fR package parameters control general I/O functions of +the tasks in the package. The parameters you might wish to change are +the output pixel type and the verbose option. Except when the input +images are short integers, the noise is significantly greater than one +digital unit, and disk space is critical, it is probably better to +allow the processing to convert the images to real pixel datatype. The +verbose parameter simply prints the information written to the log file +on the terminal. This can be useful when little else is being done and +you are just beginning. However, when doing background processing and +other IRAF reduction tasks it is enough to simply look at the end of +the logfile with the task \fBtail\fR to see the current state of the +processing. + + The \fBccdproc\fR parameters control the CCD processing. There are +many parameters but they all may be conveniently set at this point. +Many of the parameters have default values set appropriately for the +instrument you specified. The images to be processed can be specified +later. What needs to be set are the processing operations that you +want done and the parameters required for each operation. The +processing operations are selected by entering yes or no for each one. +The following items briefly describe each of the possible processing +operations and the additional parameters required. + +.LP +\fIfixpix\fR - Fix bad CCD lines and columns? +.IP +The bad pixels (cosmetic defects) in the detector are given in a file +specified by the parameter \fIfixfile\fR. This information is used +to replace the pixels by interpolating from the neighboring pixels. +A standard file for your instrument may be set by \fBsetinstrument\fR +or if the word "image" is given then the file is defined in the instrument +data file. For more on the bad pixel file see \fBinstruments\fR. +.LP +\fIoverscan\fR - Apply overscan strip correction? +.IP +The overscan or prescan region is specified by the parameter +\fIbiassec\fR. This is given as an IRAF image section. The overscan +region is averaged along the readout axis, specified by the parameter +\fIreadaxis\fR, to create a one dimensional bias vector. This bias is +fit by a function to remove cosmic rays and noise. There are a number +of parameters at the end of the parameter list which control the +fitting. The default overscan bias section and fitting parameters for +your instrument should be set by \fBsetinstrument\fR. If the word +"image" is given the overscan bias section is defined in the image +header or the instrument translation file. If an overscan section is +not set you can use \fBimplot\fR to determine the columns or rows for +the bias region and define an overscan image section. If you are +unsure about image sections consult with someone or read the +introductory IRAF documentation. +.LP +\fItrim\fR - Trim the image? +.IP +The image is trimmed to the image section given by the parameter +\fItrimsec\fR. A default trim section for your instrument should be +set by \fBsetinstrument\fR, however, you may override this default if +desired. If the word "image" is given the data +image section is given in the image header or the instrument +translation file. As with the overscan image section it is +straightforward to specify, but if you are unsure consult someone. +.LP +\fIzerocor\fR - Apply zero level correction? +.IP +The zero level image to be subtracted is specified by the parameter +\fIzero\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.LP +\fIdarkcor\fR - Apply dark count correction? +.IP +The dark count image to be subtracted is specified by the parameter +\fIdark\fR. If none is given then the calibration image will be sought +in the list of images to be processed. +.LP +\fIflatcor\fR - Apply flat field correction? +.IP +The flat field images to be used are specified by the parameter +\fIflat\fR. There must be one flat field image for each filter +or subset (see \fBsubsets\fR) to be processed. If a flat field +image is not given then the calibration image will be sought +in the list of images to be processed. +.LP +\fIreadcor\fR - Convert zero level image to readout correction? +.IP +If a one dimensional zero level readout correction vector is to be subtracted +instead of a two dimensional zero level image then, when this parameter is set, +the zero level images will be averaged to one dimension. The readout axis +must be specified by the parameter \fIreadaxis\fR. The default for your +instrument is set by \fBsetinstrument\fR. +.LP +\fIscancor\fR - Convert flat field image to scan correction? +.IP +If the instrument is operated in a scan mode then a correction to the +flat field may be required. There are two types of scan modes, "shortscan" +and "longscan". In longscan mode flat field images will be averaged +to one dimension and the readout axis must be specified. Shortscan mode +is a little more complicated. The scan correction is used if the flat +field images are not observed in scan mode. The number of scan lines +must be specified by the parameter \fInscan\fR. If they are observed in +scan mode, like the object observations, then the scan correction +operations should \fInot\fR be specified. For details of scan mode operations +see \fBccdproc\fR. The scan parameters +should be set by \fBsetinstrument\fR. If in doubt consult someone +familiar with the instrument and mode of operation. +.LP + + This description of the parameters is longer than the actual operation of +setting the parameters. The only parameters likely to change during processing +are the calibration image parameters. + + When processing many images using the same calibration files a modest +performance improvement can be achieved by keeping (caching) the +calibration images in memory to avoid disk accesses. This option +is available by specifying the amount of memory available for image +caching with the parameter \fImax_cache\fR. If the value is zero then +the images are accessed from disk as needed while if there is +sufficient memory the calibration images may be kept in memory during +the task execution. +.NH +Processing Your Data +.LP + The processing path depends on the type of data, the type of +instrument, types of calibration images, and the observing +sequence. In this section we describe two types of operations common +in reducing most data; combining calibration images and performing the +standard calibration and correction operations. Some additional special +operations are described in the following section. + + However, the first thing you might want to try before any +processing is to get a listing of the CCD images showing the CCD image +types, subsets, and processing flags. The task for this is +\fBccdlist\fR. It has three types of of output; a short one line per +image format, a longer format which shows the state of the processing, +and a format which prints the image names only (used to create files +containing lists of images of a particular CCD image type). To get a +quick listing type: + +.nf +.ft L + cl> ccdlist *.imh + ccd001.imh[544,512][short][unknown][V]:FOCUS L98-193 + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s + ccd045.imh[544,512][short][flat][V]:dflat 5s + ccd066.imh[544,512][short][flat][B]:dflat 5s + ccd103.imh[544,512][short][flat][R]:dflat 5s + ccd104.imh[544,512][short][zero][]:bias + ccd105.imh[544,512][short][dark][]:dark 3600s +.ft R +.fi + + The example shows only a sample of the images. The short format +listing tells you the name of the image, its size and pixel type, the +CCD image type as seen by the package, the subset identifier (in this +case the filter), and the title. If the data had been processed then +there would also be processing flags. If the CCD image types do not +seem right then there may be a problem with the instrument +specification. + + Many of the tasks in the \fBccdred\fR package have the parameter +\fIccdtype\fR which selects a particular type of image. To list +only the object images from the previous example: + +.nf +.ft L + cl> ccdlist *.imh ccdtype=object + ccd007.imh[544,512][short][object][V]:N2968 V 600s + ccd015.imh[544,512][short][object][B]:N3098 B 500s + ccd024.imh[544,512][short][object][R]:N4036 R 600s +.ft R +.fi + +If no CCD image type is specified (by using the null string "") +then all image types are selected. This may be +necessary if your instrument data does not contain image type identifications. +.NH 2 +Combining Calibration Images +.LP + If you do not need to combine calibration images because you only +have one image of each type, you can skip this section. Calibration +images, particularly zero level and flat field images, are combined in +order to minimize the effects of noise and reject bad pixels in the +calibrations. The basic tool for combining images is the task +\fBcombine\fR. There are simple variants of this task whose default +parameters are set appropriately for each type of calibration image. +These are the ones you will use for calibration images leaving +\fBcombine\fR for combining object images. Zero level images are +combined with \fBzerocombine\fR, dark count images with +\fBdarkcombine\fR, and flat field images with \fBflatcombine\fR. + + For example, to combine flat field images the command is: + +.nf +.ft L + cl> flatcombine *.imh + Jun 1 14:26 combine: maxreject + Images N Exp Mode Scale Offset Weight + ccd045.imh 1 5.0 INDEF 1.000 0. 0.048 + ccd046.imh 1 5.0 INDEF 1.000 0. 0.048 + \fI<... list of files ...>\fL + ccd065.imh 1 5.0 INDEF 1.000 0. 0.048 + ----------- ------ ------ + FlatV.imh 21 5.0 +.ft R +.fi + +This output is printed when verbose mode is set. The same information +is recorded in the log file. In this case the flat fields are combined +by rejecting the maximum value at each point in the image (the +"maxreject" algorithm). The images are scaled by the exposure times, +which are all the same in this example. The mode is not evaluated for +exposure scaling and the relative weights are the same because the +exposure times are the same. The example only shows part of the +output; \fBflatcombine\fR automatically groups the flat field images by +filter to produce the calibration images "FlatV", "FlatB", and +"FlatR". +.NH 2 +Calibrations and Corrections +.LP + Processing the CCD data is easy and largely automated. +First, set the task parameters with the following command: + +.ft L + cl> eparam ccdproc +.ft R + +You may have already set the parameters when you ran +\fBsetinstrument\fR, though the calibration image parameters +\fIzero\fR, \fIdark\fR, and \fIflat\fR may still need to be set or +changed. Once this is done simply give the command + +.nf +.ft L + cl> ccdproc *.imh + ccd003: Jun 1 15:13 Overscan section is [520:540,*] with mean=485.0 + ccd003: Jun 1 15:14 Trim data section is [3:510,3:510] + ccd003: Jun 1 15:14 Overscan section is [520:540,*] with mean=485.0 + FlatV: Jun 1 15:14 Trim data section is [3:510,3:510] + FlatV: Jun 1 15:15 Overscan section is [520:540,*] with mean=486.4 + ccd003: Jun 1 15:15 Flat field image is FlatV.imh with scale=138.2 + ccd004: Jun 1 15:16 Trim data section is [3:510,3:510] + ccd004: Jun 1 15:16 Overscan section is [520:540,*] with mean=485.2 + ccd004: Jun 1 15:16 Flat field image is FlatV.imh with scale=138.2 + \fI<... more ...>\fL + ccd013: Jun 1 15:22 Trim data section is [3:510,3:510] + ccd013: Jun 1 15:23 Overscan section is [520:540,*] with mean=482.4 + FlatB: Jun 1 15:23 Trim data section is [3:510,3:510] + FlatB: Jun 1 15:23 Overscan section is [520:540,*] with mean=486.4 + ccd013: Jun 1 15:24 Flat field image is FlatB.imh with scale=132.3 + \fI<... more ...>\fL +.ft R +.fi + + The output shown is with verbose mode set. It is the same as +recorded in the log file. It illustrates the principle of automatic +calibration image processing. The first object image, "ccd003", was +being processed when the flat field image was required. Since the +image was taken with the V filter the appropriate flat field was +determined to be "FlatV". Since it had not been processed, the +processing of "ccd003" was interrupted to process "FlatV". The +processed calibration image may have been cached if there was enough +memory. Once "FlatV" was processed (note that the flat field was not +flattened because the task knows this image is a flat field) the +processing of "ccd003" was completed. The next image, "ccd004", is +also a V filter image so the already processed, and possibly cached, +flat field "FlatV" is used again. The first B band image is "ccd013" +and, as before, the B filter flat field calibration image is processed +automatically. The same automatic calibration processing and image +caching occurs when using zero level and dark count calibration +images. + + Commonly the processing is done with the verbose mode turned off +and the task run as a background job. This is done with the commands + +.nf +.ft L + cl> ccdred.verbose=no + cl> ccdproc *.imh & +.ft R +.fi + +The already processed images in the input list are recognized as having been +processed and are not affected. To check the status of the processing we +can look at the end of the log file with: + +.ft L + cl> tail logfile +.ft R + +After processing we can repeat the \fBccdlist\fR command to find: + +.nf +.ft L + cl> ccdlist *.imh ccdtype=object + ccd007.imh[508,508][real][object][V][OTF]:N2968 V 600s + ccd015.imh[508,508][real][object][B][OTF]:N3098 B 500s + ccd024.imh[544,512][short][object][R][OTF]:N4036 R 600s +.ft R +.fi + +The processing flags indicate the images have been overscan corrected, +trimmed, and flat fielded. + + As you can see, processing images is very easy. There is one source +of minor confusion for beginning users and that is dealing with calibration +images. First, there is no reason that calibration images +may not be processed explicitly with \fBccdproc\fR, just remember to set +the \fIccdtype\fR to the calibration image type or to "". When processing +object images the calibration images to be used may be specified either +with the task parameter for the particular calibration image or by +including the calibration image in the list of input images. Calibration +images specified by parameter value take precedence and the task +does not check its CCD image type. Calibration images given in the +input list must have a valid CCD image type. In case too many +calibration images are specified, say because the calibration images +combined to make the master calibration images were not deleted and +so are part of the image list "*.imh", only the first one will be used. +Another point to know is that flat field, illumination, and fringe images +are subset (filter) dependent and so a calibration image for each filter +must be specified. +.NH +Special Processing Operations +.LP + The special processing operations are mostly concerned with the +flat field response correction. There are also special processing +operations available in \fBccdproc\fR for one dimensional readout +corrections in the zero level and flat field calibrations. These +were described briefly above and in more detail in \fBccdproc\fR +and are not discussed further in this guide. The processing +operations described in this section are for preparing flat fields +for two dimensional spectroscopic data, for correcting flat fields +for illuminations effects, for making a separate illumination correction, +and for applying corrections for fringe effects. For additional +discussion about flat fields and illumination corrections see the +help topic \fBflatfields\fR. +.NH 2 +Spectroscopic Flat Fields +.LP + For spectroscopic data the flat fields may have to be processed to +remove the general shape of the lamp spectrum and to replace regions outside +of the aperture where there is no flat field information with values that +will not cause bad response effects when the flat field is applied to the +data. If the shape of the lamp spectrum is not important and if the +longslit spectra have the regions outside of the slit either off the +detector or trimmed then you may use the flat field without special +processing. + + First you must process the flat field images explicitly with + +.ft L + cl> ccdproc *.imh ccdtype=flat +.ft R + +where "*.imh" may be replaced with any list containing the flat fields. +If zero level and dark count corrections are required these calibration +images must be available at this time. + + Load the \fBtwodspec\fR package and then either the \fBlongslit\fR +package, for longslit data, or the \fBapextract\fR package, for +multiaperture data such as echelles, multifiber, or aperture mask +spectra. The task for removing the longslit quartz spectrum is +\fBresponse\fR. There is also a task for removing illumination +effects, including the slit profile, from longslit spectra called +\fBillumination\fR. For more about processing longslit spectra see the +help for these tasks and the paper \fIReduction of Longslit Spectra +with IRAF\fR. The cookbook \fIReduction of Longslit Spectroscopic +Data Using IRAF (KPNO ICCD and Cryogenic Camera Data)\fR also provides +a very good discussion even if your data is from a different instrument. + + For multiaperture data the task for removing the relative shapes of +the spectra is called \fBapnormalize\fR. Again, consult the help documentation +for this task for further details. Since you will probably also be +using the package for extracting the spectra you may be interested +in the document \fIThe IRAF APEXTRACT Package\fR. +.NH 2 +Illumination Corrections +.LP + The flat field calibration images may not have the same illumination +pattern as the observations of the sky due to the way the lamp illuminates the +optical system. In this case when the flat field correction is applied +to the data there will be gradients in the sky background. To remove +these gradients a blank sky calibration image is heavily smoothed +to produce an illumination image. The illumination image +is then divided into the images during processing to correct for the +illumination difference between the flat field and the objects. +Like the flat fields, the illumination corrections images may be subset +dependent so there should be an illumination image for each subset. + +The task which makes illumination correction images is \fBmkskycor\fR. +Some examples are + +.nf +.ft L + cl> mkskycor sky004 Illum004 + cl> mkskycor sky*.imh "" +.ft R +.fi + +In the first example the sky image "sky004" is used to make the illumination +correction image "Illum004". In the second example the sky images are +converted to illumination correction images by specifying no output image +names. Like \fBccdproc\fR if the input images have not been processed they +are first processed automatically. + +To apply the illumination correction + +.nf +.ft L + cl> ccdproc *.imh ccdtype=object illumcor+ illum=Illum004 + cl> ccdproc *.imh ccdtype=object illumcor+ illum=sky*.imh +.ft R +.fi + +The illumination images could also be set using \fBeparam\fR or given +on the command line. +.NH 2 +Sky Flat Fields +.LP + You will notice that when you process images with an illumination +correction you are dividing each image by a flat field calibration and +an illumination correction. If the illumination corrections are not +done as a later step but at the same time as the rest of the processing +one will get the same calibration by multiplying the flat field by +the illumination correction and using this product alone as the +flat field. Such an image is called a \fIsky flat\fR since it is +a flat field which has been corrected to yield a flat sky when applied +to the observations. This approach has the advantage of one less +calibration image and two less computations (scaling and dividing the +illumination correction). As an added short cut, rather than compute +the illumination image with \fBmkskycor\fR and then multiplying, the +task \fBmkskyflat\fR does all this in one step. Thus, \fBmkskyflat\fR +takes an input blank sky image, processes it if needed, determines the +appropriate flat field (sky flats are also subset dependent) from the +\fBccdproc\fR parameters or the input image list, and produces an +output sky flat. Further if no output image is specified the task +converts the input blank sky calibration image into a sky flat. + + Two examples in which a new image is created and in which the +input images are converted to sky flats are + +.nf +.ft L + cl> mkskyflat sky004 Skyflat + cl> mkskyflat sky*.imh "" +.ft R +.fi +.NH 2 +Illumination Corrected Flat Fields +.LP + A third method to account for illumination problems in the flat fields +is to remove the large scale pattern from the flat field itself. This is +useful if there are no reasonable blank sky calibration images and the +astronomical exposures are evenly illuminated but the flat fields are not. +This is done by smoothing the flat field images instead of blank sky +images. As with using the sky images there are two methods, creating +an illumination correction to be applied as a separate step or fixing +the original flat field. The smoothing algorithm is +the same as that used in the other tasks. The tasks to make these types +of corrections are \fBmkillumcor\fR and \fBmkillumflat\fR. The usage +is pretty much the same as the other illumination correction tasks +except that it is more reasonable to replace the original flat fields +by the corrected flat fields when fixing the flat field. Examples +of an illumination correction and removing the illumination pattern +from the flat field are + +.nf +.ft L + cl> mkillumcor flat025 Illum025 + cl> mkillumflat flat*.imh "" +.ft R +.fi + +As with the other tasks, the input images are processed if necessary. +.NH 2 +Fringe Corrections +.LP + Some CCD detectors suffer from fringing effects due to the night +sky emission lines which are not removed by the other calibration +and correction operations. To correct for the fringing you need a +really blank sky image. There is not yet a task to remove objects from +sky images because this is often done with an interactive image display +tool (which will soon be added). The blank sky image is heavily smoothed +to determine the mean sky background and then this is subtracted from the +original image. The image should then be essentially zero except for the +fringe pattern. This fringe correction image is scaled to the same +exposure time as the image to be corrected and then subtracted to remove +the fringing. Note that since the night sky lines are variable there +may need to be an additional scaling applied. Determining this scaling +requires either an interactive display tool or a very clever task. +Such tasks will also be added in the future. + + The task to make a fringe correction image is \fBmkfringecor\fR. +the sky background is determined in exactly the same way as the illumination +pattern, in fact the same sky image may be used for both the sky +illumination and for the fringe correction. The task works consistently +with the "mk" tasks in that the input images are processed first if needed +and then the output correction image is produced with the specified name +or replaces the input image if no output image is specified. +As examples, + +.nf +.ft L + cl> mkfringecor sky004 Fringe + cl> mkfringecor sky*.imh "" +.ft R +.fi +.NH +Demonstration +.LP + A simple demonstration task is available. To run this demonstration +load the \fBccdtest\fR package; this is a subpackage of the main +\fBccdred\fR package. Then simply type + +.ft L + cl> demo +.ft R + +The demonstration will then create some artificial CCD data and reduce +them giving descriptive comments as it goes along. This demonstration uses +the "playback" facility of the command language and is actually substituting +it's own commands for terminal input. Initially you must type carriage return +or space after each comment ending with "...". If you wish to have the +demonstration run completely automatically at it's own speed then type 'g' +a the "..." prompt. Thereafter, it will simple pause long enough to give +you a chance to read the comments. When the demo is finished you will +need to remove the files created. However, feel free to examine the reduced +images, the log file, etc. \fINote that the demonstration changes the +setup parameters so be sure to run \fBsetinstrument\fI again and check +the setup parameters.\fR +.NH +Summary +.LP + The \fBccdred\fR package is very easy to use. First load the package; +it is in the \fBimred\fR package which is in the \fBnoao\fR package. +If this is your first time reducing data from a particular instrument +or if you have changed instruments then run \fBsetinstrument\fR. +Set the processing parameters for the operations you want performed. +If you need to combine calibration images to form a master calibration +image use one of the combine tasks. Spectroscopic flat fields may +need to be processed first in order to remove the lamp spectrum. +Finally, just type + +.ft L + cl> ccdproc *.imh& +.ft R +.SH +References +.LP + A general guide to using IRAF is \fIA User's Introduction to the IRAF +Command Language\fR. This document may be found in the IRAF documentation +sets and is available from the National Optical Astronomy Observatories, +Central Computer Services (NOAO-CCS). + + A more detailed description of the \fBccdred\fR package including +a discussion of the design and some of the algorithms see \fIThe IRAF +CCD Reduction Package -- CCDRED\fR" by F. Valdes. This paper is available +from NOAO-CCS and appears in the proceedings of the Santa Cruz Summer +Workshop in Astronomy and Astrophysics, \fIInstrumentation for Ground-Based +Optical Astronomy: Present and Future\fR, edited by Lloyd B. Robinson and +published by Springer-Verlag. + + The task descriptions and supplementary documentation are available +in printed form in the IRAF documentation sets, a special set +containing documentation for just the \fBccdred\fR package, and on-line +through the help task by typing + +.ft L + cl> help \fItopic\fR +.ft R + +where \fItopic\fR is one of the following. + +.nf +.ft L + badpiximage - Create a bad pixel mask image from a bad pixel file + ccdgroups - Group CCD images into image lists + ccdhedit - CCD image header editor + ccdlist - List CCD processing information + ccdproc - Process CCD images + ccdtest - CCD test and demonstration package + combine - Combine CCD images + cosmicrays - Detect and replace cosmic rays + darkcombine - Combine and process dark count images + flatcombine - Combine and process flat field images + mkfringecor - Make fringe correction images from sky images + mkillumcor - Make flat field illumination correction images + mkillumflat - Make illumination corrected flat fields + mkskycor - Make sky illumination correction images + mkskyflat - Make sky corrected flat field images +setinstrument - Set instrument parameters + zerocombine - Combine and process zero level images + + ADDITIONAL HELP TOPICS + + ccdred - CCD image reduction package + ccdtypes - Description of the CCD image types + flatfields - Discussion of CCD flat field calibrations + guide - Introductory guide to using the CCDRED package + instruments - Instrument specific data files + subsets - Description of CCD subsets +.ft R +.fi + +Printed copies of the on-line help documentation may be made with the +command + +.ft L + cl> help \fItopic\fL | lprint +.ft R + + In addition to the package documentation for \fBccdred\fR, +\fBlongslit\fR, and \fBapextract\fR there may be specific guides for +certain instruments. These specific guides, called "cookbooks", give +specific examples and parameter values for the CCD data. diff --git a/noao/imred/quadred/src/quad/doc/instruments.hlp b/noao/imred/quadred/src/quad/doc/instruments.hlp new file mode 100644 index 00000000..2b2c92f7 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/instruments.hlp @@ -0,0 +1,248 @@ +.help instruments Jun87 noao.imred.ccdred + +.ih +NAME +instruments -- Instrument specific data files +.ih +DESCRIPTION +The \fBccdred\fR package has been designed to accommodate many different +instruments, detectors, and observatories. This is done by having +instrument specific data files. Note that by instrument we mean a +combination of detector, instrument, application, and observatory, so +there might be several "instruments" associated with a particular CCD +detector. Creating and maintaining the instrument files is generally +the responsibility of the support staff, though the user may create or +copy and modify his/her own instrument/application specific files. The +task \fBsetinstrument\fR makes this information available to the user +and package easily. + +There are three instrument data files, all of which are optional. The +package may be used without the instrument files but much of the +convenience of the package, particularly with respect to using the CCD +image types, will be lost. The three files are an instrument image +header translation file, an initialization task which mainly sets +default task parameters, and a bad pixel file identifying the cosmic +bad pixels in the detector. These files are generally stored in a +system data directory which is a subdirectory of the logical +directory "ccddb$". Each file has a root name which identifies the +instrument. +.sh +1. Instrument Translation File +The instrument translation file translates the parameter names used by +the \fBccdred\fR pacakge into instrument specific parameters and also +supplies instrument specific default values. The package parameter +\fIccdred.instrument\fR specifies this file to the package. The task +\fBsetinstrument\fR sets this parameter, though it can be set +explicitly like any other parameter. For the standard instrument +translation file the root name is the instrument identification and the +extension is "dat" ("*.dat" files are protected from being removed in a +"stripped" system, i.e. when all nonessential files are removed). +Private instrument files may be given any name desired. + +The instrument translation proceeds as follows. When a package task needs +a parameter for an image, for example "imagetyp", it looks in the instrument +translation file. If the file is not found or none is specified then the +image header keyword that is requested has the same name. If an +instrument translation file is defined then the requested +parameter is translated to an image header keyword, provided a translation +entry is given. If no translation is given the package name is used. For +example the package parameter "imagetyp" might be translated to "data-typ" +(the old NOAO CCD keyword). If the parameter is not found then the default +value specified in the translation file, if present, is returned. For recording +parameter information in the header, such as processing flags, the +translation is also used. The default value has no meaning in this case. +For example, if the flag specifying that the image has been corrected +by a flat field is to be set then the package parameter name "flatcor" +might be translated to "ff-flag". If no translation is given then the +new image header parameter is entered as "flatcor". + +The format of the translation file are lines consisting of the package +parameter name, followed by the image header keyword, followed by the +default value. The first two fields are parameter names. The fields +are separated by whitespace (blanks and tabs). String default values +containing blanks must be quoted. An example is given below. + +.nf + exptime itime + darktime itime + imagetyp data-typ + subset f1pos + biassec biassec [411:431,2:573] + datasec datasec [14:385,2:573] + + fixpix bp-flag 0 + overscan bt-flag 0 + zerocor bi-flag 0 + darkcor dk-flag 0 + flatcor ff-flag 0 + fringcor fr-flag 0 +.fi + +The first two lines translate the CCD image type, and the subset parameter +without default values (see \fBccdtypes\fR and \fBsubsets\fR for more +information). The next two lines give the overscan bias strip +section and the data section with default values for the instrument. +Note that these parameters may be overridden in the task \fBccdproc\fR. +The blank line is ignored. + +The next set of translations requires further discussion. For processing +flags the package assumes that the absence of a keyword means that the +processing has not been done. If processing is always to be done with +the \fBCCDRED\fR package and no processing keywords are recorded in the raw data +then these parameters should be absent (unless you don't like the names +used by the package). However, for compatibility with the original NOAO +CCD images, which may be processed outside of IRAF and which use 0 as the +no processing value, the processing flags are translated and the false values +are indicated by the default values. + +In addition to the parameter name translations the translation file +contains translations between the value of the image type parameter +and the image types used by the package. These lines +consist of the image header type string as the first field (with quotes +if there are blanks) and the image type as recognized by the package. The +following example will make this clearer. + +.nf + 'OBJECT (0)' object + 'DARK (1)' dark + 'PROJECTOR FLAT (2)' flat + 'SKY FLAT (3)' other + 'COMPARISON LAMP (4)' other + 'BIAS (5)' zero + 'DOME FLAT (6)' flat +.fi + +The values of the image type strings in the header contain blanks so they +are quoted. Also the case of the strings is important. Note that there +are two types of flat field images and three types of object images. + +The CCD image types recognized by the package are: + +.nf + zero - zero level image such as a bias or preflash + dark - dark count image + flat - flat field image + illum - iillumination image such as a sky image + fringe - fringe correction image + object - object image +.fi + +There may be more than one image type that maps to the same package +type. In particular other standard CCD image types, such as comparison +spectra, multiple exposure, standard star, etc., should be mapped to +object or other. There may also be more than one type of flat field, +i.e. dome flat, sky flat, and lamp flat. For more on the CCD image +types see \fBccdtypes\fR. + +The complete set of package parameters are given below. +The package parameter names are generally the same as the +standard image header keywords being adopted by NOAO. + +.nf + General Image Header and Default Parameters + ccdmean darktime exptime fixfile + imagetyp ncombine biassec subset + title datasec + + CCDRED Processing Flags + ccdproc darkcor fixpix flatcor + fringcor illumcor overscan trim + zerocor + + CCDRED CCD Image Types + dark flat fringe illum + none object unknown zero +.fi + +The translation mechanism described here may become more +sophisticated in the future and a general IRAF system facility may be +implemented eventually. For the present the translation mechanism is +quite simple. +.sh +2. Instrument Setup Script +The task \fBsetinstrument\fR translates an instrument ID into a +CL script in the instrument directory. This script is then executed. +Generally this script simply sets the task parameters for an +instrument/application. However, it could do anything else the support +staff desires. Below are the first few lines of a typical instrument setup +script. + +.nf + ccdred.instrument = "ccddb$kpno/example.dat" + ccdred.pixeltype = "real" + ccdproc.fixpix = yes + ccdproc.overscan = yes + ccdproc.trim = yes + ccdproc.zerocor = no + ccdproc.darkcor = no + ccdproc.flatcor = yes + ccdproc.biassec = "[411:431,2:573]" + ccdproc.datasec = "[14:385,2:573]" +.fi + +The instrument parameter should always be set unless there is no +translation file for the instrument. The \fBccdproc\fR parameters +illustrate setting the appropriate processing flags for the +instrument. The overscan bias and trim data sections show an alternate +method of setting these instrument specific parameters. They may be +set in the setup script in which case they are given explicitly in the +user parameter list for \fBccdproc\fR. If the value is "image" then +the parameters may be determined either through the default value in +the instrument translation file, as illustrated in the previous +section, or from the image header itself. + +The instrument setup script for setting default task parameters may be +easily created by the support person as follows. Set the package +parameters using \fBeparam\fR or with CL statements. Setting the +parameters might involve testing. When satisfied with the way the +package is set then the parameters may be dumped to a setup script +using the task \fBdparam\fR. The final step is editing this script to +delete unimportant and query parameters. For example, + +.nf + cl> dparam ccdred >> file.cl + cl> dparam ccdproc >> file.cl + cl> dparam combine >> file.cl + ... + cl> ed file.cl +.fi +.sh +3. Instrument Bad Pixel File +The bad pixel file describes the bad pixels, columns, and lines in the +detector which are to be replaced by interpolation when processing the +images. This file is clearly detector specific. The file consists of +lines describing rectangular regions of the image. +The regions are specified by four numbers giving the starting and ending +columns followed by the starting and ending lines. The starting and +ending points may be the same to specify a single column or line. The +example below illustrates a bad pixel file. + +.nf + # RCA1 CCD untrimmed + 25 25 1 512 + 108 108 1 512 + 302 302 403 512 + 1 512 70 70 + 245 246 312 315 +.fi + +If there is a comment line in the file containing the word "untrimmed" +then the coordinates of the bad pixel regions apply to the original image. +If the image has been trimmed and the bad pixels are replaced at a later +stage then this word indicates that the trim region be determined from the +image header and the necessary coordinate conversion made. If the word +"untrimmed" does not appear then the coordinates are assumed to apply to +the image directly; i.e. the trimmed coordinates if the image has been +trimmed or the original coordinates if the image has not been trimmed. +The standard bad pixel files should always refer to the original, untrimmed +coordinates. + +The first two bad pixel regions are complete bad columns (the image +is 512 x 512), the next line is a partial bad column, the next line is +a bad line, and the last line is a small bad region. These files are +easy to create, provided you have a good image to work from and a way +to measure the positions with an image or graphics display. +.ih +SEE ALSO +ccdtypes, subsets, setinstrument +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/mkfringecor.hlp b/noao/imred/quadred/src/quad/doc/mkfringecor.hlp new file mode 100644 index 00000000..797f4d11 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/mkfringecor.hlp @@ -0,0 +1,90 @@ +.help mkfringecor Feb88 noao.imred.ccdred +.ih +NAME +mkfringecor -- Make fringe correction images from sky images +.ih +USAGE +mkfringecor input output +.ih +PARAMETERS +.ls input +List of input images for making fringe correction images. +.le +.ls output +List of output fringe correction images. If none is +specified or if the name is the same as the input image then the output +image replaces the input image. +.le +.ls ccdtype = "" +CCD image type to select from the input images. If none is specified +then all types are used. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed background. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +The input blank sky images are automatically processed up through the +iillumination correction before computing the fringe correction images. +The fringe corrections are subset dependent. +The slowly varying background is determined and subtracted leaving only +the fringe pattern caused by the sky emission lines. These fringe images +are then scaled and subtracted from the observations by \fBccdproc\fR. +The background is determined by heavily smoothing the image using a +moving "boxcar" average. The effects of the objects and fringes in the +image is minimized by using a sigma clipping algorithm to detect and +exclude them from the average. Note, however, that objects left in the +fringe image will affect the fringe corrected observations. Any objects +in the sky image should be removed using \fBskyreplace\fR (not yet +available). + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +To minimize the effects of the fringes and any objects in the blank sky +calibration images a sigma clipping algorithm is used to detect and +exclude features from the background. This is done by computing the +rms of the image lines relative to the smoothed background and +excluding points exceeding the specified threshold factors times the +rms. This is done before each image line is added to the moving +average, except for the first few lines where an iterative process is +used. +.ih +EXAMPLES +1. The two examples below make an fringe correction image from a blank +sky image, "sky017". In the first example a separate fringe +image is created and in the second the fringe image replaces the +sky image. + +.nf + cl> mkskycor sky017 Fringe + cl> mkskycor sky017 frg017 +.fi +.ih +SEE ALSO +ccdproc +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/mkillumcor.hlp b/noao/imred/quadred/src/quad/doc/mkillumcor.hlp new file mode 100644 index 00000000..0effd7a2 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/mkillumcor.hlp @@ -0,0 +1,92 @@ +.help mkillumcor Oct88 noao.imred.ccdred +.ih +NAME +mkillumcor -- Make flat field iillumination correction images +.ih +USAGE +mkillumcor input output +.ih +PARAMETERS +.ls input +List of input images for making flat field iillumination correction images. +.le +.ls output +List of output flat field iillumination correction images. If none is +specified or if the name is the same as the input image then the output +image replaces the input image. +.le +.ls ccdtype = "flat" +CCD image type to select from the input images. If none is specified +then all types are used. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude deviant points from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed iillumination. +.le +.ls divbyzero = 1. +The iillumination correction is the inverse of the smoothed flat field. +This may produce division by zero. A warning is given if division +by zero takes place and the result (the iillumination correction value) +is replaced by the value of this parameter. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +First, the input flat field images are automatically processed if +needed. Then, the large scale iillumination pattern of the images is +determined by heavily smoothing them using a moving "boxcar" average. +The iillumination correction, the inverse of the iillumination pattern, +is applied by \fBccdproc\fR to CCD images to remove the iillumination +pattern introduced by the flat field. The combination of the flat +field calibration and the iillumination correction based on the flat +field is equivalent to removing the iillumination from the flat field +(see \fBmkillumflat\fR). This two step calibration is generally used +when the observations have been previously flat field calibrated. This +task is closely related to \fBmkskycor\fR which determines the +iillumination correction from a blank sky image; this is preferable to +using the iillumination from the flat field as it corrects for the +residual iillumination error. For a general discussion of the options +for flat fields and iillumination corrections see \fBflatfields\fR. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +To minimize the effects of bad pixels a sigma clipping algorithm is +used to detect and reject these pixels from the iillumination. This is +done by computing the rms of the image lines relative to the smoothed +iillumination and excluding points exceeding the specified threshold +factors times the rms. This is done before each image line is added to +the moving average, except for the first few lines where an iterative +process is used. +.ih +EXAMPLES +1. The example below makes an iillumination correction image from the +flat field image, "flat017". + + cl> mkillumcor flat017 Illum +.ih +SEE ALSO +ccdproc, flatfields, mkillumflat, mkskycor, mkskyflat +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/mkillumflat.hlp b/noao/imred/quadred/src/quad/doc/mkillumflat.hlp new file mode 100644 index 00000000..8288fb85 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/mkillumflat.hlp @@ -0,0 +1,101 @@ +.help mkillumflat Oct88 noao.imred.ccdred +.ih +NAME +mkillumflat -- Make illumination corrected flat fields +.ih +USAGE +mkillumflat input output +.ih +PARAMETERS +.ls input +List of input flat field images to be illumination corrected. +.le +.ls output +List of output illumination corrected flat field images. +If none is specified or if the name is the same as the +input image then the output image replaces the input image. +.le +.ls ccdtype = "flat" +CCD image type to select from the input images. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed illumination. +.le +.ls divbyzero = 1. +The illumination flat field is the ratio of the flat field to a +smoothed flat field. This may produce division by zero. A warning is +given if division by zero takes place and the result (the illumination +corrected flat field value) is replaced by the value of this +parameter. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +First, the input flat field images are processed as needed. Then the +large scale illumination pattern of the images is removed. The +illumination pattern is determined by heavily smoothing the image using +a moving "boxcar" average. The output image is the ratio of the input +image to the illumination pattern. The illumination pattern is +normalized by its mean to preserve the mean level of the input image. + +When this task is applied to flat field images only the small scale +response effects are retained. This is appropriate if the flat field +images have illumination effects which differ from the astronomical +images and blank sky images are not available for creating sky +corrected flat fields. When a high quality blank sky image is +available the related task \fBmkskyflat\fR should be used. Note that +the illumination correction, whether from the flat field or a sky +image, may be applied as a separate step by using the task +\fBmkillumcor\fR or \fBmkskycor\fR and applying the illumination +correction as a separate operation in \fBccdproc\fR. However, creating +an illumination corrected flat field image before processing is more +efficient since one less operation per image processed is needed. For +more discussion about flat fields and illumination corrections see +\fBflatfields\fR. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +To minimize the effects of bad pixels a sigma clipping algorithm is +used to detect and reject these pixels from the illumination. This is +done by computing the rms of the image lines relative to the smoothed +illumination and excluding points exceeding the specified threshold +factors times the rms. This is done before each image line is added to +the moving average, except for the first few lines where an iterative +process is used. +.ih +EXAMPLES +1. Two examples in which a new image is created and in which the +input flat fields are corrected in place are: + +.nf + cl> mkllumflat flat004 FlatV + cl> mkillumflat flat* "" +.fi +.ih +SEE ALSO +ccdproc, flatfields, mkfringecor, mkillumcor, mkskycor, mkskyflat +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/mkskycor.hlp b/noao/imred/quadred/src/quad/doc/mkskycor.hlp new file mode 100644 index 00000000..15cfacf6 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/mkskycor.hlp @@ -0,0 +1,103 @@ +.help mkskycor Feb88 noao.imred.ccdred +.ih +NAME +mkskycor -- Make sky iillumination correction images +.ih +USAGE +mkskycor input output +.ih +PARAMETERS +.ls input +List of input images for making sky iillumination correction images. +.le +.ls output +List of output flat field iillumination correction images. If none is +specified or if the name is the same as the input image then the output +image replaces the input image. +.le +.ls ccdtype = "" +CCD image type to select from the input images. If none is specified +then all types are used. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed iillumination. +.le +.ls ccdproc (parameter set) +CCD processing parameters. +.le +.ih +DESCRIPTION +The large scale iillumination pattern of the input images, generally +blank sky calibration images, is determined by heavily smoothing +the image using a moving "boxcar" average. The effects of objects in +the image may be minimized by using a sigma clipping algorithm to +detect and exclude the objects from the average. This +iillumination image is applied by \fBccdproc\fR to CCD images to remove +the iillumination pattern. + +The input images are automatically processed up through flat field +calibration before computing the iillumination. The iillumination +correction is that needed to make the processed images flat +over large scales. The input images are generally blank sky calibration +images which have the same iillumination and instrumental effects +as the object observations. Object images may be used but removal +of the objects may not be very good; particularly large, bright objects. +For further discussion of flat fields and iillumination corrections +see \fBflatfields\fR. + +You will notice that when you process images with an iillumination +correction you are dividing each image by a flat field calibration and +an iillumination correction. If the iillumination corrections are not +done as a later step but at the same time as the rest of the processing +one will get the same calibration by multiplying the flat field by the +iillumination correction and using this product alone as the flat +field. This approach has the advantage of one less calibration image +and two less computations (scaling and dividing the iillumination +correction). Such an image, called a \fIsky flat\fR, may be created by +\fBmkskyflat\fR as an alternative to this task. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +Blank sky images may not be completely blank so a sigma clipping +algorithm may be used to detect and exclude objects from the +iillumination pattern. This is done by computing the rms of the image +lines relative to the smoothed background and excluding points +exceeding the specified threshold factors times the rms. This is done +before each image line is added to the moving average, except for the +first few lines where an iterative process is used. +.ih +EXAMPLES +1. The two examples below make an iillumination image from a blank sky image, +"sky017". In the first example a separate iillumination image is created +and in the second the iillumination image replaces the sky image. + +.nf + cl> mkskycor sky017 Illum + cl> mkskycor sky017 sky017 +.fi +.ih +SEE ALSO +ccdproc, flatfields, mkillumcor, mkillumflat, mkskyflat +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/mkskyflat.hlp b/noao/imred/quadred/src/quad/doc/mkskyflat.hlp new file mode 100644 index 00000000..3d9ac1ca --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/mkskyflat.hlp @@ -0,0 +1,111 @@ +.help mkskyflat Feb88 noao.imred.ccdred +.ih +NAME +mkskyflat -- Make sky corrected flat field images +.ih +USAGE +mkskyflat input output +.ih +PARAMETERS +.ls input +List of blank sky images to be used to create sky corrected flat field +calibration images. +.le +.ls output +List of output sky corrected flat field calibration images (called +sky flats). If none is specified or if the name is the same as the +input image then the output image replaces the input image. +.le +.le +.ls ccdtype = "" +CCD image type to select from the input images. +.le +.ls xboxmin = 5, xboxmax = 0.25, yboxmin = 5, yboxmax = 0.25 +Minimum and maximum smoothing box size along the x and y axes. The +minimum box size is used at the edges and grows to the maximum size in +the middle of the image. This allows the smoothed image to better +represent gradients at the edge of the image. If a size is less then 1 +then it is interpreted as a fraction of the image size. If a size is +greater than or equal to 1 then it is the box size in pixels. A size +greater than the size of image selects a box equal to the size of the +image. +.le +.ls clip = yes +Clean the input images of objects? If yes then a clipping algorithm is +used to detect and exclude objects from the smoothing. +.le +.ls lowsigma = 2.5, highsigma = 2.5 +Sigma clipping thresholds above and below the smoothed iillumination. +.le +.ls ccdproc (pset) +CCD processing parameter set. +.le +.ih +DESCRIPTION +A sky corrected flat field calibration image, called a sky flat, is a +flat field that when applied to observations of the sky have no large +scale gradients. Flat field images are generally obtained by exposures +to lamps either illuminating the telescope field or a surface in the dome +at which the telescope is pointed. Because the detector is not illuminated +in the same way as an observation of the sky there may be large +scale iillumination patterns introduced into the observations with such +a flat field. To correct this type of flat field a blank sky observation +(which has been divided by the original flat field) is heavily smoothed +to remove the noise leaving only the residual large scale iillumination +pattern. This iillumination pattern is divided into the original flat +field to remove this residual. + +The advantage of creating a sky flat field is that when processing +the observations no additional operations are required. However, +if the observations have already been processed with the original +flat field then the residual iillumination pattern of blank sky +calibration images may be created as an iillumination correction +to be applied by \fBccdproc\fR. Such a correction is created by the +task \fBmkskycor\fR. If a good blank sky image is not +available then it may be desirable to remove the iillumination pattern +of the flat field image using \fBmkillumflat\fR or \fBmkillumcor\fR +provided the sky observations are truly uniformly illuminated. +For more on flat fields and iillumination corrections see \fBflatfields\fR. + +The input, blank sky images are first processed, based on the +\fBccdproc\fR parameters, if needed. These parameters also determine +the flat field image to be used in making the sky flat. The residual +iillumination pattern is determined by heavily smoothing the image using +a moving "boxcar" average. The effects of objects in the input image +may be minimized by using a sigma clipping algorithm to detect and +exclude the objects from the average. The output image is ratio of the +flat field image, for the same subset as the input image, to the +residual iillumination pattern determined from the processed blank sky +input image. The iillumination pattern is normalized by its mean to +preserve the mean level of the flat field image. + +The smoothing algorithm is a moving average over a two dimensional +box. The algorithm is unconvential in that the box size is not fixed. +The box size is increased from the specified minimum at the edges to +the maximum in the middle of the image. This permits a better estimate +of the background at the edges, while retaining the very large scale +smoothing in the center of the image. Note that the sophisticated +tools of the \fBimages\fR package may be used for smoothing but this +requires more of the user and, for the more sophisticated smoothing +algorithms such as surface fitting, more processing time. + +Blank sky images may not be completely blank so a sigma clipping +algorithm may be used to detect and exclude objects from the +iillumination pattern. This is done by computing the rms of the image +lines relative to the smoothed background and excluding points +exceeding the specified threshold factors times the rms. This is done +before each image line is added to the moving average, except for the +first few lines where an iterative process is used. +.ih +EXAMPLES +1. Two examples in which a new image is created and in which the +input sky images are converted to sky flats are: + +.nf + cl> mkskyflat sky004 Skyflat + cl> mkskyflat sky* "" +.fi +.ih +SEE ALSO +ccdproc, flatfields, mkfringecor, mkillumcor, mkillumflat, mkskycor +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/quad.hlp b/noao/imred/quadred/src/quad/doc/quad.hlp new file mode 100644 index 00000000..7f8754ee --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/quad.hlp @@ -0,0 +1,121 @@ +.help package Sep93 arcon.quad +.ih +NAME +quad -- reduction package for CCD images obtained with Arcon + +.ih +USAGE +quad + +This package \fBmust\fR be used in place of \fBccdred\fR for the first steps +(overscan correction and trimming) of multi-readout (quad or dual) images. +Either package can be used for the subsequent stages, and for the complete +reduction of single readout images. + +.ih +PARAMETERS +.ls pixeltype = "real real" +Output pixel datatype and calculation datatype. When images are processed +or created the output pixel datatype is determined by this parameter. +The allowed types are "short" for short integer, and "real" for real +floating point. The calculation datatypes are also short and real with a +default of real if none is specified. Note that Arcon generates images of type +"ushort" (unsigned 16-bit integers). In general both the output and calculation +types should, therefore, be set to "real" to avoid truncation and wrap +around errors, although this means that the reduced images will occupy twice as +much disk space. +.le +.ls verbose = no +Print log information to the standard output? +.le +.ls logfile = "logfile" +Text log file. If no filename is specified then no log file is kept. +.le +.ls plotfile = "" +Log metacode plot file for the overscan bias vector fits. If +no filename is specified then no metacode plot file is kept. +.le +.ls backup = "" +Backup prefix for backup images. If no prefix is specified then no backup +images are kept when processing. If specified then the backup image +has the specified prefix. +.le +.ls instrument = "" +CCD instrument translation file. This is usually set with \fBsetinstrument\fR. +.le +.ls ssfile = "subsets" +Subset translation file used to define the subset identifier. See +\fBsubsets\fR for more. +.le +.ls graphics = "stdgraph" +Interactive graphics output device when fitting the overscan bias vector. +.le +.ls cursor = "" +Graphics cursor input. The default is the standard graphics cursor. +.le +.ls version = "Version 2.0 - Sept 93" +Package version. +.le +.ih +DESCRIPTION +The \fBquad\fR package contains all the basic tasks necessary for the +reduction of CCD data obtained with Arcon. With Arcon images are often readout +using four ("quad") or two ("dual") amplifiers in order to reduce readout time. +The \fBquad\fR package includes the few special tasks needed to deal with such +multi-readout data, as well as many standard tasks taken directly from the +\fBccdred\fR package. The \fBquad\fR package must be used for the first +reduction steps, overscan correction and trimming, of multi-readout images; +subsequent steps can be performed using \fBquad\fR or \fBccdred\fR. Either +package can be used for the complete reduction of conventional single readout +CCD images. + +The \fBquad\fR package also contains the tasks \fBqstatistics\fR and +\fBqhistogram\fR which can be used for examining raw multi-readout images. + +The \fBquad\fR package task itself has several parameters which are common to +many of the tasks in the package. When images are processed or new image are +created the output pixel datatype is that specified by the parameter +\fBpixeltype\fR. Note that CCD processing replaces the original image by the +processed image so the pixel type of the CCD images may change during +processing. It is unlikely that real images will be processed to short images +but the reverse is quite likely. Processing images from short to real +pixel datatypes will generally increase the amount of disk space +required (a factor of 2 on most computers). + +The tasks produce log output which may be printed on the standard +output (the terminal unless redirected) and appended to a file. The +parameter \fIverbose\fR determines whether processing information +is printed. This may be desirable initially, but when using background +jobs the verbose output should be turned off. The user may look at +the end of the log file (for example with \fBtail\fR) to determine +the status of the processing. + +The package was designed to work with data from many different observatories +and instruments. In order to accomplish this an instrument translation +file is used to define a mapping between the package parameters and +the particular image header format. The instrument translation file +is specified to the package by the parameter \fIinstrument\fR. This +parameter is generally set by the task \fBsetinstrument\fR. The other +file used is a subset file. This is generally created and maintained +by the package and the user need not do anything. For more sophisticated +users see \fBinstruments\fR and \fBsubsets\fR. + +The package has very little graphics +output. The exception is the overscan bias subtraction. The bias +vector is logged in the metacode plot file if given. The plot file +may be examined with the tasks in the \fBplot\fR package such as +\fBgkimosaic\fR. When interactively fitting the overscan vector +the graphics input and output devices must be specified. The defaults +should apply in most cases. + +Because processing replaces the input image by the processed image it +may be desired to save the original image. This may be done by +specifying a backup prefix with the parameter \fIbackup\fR. For +example, if the prefix is "orig" and the image is "ccd001", the backup +image will be "origccd001". The prefix may be a directory but if so it must +end with '/' or '$' (for logical directories) and the directory must already +exist. +.ih +SEE ALSO +instruments, setinstrument, subsets +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/quadman.hlp b/noao/imred/quadred/src/quad/doc/quadman.hlp new file mode 100644 index 00000000..55bfe10d --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/quadman.hlp @@ -0,0 +1,1330 @@ +.help quadman Sep93 Version-0.0 +.ce +\fIV-ARCON. Reduction of CCD data obtained with Arcon\fR + +.nf + 1. Introduction + 2. Getting the Data + 3. Reducing the Data + 3.1 Introduction + 3.2 Reading the Data from Tape + 3.3 Setting Up the Translation File + 3.3.1 Setinstrument for CCD Used + 3.3.2 Setting the Subsets Parameter and Preparations + 3.4 Preparing Individual Calibration Frames + 3.5 Processing the calibration frames + 3.6 Preparing a Final Flat Field + 3.7 Processing the Images + 3.8 Creating the Bad Pixel File + 3.9 Writing the Data to Tape + + + + + + + A thinly disguised version of CCDMAN + + + Lisa Wells + + Mario Hamuy + + September 30, 1993 +.fi +.bp +.ls \fI1. Introduction\fR + +CCDRED is a package in IRAF used for CCD data reduction. It is primarily used +for CCD imaging although various aspects are also used for spectral reduction. +This document is intended as a guideline to reducing direct CCD images using +IRAF. If you are reducing spectra, see Section III or IV of +this manual. If you do not have experience using IRAF we suggest that +you start by +reading "An Introduction to IRAF" which will give you a general idea of IRAF +structure as well as a few fundamental tools. If you plan to use this package +extensively, there is a demo on the computer as well which will run through +the reduction process interactively. Once you login and enter IRAF +type the following: + +.nf + cl> noao + no> imred + im> ccdred + cc> ccdtest + cc> +.fi + +The cc> prompt indicates the package has been loaded. 'ccdtest' contains several +tasks one of which is 'demo'. Now type 'demo' and follow the instructions. It +will prompt you, from time to time, to continue to the next section. + +The examples shown here are just that, examples. The user must decide +upon the reduction procedure, naming, convention, etc..., appropriate for +his/her own data and use the cookbook and examples as guidelines only. The +examples are shown with prompts, for the package containing the tasks (do not +type the prompts, of course). It is strongly recommended that you perform an +'lpar' on every task immediately before using it, unless you are familiar with +all of its parameters. This is not always shown in the examples, but is normal +practice even among seasoned IRAF users. + +IRAF uses two conventions that you should always keep in mind. +First, images consist of lines and columns (not rows and columns). Keep in mind +that the mountain Forth systems for the CCDs are zero-indexed and use rows and +lines. IRAF uses one-indexed coordinates for images (see figure 1). Second, +the "order" of a function is the number of independent coefficients for a +polynomial, or the number of intervals for a spline curve. For example, a cubic +(third-degree) polynomial is described as "order=4" (four coefficients). + +If you require personal assistance in your reductions please contact +either Mauricio Navarrete or Nelson Saavedra on Tololo (ex 422), +or Mario Hamuy(ex 210) in La Serena. +.le +.bp +.ls \fI2. Getting the Data\fR + +Many observers often get confused about the number and kind of calibration +frames that must be taken to properly reduce their data. During a whole +run you should get the following: + +1) Bias frames for each night are essential no matter what setup and chip +you are using. 20-30 of these should be taken and combined. + +2) Dome flats should be taken for each filter you are using. Preferably +this should be done every day, but you can get by with just one per run, +as long as you take a sufficient number of them, say 20-30. + +3) Twilight sky flats are necessary if you want to do photometry to better +than 2%-3%, if you are doing surface photometry of extended objects, +or if sky subtraction is critical. We suggest that everyone take sky flats +since it is a good check of your dome flat iillumination. For some CCDs it +is better to use a sky flat to flatten your objects and this may depend +upon the filters being used. +It was found at the 0.9-m and 1.5-m telescopes that +sky flats do better in flattening your +images in the U and B filters. It is therefore +suggested that you concentrate on getting many U and B sky flats (10 or more) +since you will probably process your data using them. +These will be combined in the first steps of the reduction. + +4) Darks are worth taking to check that things are working but dark correction +is not necessary for any of the CCDs now used at CTIO. The dark current +should be <10 e-/hour/pixel, if greater, then something is wrong and you +should get it fixed. + +5) Photometric standard stars should be taken when needed and as many as +necessary (>20) to properly calibrate your objects. + +You should refer to the instrument manual for more details. We +suggest that you start taking your calibration frames early in the afternoon so +that you have enough time left for supper. It is important to note on your +calibration frames, the positions of bad pixels and then avoid observing +your objects on these regions of the CCD, especially if you plan to do +photometry. At the end of the reductions, you may wish to use a bad pixel map +to correct the bad pixels. This will be discussed later (section 3.8) +in more detail. +.le +.bp +.ce +\fI3. Reducing the Data\fR +.ls \fI3.1 Introduction\fR + +A full reduction of CCD data requires the following operations (see the +flow diagram on the next page): + +.nf + 1) Combine the bias, flats and darks. + 2) Fit and subtract a readout bias given by the overscan strip. + 3) Trim the image of overscan strip and border rows and columns. + 4) Subtract the dark, if appropriate. + 5) Prepare a final flat. + 6) Divide by the flat field. + 7) Fix the bad pixels in all the images. + 8) Fringing corrections may be done at the end. +.fi + +The most general processing, described in this manual, consists of +overscan subtracting, trimming, bias subtracting, dark subtracting, +and flat fielding your data. Although dark subtraction is rarely used, +it is included in this example for generality. + +Individual bias, darks, dome and sky flats must be properly combined to give +good signal to noise calibration frames and to remove cosmic rays. The +algorithm used to combine the images must have 10 or more frames to do a good +job of cleaning the cosmic rays. IRAF offers several algorithms for combining +the individual frames. You should always carefully inspect all the individual +frames, and the final image to check +for potential problems. + +Having obtained the combined calibration images you should flatten your sky flat +using the dome flat and examine the result for any significant iillumination +variation. If these variations are significant they may be fit in order to +correct your dome flat. +Fringing corrections may be +applied. This should be done separately. This is only needed for the RCA +chips at the 4 meter PF/CCD and the TI chips in the I band. We do not +currently support this but direction can be given as to what might possibly +work. + +At this level +the images should be finished except for fixing any bad pixels that may not +have been taken care of in the flat fielding. Once this is done you may +do photometry. A photometry manual is available to do your analysis, +see section VI of this manual, which describe the use of the +aperture photometry routines in IRAF and the transformation from +instrumental to standard magnitudes. There is also a manual which was +written by the IRAF group in Tucson which is a good guide; "A User's +Guide to Stellar CCD Photometry with IRAF". +.bp +.nf + + ========================== + = Set Instrument = + = and Translation File = + ========================== + ***** + *** + * + ================================== + = Combining Calibration Frames = + = and Removing Cosmic Rays = + ================================== + ***** + *** + * + =============================== + = Processing = + = Calibration Frames = + =============================== + ***** + *** + * + ================================ + = Preparing a Flat Field = + ================================ + ***** + *** + * + ========================== + = Processing the data = + ========================== + ***** + *** + * + ========================== + = Fixing bad pixels = + ========================== + ***** + *** + * + ========================== + = Fringe Corrections = + = (optional) = + ========================== +.fi +.le +.bp +.ls \fI3.2 Reading the data from Tape\fR + +Load the 'dataio' package and allocate the appropriate tape drive: + +.nf + cl> dataio + da> alloc mta +.fi + +If you wish to read fits files from an exabyte, use 'mtx' as the device +designation. Now mount your tape on the tape drive and be sure you've +removed the write ring. It is best to create a separate directory for +each night. This is done using the command 'mkdir night1'. Now change +to this directory by typing 'cd night1'. Read the data using the task +'rfits'. You must specify the tape drive name, the list of files you wish +to read and the "root" file name. If you transferred your data to the +SUN using 'getpix' then you may wish to use the naming convention given +to your files, in this case just set the parameter 'oldirafname' to yes. +In choosing the root file name, it is usually a good idea to include a +digit in the name to indicate the tape number (eg, "tape1" for the +first tape; files would be called "tape10001, tape10002,.."); alternatively, +an offset may be added (eg, offset=89 means the first files would be +called "tape10090, tape10091,.." or 1+89, 2+89,..). + +.nf + da> epar rfits (check the parameter list) + da> rfits mta 1-999 tape1 +.fi + +The file list "1-999" should more than cover the number of files on tape; +the task will end gracefully at the end of the tape. When finished, +rewind the tape and deallocate the drive, + +.nf + da> rew mta + da> dealloc mta +.fi + +and remove your tape from the drive. We assume that you kept the old IRAF +name given to your files throughout the rest of this manual. + +.nf + \fIrfits\fR + + fits_file = "mta" FITS data source + file_list = "1-999" File list + iraf_file = "tape1" IRAF filename + (make_image = yes) Create an IRAF image? + (long_header = no) Print FITS header cards? +(short_header = yes) Print short header? + (datatype = "") IRAF data type + (blank = 0.) Blank value + (scale = yes) Scale the data? + (oldirafname = no) Use old IRAF name in place of iraf_file? + (offset = 0) Tape file offset + (mode = "ql") +.fi +.le +.bp +.ce +\fI3.3 Setting Up the Translation File\fR +.ls \fI3.3.1 Setinstrument for CCD Used\fR + +Start by loading the 'imred' and 'ccdred' packages. + +.nf + cl> noao + no> imred + im> ccdred + cc> +.fi + +Because the 'ccdred' package was +designed to work with data from many different observatories and instruments, +an \fIinstrument translation file\fR is required to define a mapping +between the package parameters and the particular image header format. +An example of a translation file is shown on the next page. +You must define a translation file using the task 'setinstrument'. +Edit the parameters for the task 'setinstrument' according to the +list given below and run it. +The choices for instrument can be found in Appendix A. If the CCD you +used is not listed, use the generic "ccd" or "cfccd" in the instrument +parameter in the task. + +.nf + cc> epar setinstrument (check the parameter list) + cc> setinstrument +.fi + +The task will run and prompt you for the instrument ID (a "?" will list +the possible choices). Answer 'cfccd' and the task will send you into +the parameter editing mode for +the task 'ccdred'. It will automatically set the \fIinstrument\fR +(translation) parameter +so do not change it. It is a good idea while processing your calibration +frames to save a copy of the unprocessed images. This will be done +by the package 'ccdred' if you specify a prefix or subdirectory in the +parameter 'backup'. In our example, the files will be saved with the prefix +'B'. When you type 'ctrl-z' to exit you will then be sent to 'ccdproc' +to edit its parameters. We will edit +'ccdproc' later so don't worry about it for now. + +.nf + \fIsetinstrument parameters\fR + + \fIinstrument\fR = "cfccd" Instrument ID (type ? for a list) + (\fIsite\fR = "ctio") Site ID + (\fIdirectory\fR = "ccddb$") Instrument directory + (review = yes) Review instrument parameters? + query = "" Instrument ID (type q to quit) + (mode = "ql") + + \fIccdred parameters\fR + + (pixeltype = "real real") Output and calculation pixel datatypes + (verbose = no) Print log information to the standard output? + (logfile = "ccdlog") Text log file + (plotfile = "ccdplot") Log metacode plot file + (\fIbackup\fR = "B") Backup directory or prefix + (\fIinstrument\fR = "ccddb$ctio/cfccd.dat") CCD instrument file + (\fIssfile\fR = "subsets") Subset translation file + (graphics = "stdgraph") Interactive graphics output device + (cursor = "") Graphics cursor input + (version = "2: October 1987") + (mode = "ql") + ($nargs = 0) + +.fi + +An example of the \fIinstrument translation file\fR follows. + +.nf + + exptime exptime + darktime darktime + imagetyp imagetyp + subset filters + biassec biassec + datasec datasec + trimsec trimsec + fixfile fixfile + + fixpix bp-flag 0 + overscan bt-flag 0 + zerocor bi-flag 0 + darkcor dk-flag 0 + flatcor ff-flag 0 + fringcor fr-flag 0 + + OBJECT object + DARK dark + "PROJECTOR FLAT" flat + "SKY FLAT" other + COMPARISON other + BIAS zero + "DOME FLAT" flat + MASK other +.fi + + +The instrument file consists of first, the IRAF parameter and its associated +image header keyword from the objects. These header values may vary depending +upon the instrument being used. +The second section is the +header flag section which is added to the object as it is being processed to +mark that the operation has been done. Once an image is zero corrected, the +keyword bi-flag parameter is added to the image header and set to '0' so that +the process is not repeated later. The last section is the image-type +specification. This is used when the imagetyp parameter is read. The header +value 'OBJECT' is translated to 'object', and a 'SKY FLAT' becomes 'other'. +If you need to modify the translation file you may copy this file to your +directory and change the appropriate parameters. If the name of the imagetype +contains 2 words, then you must put quotes around the designation, for example, +look at the 'SKY FLAT' line in the above file. Comparison arcs are not used +in direct imaging and are not needed. + +Without the proper translation file, the task 'ccdlist' will give the following: + +.nf + cc> ccdlist *.imh + + bias001.imh[400,420][short][unknown][]:median of 4 ave of 6 bias + flat002.imh[400,420][short][unknown][]:median of 4 ave of 6 flat + flat003.imh[400,420][short][unknown][]:median of 4 ave of 6 flat + obj004.imh[400,420][short][unknown][]:IC 481 V-filter ........ + obj005.imh[400,420][short][unknown][]:IC 481 B-filter......... + : : : : : : : : + obj036.imh[400,420][short][unknown][]:Sky flat B-filter ....... + obj037.imh[400,420][short][unknown][]:Sky flat V-filter ....... + : : : : : : : : +.fi + +The [unknown] in the listing means that the frame type is unknown, ie, if +it is a dark, flat, bias,... etc. The blank brackets should contain the +filter type and will be taken care of in the next section. +.le +.bp +.ls \fI3.3.2 Setting the Subsets Parameter and Preparing for Processing\fR + +The 'ccdred' package groups observations into subsets. The image header +parameter used to identify the subsets is defined in the \fIinstrument +translation file\fR. For example, to select subsets by the header parameter +'filters' the instrument translation file would contain the line + +.nf + subset filters +.fi + +Now do a 'ccdlist' and you should see the +correct image type specified for each frame: + +.nf + cc> ccdlist *.imh + + bias001.imh[400,420][short][zero][]:median of 4 ave of 6 bias + flat002.imh[400,420][short][flat][]:median of 4 ave of 6 flat + flat003.imh[400,420][short][flat][]:median of 4 ave of 6 flat + obj004.imh[400,420][short][object][]:IC 481 V-filter ...... + obj005.imh[400,420][short][object][]:IC 481 B-filter....... + : : : : : : : + : : : : : : : + obj036.imh[400,420][short][object][]:Sky flat B-filter ..... + obj037.imh[400,420][short][object][]:Sky flat V-filter ..... + : : : : : : : +.fi + +Doing this will create a file in your directory called 'subsets'. This is a +file with a listing of the filter type read from the image header consisting +of the combination of the upper and lower filter bolt positions. The flats +were taken with a color balance filter so they have different values from the +objects in the list. Therefore, you need to edit this file to set the filter +types properly. The bias and dark were probably taken with a setting that +corresponds to one of the filters, but it will be ignored. If however, the +filter type set in the header corresponds to something other than a standard +filter, set this to some arbitrary value, for example below it is assumed that +the filter positions for the bias were '0 1', so this is set to '0'. + +.nf + cc> edit subsets +.fi + +An example of the necessary changes is: + +.nf + \fISubsets Before | Subsets After\fR + '1 0' 1 | '1 0' U + '2 0' 2 | '2 0' B + '3 0' 3 | '3 0' V + '4 0' 4 | '4 0' R + '5 0' 5 | '5 0' I + '0 1' 01 | '0 1' 0 + '1 1' 11 | '1 1' U + '2 1' 21 | '2 1' B + '3 1' 31 | '3 1' V + '4 1' 41 | '4 1' R + '5 1' 51 | '5 1' I +.fi + +If any of the parameters are not specified properly after doing 'ccdlist', +then the translation file may be set up improperly. Consult +Mauricio Navarrete, Nelson Saavedra, or Mario Hamuy. + +You must also specify the overscan and trim region. +The overscan is the region to the far right of the image when plotting a line +of the image (see figure 2a). The overscan region should be +specified by the beginning and ending column followed by the beginning and +ending line. This region should begin at least 5-10 pixels from the edge of +the active region of the CCD. The correct position can be found by plotting +several lines using ':l #1 #2' (#1 and #2 specify a range of lines to be +averaged and plotted) in 'implot', and expanding the right side (see figure 2b). +Notice the signal decays toward the right just the other side of the step with +an e-folding distance of a few pixels. The overscan strip should begin in 3 to +4 e-folding distances from the break. You want to fit this region all along +the image so the range of lines should include everything. Be sure to write +down these values so you don't forget them. + +The trim region is the section of the image to be kept after processing. +Choose this area excluding the overscan region +and any bad lines or columns at the edges of the image. Figures 2c and 2d show +a plot of a line and column, respectively. They show the edges which should be +trimmed from the image. Later (section 3.5) +you will edit the parameters for 'ccdproc' and enter the overscan and trim +section. + +The bad pixel file will be used after the images have been flattened. In many +cases the flatfielding will correct the bad pixel regions also so it is best +left until the end. If you specify the bad pixel file during the processing +be aware that the corrections are done on the file first. +.le +.bp +.ls \fI3.4 Preparing Individual Calibration Frames\fR + +If you have taken many calibration frames (bias, dome flats, darks +and sky flats) that you wish to combine in order to remove cosmic +rays, we suggest you to perform a combination of images separately for +each night. In some cases, when the instrument response remains +stable during your run, it is worth combining data from many nights, +especially the darks and sky flats, to improve the signal-to-noise +ratio. If you have already combined the data on the mountain skip +over this section and go on to the next one. + +Most likely you used "getpix" to transfer your data from the LSI +to the SUN so you have preserved the naming convention. This makes +it very easy to combine your images. + +There are tasks which combine each type of object together. For the +bias do an 'epar' on the 'zerocombine' task, and fill the parameters +according to the list given below. We suggest selecting the 'avsigclip' +rejection operation which applies a +sigma clipping algorithm to each pixel to remove cosmic rays. This +algorithm requires at least three input images (best with +more than 10) to work effectively. According to your particular needs you may +select other options to combine the data like minmax, ccdclip, etc. +No scaling is performed on the biases. +If your bias frames vary greatly, then there +is something wrong with your setup. In this case, have someone check +out the electronics. + +.nf + cc> epar zerocombine (check the list given below) +.fi + +Then combine the bias frames. Optionally you may add a '&' at the end +of the command line to submit the task as a background job. + +.nf + cc> zerocombine bias*.imh output=zero1 & +.fi + +Now, proceed with the combination of the darks. + +.nf + cc> epar darkcombine (check the list given below) + cc> darkcombine dark*.imh output=dark1 & +.fi + +Before combining the flat frames, you must +find a region free of bad pixels to be used to scale the individual +flat frames. The box should be chosen where there is signal. +Once you have determined the position of the box, you +should first run 'imstat' for all your images using this region. + +.nf + cc> imstat flat*[200:250,300:330] + + # IMAGE NPIX MEAN STDDEV MIN MAX + flat001.imh[200:250,300:330] 37901 5489. 1510. 678. 9020. + flat002.imh[200:250,300:330] 37901 5485. 1508. 694. 8484. + flat003.imh[200:250,300:330] 37901 5478. 1507. 691. 8472. + flat004.imh[200:250,300:330] 37901 5475. 1506. 671. 8397. + flat005.imh[200:250,300:330] 37901 5474. 1506. 659. 8540. + flat006.imh[200:250,300:330] 37901 5472. 1505. 663. 8422. + flat007.imh[200:250,300:330] 37901 5467. 1504. 655. 15513. + flat008.imh[200:250,300:330] 37901 5464. 1502. 673. 8471. + flat009.imh[200:250,300:330] 37901 5458. 1501. 684. 8503. +.fi + +If the mean varies by a large amount, say greater than a few percent, then +you should use the mode section for the scaling. +In the example below the flats are combined with scaling. +Enter this region in the 'statsec' parameter in the following format, + +.nf + [x1:x2,y1:y2] +.fi + +where the values x1 and x2 are the limits of the box in the x-axis, y1 +and y2 are the limits along the y-axis. This will only be +used if 'median' is specified in the 'scale' parameter of the task +'flatcombine'. In this example we have chosen to combine the individual +flats using the 'ccdclip' option, which rejects pixels using the CCD noise +parameters, namely the read-out-noise and the gain. You must enter these +values in the parameters 'rdnoise' and 'gain' in 'flatcombine'. In this +example we have entered a read-out-noise of 5 electrons, and a gain of +2.5 electrons/ADU. + +.nf + cc> epar flatcombine (check the list given below) +.fi + +Proceed with the combination of the flats. + +.nf + cc> flatcombine flat.*imh output=flat1 & +.fi + +If you have 3 or more sky flats, they may be combined in the same manner +as the dome flats (change the 'ccdtype' parameter to 'other'). + +.nf + cc> epar flatcombine (check the list given below) + cc> flatcombine sky.*imh output=sky1 & +.fi + +The output at this point is a set of images clean of cosmic rays called zero1, +dark1, flat1, and sky1, where the 1 stands for the first night and the flats +and skys end in their respective filter type ie, flat1U, flat1B,... Using the +parameter subsets, the combination occurred for all the flats and skys of the +same filter type (subset). We suggest that you examine these images, either by +displaying, or imploting them. For example; + +.nf + cc> display zero1 + cc> implot zero1 +.fi + +We suggest also to look at the logfile created by the package 'ccdred' in +order to check that the images were properly combined by the same filter +type. + +.nf + cc> page ccdlog +.fi +.bp + +.nf + \fIzerocombine\fR + + input = "bias*.imh" List of zero level images to combine + (output = "zero1") Output zero level name + (\fIcombine\fR = "average") Type of combine operation + (\fIreject\fR = "avsigclip") Type of rejection + (\fIccdtype\fR = "zero") CCD image type to combine + (process = no) Process images before combining? + (delete = no) Delete input images after combining? + (clobber = no) Clobber existing output image? + (\fIscale\fR = "none") Image scaling + (statsec = "") Image section for computing statistics + (nlow = 0) minmax: Number of low pixels to reject + (nhigh = 1) minmax: Number of high pixels to reject + (nkeep = 1) Minimum to keep (pos) or maximum to reject + (mclip = yes) Use median in sigma clipping algorithms? + (lsigma = 3.) Lower sigma clipping factor + (hsigma = 3.) Upper sigma clipping factor + (rdnoise = "0") ccdclip: CCD readout noise (electrons) + (gain = "1") ccdclip: CCD gain (electrons/DN) + (snoise = "0.") ccdclip: Sensitivity noise (fraction) + (pclip = -0.5) pclip: Percentile clipping parameter + (blank = 0.) Value if there are no pixels + (mode = "ql") + + \fIdarkcombine\fR + + input = "dark.*imh" List of dark images to combine + (output = "dark1") Output flat field root name + (\fIcombine\fR = "average") Type of combine operation + (\fIreject\fR = "avsigclip") Type of rejection + (\fIccdtype\fR = "dark") CCD image type to combine + (process = no) Process images before combining? + (delete = no) Delete input images after combining? + (clobber = no) Clobber existing output image? + (\fIscale\fR = "exposure") Image scaling + (statsec = "") Image section for computing statistics + (nlow = 1) minmax: Number of low pixels to reject + (nhigh = 1) minmax: Number of high pixels to reject + (nkeep = 1) Minimum to keep (pos) or maximum to reject + (mclip = yes) Use median in sigma clipping algorithms? + (lsigma = 3.) Lower sigma clipping factor + (hsigma = 3.) Upper sigma clipping factor + (rdnoise = "0.") ccdclip: CCD readout noise (electrons) + (gain = "1.") ccdclip: CCD gain (electrons/DN) + (snoise = "0.") ccdclip: Sensitivity noise (fraction) + (pclip = -0.5) pclip: Percentile clipping parameter + (blank = 0.) Value if there are no pixels + (mode = "ql") + + \fIflatcombine for dome flats\fR + + input = "flat.*imh" List of flat field images to combine + (output = "flat1") Output flat field root name + (combine = "median") Type of combine operation + (reject = "ccdclip") Type of rejection + (ccdtype = "flat") CCD image type to combine + (process = no) Process images before combining? + (subsets = yes) Combine images by subset parameter? + (delete = no) Delete input images after combining? + (clobber = no) Clobber existing output image? + (scale = "median") Image scaling + (statsec = "[m:n,p:q]") Image section for computing statistics + (nlow = 1) minmax: Number of low pixels to reject + (nhigh = 1) minmax: Number of high pixels to reject + (nkeep = 1) Minimum to keep (pos) or maximum to reject + (mclip = yes) Use median in sigma clipping algorithms? + (lsigma = 3.) Lower sigma clipping factor + (hsigma = 3.) Upper sigma clipping factor + (rdnoise = "5") ccdclip: CCD readout noise (electrons) + (gain = "2.5") ccdclip: CCD gain (electrons/DN) + (snoise = "0.") ccdclip: Sensitivity noise (fraction) + (pclip = -0.5) pclip: Percentile clipping parameter + (blank = 1.) Value if there are no pixels + (mode = "ql") + + \fIflatcombine for sky flats\fR + + input = "sky.*imh" List of flat field images to combine + (output = "sky1") Output flat field root name + (combine = "median") Type of combine operation + (reject = "ccdclip") Type of rejection + (ccdtype = "other") CCD image type to combine + (process = no) Process images before combining? + (subsets = yes) Combine images by subset parameter? + (delete = no) Delete input images after combining? + (clobber = no) Clobber existing output image? + (scale = "median") Image scaling + (statsec = "[m:n,p:q]") Image section for computing statistics + (nlow = 1) minmax: Number of low pixels to reject + (nhigh = 1) minmax: Number of high pixels to reject + (nkeep = 1) Minimum to keep (pos) or maximum to reject + (mclip = yes) Use median in sigma clipping algorithms? + (lsigma = 3.) Lower sigma clipping factor + (hsigma = 3.) Upper sigma clipping factor + (rdnoise = "5") ccdclip: CCD readout noise (electrons) + (gain = "2.5") ccdclip: CCD gain (electrons/DN) + (snoise = "0.") ccdclip: Sensitivity noise (fraction) + (pclip = -0.5) pclip: Percentile clipping parameter + (blank = 1.) Value if there are no pixels + (mode = "ql") +.fi +.le +.bp +.ls \fI3.5 Processing the calibration frames\fR + +Although all the following steps may be done only in one step, our +approach is to do them separately to allow you to start at any point in +this section in case you have already started reducing the images on +the mountain. + +You must start by overscan subtracting and trimming 'zero1'. Edit the +parameters for the 'ccdproc' task according to the list given below. +There is a series of parameters that are set to 'yes' and 'no'. You must set +only 'overscan' and 'trim' to 'yes' and the rest to 'no'. Also +'ccdtype' must be +set to 'zero'. Be sure to properly specify the parameters +'biassec' and 'trimsec' which you determined while reading section 3.3.2. +The range of columns comes first separated by a ':' and the range of lines +is again separated by a ':', i.e., [29:425,21:402]. + +\fIDo not change these parameters until having +processed all your images\fR. + +.nf + cc> epar ccdproc (check the list given below) + cc> ccdproc zero1 +.fi + +If the parameter 'interactive' was set to 'yes' you will be required to +fit interactively the overscan region with a certain function. Once +presented by the task with the plot of the overscan region (see figure 3) +you may change the fitting function, for example, with ':function chebyshev' +and its order with +':order 4'. To try a new fit type 'f'. We suggest a spline3 of order 2. +You may also select the sample to be fit using the 's' command twice. +If you are happy with +the fit type 'q' to quit. The task will then process 'zero1' accordingly. + +Continue by overscan subtracting, trimming and bias subtracting 'dark1'. +In this case you have to change in 'ccdproc' the 'ccdtype' parameter +to 'dark' and the +'zerocor' parameter to 'yes'. + +.nf + cc> epar ccdproc (check the list given below) + cc> ccdproc dark1 +.fi + +The dark image must be examined before proceeding. \fIFor instance, if the +dark level is low enough (<10 counts/hour/pixel) +compared to the flats, you will +probably disregard dark subtracting your images, to avoid +introducing noise in your data.\fR +However, if you notice some structure in the dark image, it would be +worth trying to dark correct the data. In the following examples, for +generality's sake, we consider dark subtraction as part of the overall +processing. If this is not your case, do not forget to set the 'darkcor' +parameter in 'ccdproc' to 'no' and leave it alone. + +Then process the flats. Check that the 'ccdtype' parameter is set +to 'flat', that 'overscan', 'trim', 'zerocor' and 'darkcor' are all set +to 'yes' and execute the task. + +.nf + cc> epar ccdproc (check the list given below) + cc> ccdproc flat1*imh +.fi + +IRAF records all the reduction operations in the image headers. You +may check from the headers of 'zero1', 'dark1', 'sky1' and 'flat1' that +BT-FLAG, BI-FLAG, DK-FLAG are properly set. For instance, + +.nf + cc> imheader flat1R l+ +.fi + +It is also possible to check this by using the 'ccdlist' task which will check +the header automatically and list the operations which have been performed on +the image. + +.nf + cc> ccdlist flat1R + +.fi +You should get the following, + +.nf + flat1R[496,501][real][flat][R][OTZ]:dflat R + +.fi +'ccdlist' should tell you the image name (flat1R), the image size [496,501] +left after trimming, the pixel type [real] of the image (the processed +images should normally have pixels in real format, as opposed to +the raw images which generally have pixels in 'short' format), +the image type [flat], the filter used in the observation +[R], the level of processing of the image [OTZ], and the image title (dflat R). + +At this level you should have the flats processed up through [OTZ] which +means that the image has been [O]verscan subtracted, [T]rimmed, and +[Z]ero subtracted. Additional codes for other operations in 'ccdproc' +are [F]lat correction, [B]ad pixel correction, [I]illumination correction. + +.nf + \fIccdproc for zero1\fR + + images = "zero1" List of CCD images to correct + (\fIccdtype\fR = "zero") CCD image type to correct + (max_cache = 0) Maximum image caching memory (in Mbytes) + (noproc = no) List processing steps only? + (fixpix = no) Fix bad CCD lines and columns? + (\fIoverscan\fR = yes) Apply overscan strip correction? + (\fItrim\fR = yes) Trim the image? + (\fIzerocor\fR = no) Apply zero level correction? + (\fIdarkcor\fR = no) Apply dark count correction? + (\fIflatcor\fR = no) Apply flat field correction? + (illumcor = no) Apply iillumination correction? + (fringecor = no) Apply fringe correction? + (readcor = no) Convert zero level image to readout cor? + (scancor = no) Convert flat field image to scan correction? + (readaxis = "line") Read out axis (column|line) + (fixfile = "") File describing the bad lines and columns + (\fIbiassec\fR = "[m:n,*]") Overscan strip image section + (\fItrimsec\fR = "[r:s,t:u]") Trim data section + (\fIzero\fR = "zero1") Zero level calibration image + (\fIdark\fR = "dark1") Dark count calibration image + (\fIflat\fR = "") Flat field images + (illum = "") Iillumination correction images + (fringe = "") Fringe correction images + (minreplace = 1.) Minimum flat field value + (scantype = "shortscan") Scan type (shortscan|longscan) + (nscan = 1) Number of short scan lines\n + (interactive = yes) Fit overscan interactively? + (function = "spline3") Fitting function + (order = 2) Number of polynomial terms or spline pieces + (sample = "*") Sample points to fit + (naverage = 1) Number of sample points to combine + (niterate = 1) Number of rejection iterations + (low_reject = 3.) Low sigma rejection factor + (high_reject = 3.) High sigma rejection factor + (grow = 0.) Rejection growing radius + (mode = "ql") + + \fIccdproc for dark1\fR + + images = "dark1" List of CCD images to correct + (\fIccdtype\fR = "dark") CCD image type to correct + (max_cache = 0) Maximum image caching memory (in Mbytes) + (noproc = no) List processing steps only? + (fixpix = no) Fix bad CCD lines and columns? + (\fIoverscan\fR = yes) Apply overscan strip correction? + (\fItrim\fR = yes) Trim the image? + (\fIzerocor\fR = yes) Apply zero level correction? + (\fIdarkcor\fR = no) Apply dark count correction? + (\fIflatcor\fR = no) Apply flat field correction? + (illumcor = no) Apply iillumination correction? + (fringecor = no) Apply fringe correction? + (readcor = no) Convert zero level image to readout corr? + (scancor = no) Convert flat field image to scan correction? + (readaxis = "line") Read out axis (column|line) + (fixfile = "") File describing the bad lines and columns + (\fIbiassec\fR = "[m:n,*]") Overscan strip image section + (\fItrimsec\fR = "[r:s,t:u]") Trim data section + (\fIzero\fR = "zero1") Zero level calibration image + (\fIdark\fR = "dark1") Dark count calibration image + (\fIflat\fR = "") Flat field images + (illum = "") Iillumination correction images + (fringe = "") Fringe correction images + (minreplace = 1.) Minimum flat field value + (scantype = "shortscan") Scan type (shortscan|longscan) + (nscan = 1) Number of short scan lines\n + (interactive = yes) Fit overscan interactively? + (function = "spline3") Fitting function + (order = 2) Number of polynomial terms or spline pieces + (sample = "*") Sample points to fit + (naverage = 1) Number of sample points to combine + (niterate = 1) Number of rejection iterations + (low_reject = 3.) Low sigma rejection factor + (high_reject = 3.) High sigma rejection factor + (grow = 0.) Rejection growing radius + (mode = "ql") + + \fIccdproc for flat1\fR + + images = "flat1*imh" List of CCD images to correct + (\fIccdtype\fR = "flat") CCD image type to correct + (max_cache = 0) Maximum image caching memory (in Mbytes) + (noproc = no) List processing steps only? + (fixpix = no) Fix bad CCD lines and columns? + (\fIoverscan\fR = yes) Apply overscan strip correction? + (\fItrim\fR = yes) Trim the image? + (\fIzerocor\fR = yes) Apply zero level correction? + (\fIdarkcor\fR = yes) Apply dark count correction? + (\fIflatcor\fR = no) Apply flat field correction? + (illumcor = no) Apply iillumination correction? + (fringecor = no) Apply fringe correction? + (readcor = no) Convert zero level image to readout corr? + (scancor = no) Convert flat field image to scan correction? + (readaxis = "line") Read out axis (column|line) + (fixfile = "") File describing the bad lines and columns + (\fIbiassec\fR = "[m:n,*]") Overscan strip image section + (\fItrimsec\fR = "[r:s,t:u]") Trim data section + (\fIzero\fR = "zero1") Zero level calibration image + (\fIdark\fR = "dark1") Dark count calibration image + (\fIflat\fR = "") Flat field images + (illum = "") Iillumination correction images + (fringe = "") Fringe correction images + (minreplace = 1.) Minimum flat field value + (scantype = "shortscan") Scan type (shortscan|longscan) + (nscan = 1) Number of short scan lines\n + (interactive = yes) Fit overscan interactively? + (function = "spline3") Fitting function + (order = 2) Number of polynomial terms or spline pieces + (sample = "*") Sample points to fit + (naverage = 1) Number of sample points to combine + (niterate = 1) Number of rejection iterations + (low_reject = 3.) Low sigma rejection factor + (high_reject = 3.) High sigma rejection factor + (grow = 0.) Rejection growing radius + (mode = "ql") +.fi +.le +.bp +.ls \fI3.6 Preparing a Final Flat Field\fR + +Next we want to check the iillumination of the flat by applying it to the sky +flat. Load the 'imred' and 'ccdred' packages if they are +not already loaded. +There is no tried and true method for testing the iillumination of +the sky flats except for applying them on a long exposure taken in the same +filter and implotting it or doing image statistics. + +We must use 'ccdproc' to overscan subtract, trim, bias subtract, dark subtract +(if needed), and flat field the sky images. Again, we assume here +that the dark is +necessary for the reductions. Be sure that the images you are using for +the flat field division have the proper filter identification. +First edit the parameters for 'ccdproc', and then run +this task to process your skys to see if the iillumination of the dome +flat is uniform; + +.nf + cl> noao + im> imred + im> ccdred + cc> epar ccdproc (check the parameter list) + cc> ccdproc sky1R +.fi + +Now we must check the sky to see if it really is flat. This is done +using 'implot' and plotting some lines or a group of lines, + +.nf + cc> implot sky1R + :l 150 200 + :c 200 250 + q +.fi + +Ideally if the iillumination was uniform, sky1R should not show any variation. +In plotting several lines together, you may get a small slope (see figure 4). +If you see any +variations, including a slope even near the edges of the image, then you must +run the task 'mkskyflat'. Select a smoothing box size which preserves the large +scale features while adequately suppressing the noise. The scale of the +largest features expected shouldn't be smaller than say 1/10 to 1/12 the +image size. + +.nf + cc> epar mkskyflat (check the parameter list) + cc> mkskyflat sky1R finalflat1R +.fi + +The output image is the corrected flat image so call it "finalflat1R". +We need to recopy the sky flat to process it again. Do an imcopy of the sky1R +from the backup image after deleting the processed one. + +.nf + cc> imdel sky1R + cc> imcopy Bsky1R sky1R + cc> epar ccdproc (change the parameter 'flat' from + flat1R to finalflat1R) + cc> ccdproc sky1R +.fi + +Again, check to see if the sky is really flat. If it is flat +(see figure 5), then you are +done and must process the flats for your other filters. If it is not flat, +go back to the beginning and ask for help. It may be that you need to play +with the smoothing parameters. Repeat these steps until you are satisfied. +Now you can process your images. + +.nf + + \fIccdproc for sky flats\fR + + images = "sky1*imh" List of CCD images to correct + (\fIccdtype\fR = "other") CCD image type to correct + (max_cache = 0) Maximum image caching memory (in Mbytes) + (noproc = no) List processing steps only? + (fixpix = no) Fix bad CCD lines and columns? + (\fIoverscan\fR = yes) Apply overscan strip correction? + (\fItrim\fR = yes) Trim the image? + (\fIzerocor\fR = yes) Apply zero level correction? + (\fIdarkcor\fR = yes) Apply dark count correction? + (\fIflatcor\fR = yes) Apply flat field correction? + (illumcor = no) Apply iillumination correction? + (fringecor = no) Apply fringe correction? + (readcor = no) Convert zero level image to readout corr? + (scancor = no) Convert flat field image to scan correction? + (readaxis = "line") Read out axis (column|line) + (fixfile = "") File describing the bad lines and columns + (\fIbiassec\fR = "[m:n,*]") Overscan strip image section + (\fItrimsec\fR = "[r:s,t:u]") Trim data section + (\fIzero\fR = "zero1") Zero level calibration image + (\fIdark\fR = "dark1") Dark count calibration image + (\fIflat\fR = "flat1*imh") Flat field images + (illum = "") Iillumination correction images + (fringe = "") Fringe correction images + (minreplace = 1.) Minimum flat field value + (scantype = "shortscan") Scan type (shortscan|longscan) + (nscan = 1) Number of short scan lines\n + (interactive = yes) Fit overscan interactively? + (function = "spline3") Fitting function + (order = 2) Number of polynomial terms or spline pieces + (sample = "*") Sample points to fit + (naverage = 1) Number of sample points to combine + (niterate = 1) Number of rejection iterations + (low_reject = 3.) Low sigma rejection factor + (high_reject = 3.) High sigma rejection factor + (grow = 0.) Rejection growing radius + (mode = "ql") +.fi +.bp +.nf + \fImkskyflat\fR + + input = "sky1R" Input CCD images + output = "finalflat1R" Output images (same as input if none) + (ccdtype = "") CCD image type to select + (\fIxboxmin\fR = 0.1) Minimum smoothing box size in x at edges + (\fIxboxmax\fR = 0.1) Maximum smoothing box size in x + (\fIyboxmin\fR = 0.1) Minimum smoothing box size in y at edges + (\fIyboxmax\fR = 0.1) Maximum smoothing box size in y + (clip = yes) Clip input pixels? + (lowsigma = 2.5) Low clipping sigma + (highsigma = 2.5) High clipping sigma + (ccdproc = "") CCD processing parameters + (mode = "ql") +.fi +.le +.bp +.ls \fI3.7 Processing the Images\fR + +This can be setup to be done all at once. The final flat has been made so now +we divide it into your objects. +If the calibration frames themselves +have not been processed it will do this as well. The program takes into +account the differing filters. Be sure that you have deleted all the individual +calibration frames that were combined. Edit the parameters for 'ccdproc' and run +it. For tasks that take a long time, it is a good idea to set them going as a +background job and send the output to a file which you can watch to keep track +of the progress. You may wish to make a list of the objects to be processed +to avoid reprocessing your sky flats. + +.nf + cc> epar ccdproc (check the parameter list) + cc> ccdproc obj*.imh +.fi + +To check the progress of the task, just type + +.nf + cc> tail ccdlog +.fi + +This will print out the last lines of the file named 'ccdlog' +which is the latest +operation done by the task 'ccdproc'. +Once this is done, check your images to see +that they have been properly flat fielded. If you are satisfied, then you may +now check your images for bad pixel regions, the structure of which may not have +been processed out of your images. + +We suggest also to do a 'ccdlist' on the objects in order to +check whether the frames have been duly processed, i.e., + +.nf + cc> ccdlist obj*.imh +.fi + +You should get a listing of your images in the format that follows, + +.nf + + obj1010.imh[496,501][real][object][R][OTZF]:rubin 152 R + obj1011.imh[496,501][real][object][I][OTZF]:rubin 152 I + obj1012.imh[496,501][real][object][B][OTZF]:rubin 149 B + obj1013.imh[496,501][real][object][V][OTZF]:rubin 149 V + obj1014.imh[496,501][real][object][R][OTZF]:rubin 149 R + obj1015.imh[496,501][real][object][I][OTZF]:rubin 149 I + obj1016.imh[496,501][real][object][B][OTZF]:rubin 149 B + obj1017.imh[496,501][real][object][V][OTZF]:rubin 149 V + obj1018.imh[496,501][real][object][R][OTZF]:rubin 149 R + obj1019.imh[496,501][real][object][I][OTZF]:rubin 149 I +.fi + +where the [OTZF] code reveals the degree of processing of the images. + +.nf + \fIccdproc for objects\fR + + images = "obj*imh)" List of CCD images to correct + (\fIccdtype\fR = "object") CCD image type to correct + (max_cache = 0) Maximum image caching memory (in Mbytes) + (noproc = no) List processing steps only? + (fixpix = no) Fix bad CCD lines and columns? + (\fIoverscan\fR = yes) Apply overscan strip correction? + (\fItrim\fR = yes) Trim the image? + (\fIzerocor\fR = yes) Apply zero level correction? + (\fIdarkcor\fR = yes) Apply dark count correction? + (\fIflatcor\fR = yes) Apply flat field correction? + (illumcor = no) Apply iillumination correction? + (fringecor = no) Apply fringe correction? + (readcor = no) Convert zero level image to readout corr? + (scancor = no) Convert flat field image to scan correction? + (readaxis = "line") Read out axis (column|line) + (fixfile = "") File describing the bad lines and columns + (\fIbiassec\fR = "[m:n,*]") Overscan strip image section + (\fItrimsec\fR = "[r:s,t:u]") Trim data section + (\fIzero\fR = "zero1") Zero level calibration image + (\fIdark\fR = "dark1") Dark count calibration image + (\fIflat\fR = "finalflat1*imh") Flat field images + (illum = "") Iillumination correction images + (fringe = "") Fringe correction images + (minreplace = 1.) Minimum flat field value + (scantype = "shortscan") Scan type (shortscan|longscan) + (nscan = 1) Number of short scan lines\n + (interactive = yes) Fit overscan interactively? + (function = "spline3") Fitting function + (order = 2) Number of polynomial terms or spline pieces + (sample = "*") Sample points to fit + (naverage = 1) Number of sample points to combine + (niterate = 1) Number of rejection iterations + (low_reject = 3.) Low sigma rejection factor + (high_reject = 3.) High sigma rejection factor + (grow = 0.) Rejection growing radius + (mode = "ql") + +.fi + +.le +.bp +.ls \fI3.8 Creating the Bad Pixel File\fR + +Now it is time to check your images to see if structure remains in the bad +pixel regions. If you were careful to observe your objects far away from any +obvious bad regions then you may only need to 'fix' pixels for cosmetic reasons. +We suggest you to examine these files using the display window, +imtool (this can only be done at one of the SUN consoles), and then plotting +them using 'implot'. Start by loading the 'noao', 'images', 'tv', 'imred', +and 'ccdred' packages. Display +a processed image and plot it using 'implot', + +.nf + cl> noao + no> images + im> tv + tv> imred + im> ccdred + cc> display obj002 1 + cc> implot obj002 +.fi + +Do not exit from implot, but move the cursor to the imtool window and press +'F6'. The cursor coordinates are now visible at the bottom of the screen. +Find the position of a bad pixel using this, return to the graph and +plot it using 'implot' with the ':l #'(line) and ':c #'(column) commands. +You can define them more precisely using 'implot' by overplotting lines and +columns around the center of the bad regions. Do this by typing 'o' followed +by ':c #' and ':l #'. Once you have these regions written down, check some of +your images to see if the bad pixels are a problem, ie >1% in signal. If it is +a problem then create a file called 'badpix' with the listing of +the pixels you wish to fix; + +.nf + cc> edit badpix (according to the following format) +.fi + +The following example is to illustrate the format of a bad pixel file. + +.nf + 84 87 1 450 + 87 90 105 109 + 87 89 110 111 + 124 126 112 116 + 206 206 80 80 +.fi + +Each line stands for a rectangular region of the image to be fixed. The regions +are specified by four numbers giving the starting and ending columns followed +by the starting and ending lines. The starting and ending points may be the +same to specify a single column, line or pixel. Note that each region is +"fixed" by interpolating across its shortest dimension and that regions are +processed in the order that they appear in the bad pixel list. For bad regions +of complex shape, some care may be required in specifying the regions in order +to achieve an optimal result. \fIIt is strongly recommended that you test your +bad pixel file by using it to correct a copy of one of your images before going +into production\fR. This bad pixel file will be specified in the task 'ccdproc' +in the parameter 'fixfile'. So now edit and run the task: + +.nf + cc> epar ccdproc + cc> ccdproc obj*.imh +.fi + +Remember that 'ccdproc' already knows that the images +have been processed so those +operations will not be performed again. Now you have final images that are ready +to be used. If you plan to do photometry in IRAF see section VI of this manual. + +.nf + \fIccdproc\fR + + images = "obj*imh)" List of CCD images to correct + (\fIccdtype\fR = "object") CCD image type to correct + (max_cache = 0) Maximum image caching memory (in Mbytes) + (noproc = no) List processing steps only? + (\fIfixpix\fR = yes) Fix bad CCD lines and columns? + (\fIoverscan\fR = yes) Apply overscan strip correction? + (\fItrim\fR = yes) Trim the image? + (\fIzerocor\fR = yes) Apply zero level correction? + (\fIdarkcor\fR = yes) Apply dark count correction? + (\fIflatcor\fR = yes) Apply flat field correction? + (illumcor = no) Apply iillumination correction? + (fringecor = no) Apply fringe correction? + (readcor = no) Convert zero level image to readout corr? + (scancor = no) Convert flat field image to scan correction? + (readaxis = "line") Read out axis (column|line) + (\fIfixfile\fR = "badpix") File describing the bad lines and columns + (\fIbiassec\fR = "[m:n,*]") Overscan strip image section + (\fItrimsec\fR = "[r:s,t:u]") Trim data section + (\fIzero\fR = "zero1") Zero level calibration image + (\fIdark\fR = "dark1") Dark count calibration image + (\fIflat\fR = "finalflat1*imh") Flat field images + (illum = "") Iillumination correction images + (fringe = "") Fringe correction images + (minreplace = 1.) Minimum flat field value + (scantype = "shortscan") Scan type (shortscan|longscan) + (nscan = 1) Number of short scan lines\n + (interactive = yes) Fit overscan interactively? + (function = "spline3") Fitting function + (order = 2) Number of polynomial terms or spline pieces + (sample = "*") Sample points to fit + (naverage = 1) Number of sample points to combine + (niterate = 1) Number of rejection iterations + (low_reject = 3.) Low sigma rejection factor + (high_reject = 3.) High sigma rejection factor + (grow = 0.) Rejection growing radius + (mode = "ql") + +.fi +.le +.bp +.ls \fI3.9 Writing the Data to Tape\fR + +Allocate the device, for the SUNs it is "mta". Then mount your tape, don't +forget to put in a write ring. If you wish to use the exabyte drive, use "mtx" +for the device name. Load the package 'dataio' and do an epar on 'wfits'. + +.nf + cl> dataio + da> epar wfits + da> wfits *.imh mta.6250 new+ +.fi + +The parameter newtape should be "yes" +if you are starting on a new tape. Otherwise you will not want to overwrite +data which has already been placed on a tape, so use "no". + +After writing all the desired files to tape, deallocate the drive and retrieve +your tape. + +.nf + da> dealloc mta +.fi +.le +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/quadproc.hlp b/noao/imred/quadred/src/quad/doc/quadproc.hlp new file mode 100644 index 00000000..8d445097 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/quadproc.hlp @@ -0,0 +1,672 @@ +.help quadproc Sept93 arcon.quad +.ih +NAME +quadproc -- Process multi-readout CCD images +.ih +USAGE +quadproc images +.ih +PARAMETERS +.ls images +List of input CCD images to process. The list may include processed +images and calibration images. +.le +.ls ccdtype = "" +CCD image type to select from the input image list. If no type is given +then all input images will be selected. The recognized types are described +in \fBccdtypes\fR. +.le +.ls max_cache = 0 +Maximum image caching memory (in Mbytes). If there is sufficient memory +the calibration images, such as zero level, dark count, and flat fields, +will be cached in memory when processing many input images. This +reduces the disk I/O and makes the task run a little faster. If the +value is zero image caching is not used. +.le +.ls noproc = no +List processing steps only? +.le + +.ce +PROCESSING SWITCHES +.ls fixpix = yes +Fix bad CCD lines and columns by linear interpolation from neighboring +lines and columns? If yes then a bad pixel file must be specified. +.le +.ls overscan = yes +Apply overscan or prescan bias correction? If yes then the overscan +image section and the readout axis must be specified. +.le +.ls trim = yes +Trim the image of the overscan region and bad edge lines and columns? +If yes then the trim section must be specified. +.le +.ls zerocor = yes +Apply zero level correction? If yes a zero level image must be specified. +.le +.ls darkcor = yes +Apply dark count correction? If yes a dark count image must be specified. +.le +.ls flatcor = yes +Apply flat field correction? If yes flat field images must be specified. +.le +.ls illumcor = no +Apply iillumination correction? If yes iillumination images must be specified. +.le +.ls fringecor = no +Apply fringe correction? If yes fringe images must be specified. +.le +.ls readcor = no +Convert zero level images to readout correction images? If yes then +zero level images are averaged across the readout axis to form one +dimensional zero level readout correction images. +.le +.ls scancor = no +Convert flat field images to scan mode flat field images? If yes then the +form of scan mode correction is specified by the parameter \fIscantype\fR. +.le + +.ce +PROCESSING PARAMETERS +.ls readaxis = "line" +Read out axis specified as "line" or "column". +.le +.ls fixfile +File describing the bad lines and columns. If "image" is specified then +the file is specified in the image header or instrument translation file. +See Section 2. of Description for further information on bad pixel files. +.le +.ls biassec +Overscan bias strip image section. If "image" is specified then the overscan +bias section is specified in the image header or instrument translation file. +See Section 3. of Description for further information on setting this parmeter. +.le +.ls trimsec +image section for trimming. If "image" is specified then the trim +image section is specified in the image header or instrument translation file. +See Section 4. of Description for further information on setting this parmeter. +.le +.ls zero = "" +Zero level calibration image. The zero level image may be one or two +dimensional. The CCD image type and subset are not checked for these +images and they take precedence over any zero level calibration images +given in the input list. +.le +.ls dark = "" +Dark count calibration image. The CCD image type and subset are not checked +for these images and they take precedence over any dark count calibration +images given in the input list. +.le +.ls flat = "" +Flat field calibration images. The flat field images may be one or +two dimensional. The CCD image type is not checked for these +images and they take precedence over any flat field calibration images given +in the input list. The flat field image with the same subset as the +input image being processed is selected. +.le +.ls illum = "" +Iillumination correction images. The CCD image type is not checked for these +images and they take precedence over any iillumination correction images given +in the input list. The iillumination image with the same subset as the +input image being processed is selected. +.le +.ls fringe = "" +Fringe correction images. The CCD image type is not checked for these +images and they take precedence over any fringe correction images given +in the input list. The fringe image with the same subset as the +input image being processed is selected. +.le +.ls minreplace = 1. +When processing flat fields, pixel values below this value (after +all other processing such as overscan, zero, and dark corrections) are +replaced by this value. This allows flat fields processed by \fBquadproc\fR +to be certain to avoid divide by zero problems when applied to object +images. +.le +.ls scantype = "shortscan" +Type of scan format used in creating the CCD images. The modes are: +.ls "shortscan" +The CCD is scanned over a number of lines and then read out as a regular +two dimensional image. In this mode unscanned flat fields are numerically +scanned to form scanned flat fields comparable to the observations. If +the flat field calibration images are taken in scanned mode then +\fIscancor\fR should be no and the processing performed in the same manner +as in unscanned mode. +.le +.ls "longscan" +In this mode the CCD is clocked and read out continuously to form a long +strip. Flat fields are averaged across the readout axis to +form a one dimensional flat field readout correction image. This assumes +that all recorded image lines are clocked over the entire active area of the +CCD. +.le +.le +.ls nscan +Number of scan readout lines used in short scan mode. This parameter is used +when the scan type is "shortscan". +.le + +.ce +OVERSCAN FITTING PARAMETERS +.ls interactive = no +Fit the overscan vector interactively? If yes the overscan vector is fit +interactively using the \fBicfit\fR package. If no then the fitting parameters +given below are used. +.le +.ls function = "legendre" +Overscan fitting function. The function types are "legendre" polynomial, +"chebyshev" polynomial, "spline1" linear spline, and "spline3" cubic +spline. +.le +.ls order = 1 +Number of polynomial terms or spline pieces in the overscan fit. +.le +.ls sample = "*" +Sample points to use in the overscan fit. The string "*" specified all +points otherwise an \fBicfit\fR range string is used. +.le +.ls naverage = 1 +Number of points to average or median to form fitting points. Positive +numbers specify averages and negative numbers specify medians. +.le +.ls niterate = 1 +Number of rejection iterations to remove deviant points from the overscan fit. +If 0 then no points are rejected. +.le +.ls low_reject = 3., high_reject = 3. +Low and high sigma rejection factors for rejecting deviant points from the +overscan fit. +.le +.ls grow = 0. +One dimensional growing radius for rejection of neighbors to deviant points. +.le +.ih +DESCRIPTION +\fBQuadproc\fR processes CCD images to remove all "instrumental signatures" from +the data. The operations performed are: +.ls +.nf +o correct detector defects (bad lines and columns) +o determine readout bias level using overscan and subtract it +o trim off the overscan regions and unwanted border pixels +o subtract zero level bias +o subtract dark counts +o correct for pixel-to-pixel sensitivity variations +o correct for non-uniform iillumination +o correct for fringing +.fi +.le +.sp 1 +\fBQuadproc\fR is a cl script based on the task \fBccdproc\fR in the +\fBccdred\fR package. It is specifically designed to deal with Arcon data +obtained in multi-readout mode (see \fBquadformat\fR). A feature of such +images is that each readout typically has a slightly different, DC bias +level, gain, and readout noise. As a result both zero frames and uniformly +illuminated exposures show a characteristic chequer board pattern, the +sections of the image read through each amplifier having different levels. +In addition, there will be a separate overscan strip, used to monitor the zero +level, for each readout. The location of these overscan strips in the raw +frame depends on which amplifiers are used. \fBQuadproc\fR splits each +multi-readout image into subimages, one for each amplifier, and also calculates +the biassec and trimsec appropriately for each. It then calls \fBccdproc\fR to +perform the first three operations listed above. The sub-images are then glued +back together. Finaly, \fBccdproc\fR is called a second time to perform all the +remaining reduction steps. + +\fBQuadproc\fR MUST be used for the reduction of multi-readout data up to and +including the trimming step, and it is convenient to use it for the entire +reduction process. However, once ALL images have been trimmed it is possible +to finish the reductions using \fBccdproc\fR if the \fBquad\fR package is not +available at your home institution. \fBQuadproc\fR recognizes mono-readout +images and processes them directly using \fBccdproc\fR. If your images are a +mixture of multi- and mono- readout use \fBquadproc\fR; if you only have +mono-readout data use \fBccdproc\fR. + +\fBQuadproc\fR is identical to \fBccdproc\fR in the way it is used, and has +exactly the same parameters; as far as possible it also behaves in the same way. +To run it, all one has to do is set the parameters and then begin processing +the images. The task takes care of most of the record keeping and +automatically does the prerequisite processing of calibration images. For +ease of reference, the following sections provide a simple outline of how to +use the task, together with a description of the operations performed. They +are taken almost verbatim from the help page for \fBccdproc\fR. If you are +already familiar with that task you should read sections 2., 3. and 4. below, +which include information on the preparation of the badpixel file, and on how +to specify \fBbiassec\fR and \fBtrimsec\fR parameters. See section 12. for a +description of the differences between the two tasks. For a user's guide and +cookbook for the \fBquad\fR package see \fBguide\fR. +.sh +1. Parameters +There are many parameters but they may be easily reviewed and modified using +the task \fBeparam\fR. +The input CCD images to be processed are given as an image list. +Previously processed images are ignored and calibration images are +recognized, provided the CCD image types are in the image header (see +\fBinstruments\fR and \fBccdtypes\fR). \fBQuadproc\fR separates multi- and +mono-readout images in the input list and handles them accordingly. +Therefore it is permissible to use simple image templates such as "*.imh". +The \fIccdtype\fR parameter may be used to select only certain types of CCD +images to process (see \fBccdtypes\fR). + +The processing operations are selected by boolean (yes/no) parameters. +Because calibration images are recognized and processed appropriately, +the processing operations for object images should be set. Any combination of +operations may be specified. Two of the operations, \fBreadcor\fR and \fBscancor\fR, are only applicable to zero level and flat field images respectively. These +are used for certain types of CCDs and modes of operation. + +The processing steps selected have related parameters which must be +set. These are things like image sections defining the overscan and +trim regions and calibration images. There are a number of parameters +used for fitting the overscan or prescan bias section. These are +parameters used by the standard IRAF curve fitting package \fBicfit\fR. +The parameters are described in more detail in the following sections. + +In addition to the task parameters there are package parameters +which affect \fBquadproc\fR. These include the instrument and subset +files, the text and plot log files, the output pixel datatype, +the verbose parameter for logging to the terminal, and the backup +prefix. These are described in \fBquad\fR. + +Calibration images are specified by task parameters and/or in the +input image list. If more than one calibration image is specified +then the first one encountered is used. Calibration images specified by +task parameters take precedence over calibration images in the input list. +These images also need not have a CCD image type parameter since the task +parameter identifies the type of calibration image. This method is +best if there is only one calibration image for all images +to be processed, almost always true for zero level and dark +count images. If no calibration image is specified by task parameter +then calibration images in the input image list are identified and +used. This requires that the images have CCD image types recognized +by the package. This method is useful if one may simply say "*.imh" +as the image list to process all images or if the images are broken +up into groups, in "@" files for example, each with their own calibration +frames. +.sh +2. Fixpix +Regions of bad lines and columns may be replaced by linear +interpolation from neighboring lines and columns when the parameter +\fIfixpix\fR is set. The bad regions are specified in a bad pixel +file. The file consists of lines with four fields, the starting and +ending columns and the starting and ending lines. Any number of +regions may be specified. Currently, the coordinates given for the bad regions +must be those that would be applicable if the CCD was used in SINGLE READOUT +MODE, even if multi-readout images are being reduced. A task is being written +to aid in the preparation of an appropriate bad-pixel file given measurements +made on a raw multi-readout image. + +Comment lines beginning with the character '#' may be included. If a comment +line preceding the bad regions contains the word "untrimmed" then the +coordinate system refers to the original format of the images; i.e. before +trimming. If an image has been trimmed previously then the trim region +specified in the image header is used to convert the coordinates in the bad +pixel file to those of the trimmed image. If the file does not contain the +word "untrimmed" then the coordinate system must match that of the image +being corrected; i.e. untrimmed coordinates if the image has not been +trimmed and trimmed coordinates if the image has been trimmed. +Standard bad pixel files should always be specified in terms of the original +format. + +The bad pixel file may be specified explicitly with the parameter \fIfixfile\fR +or indirectly if the parameter has the value "image". In the latter case +the instrument file must contain the name of the file. +.sh +3. Overscan +The portion of the image used to determine the readout bias level is specified +with the parameter \fBbiassec\fR. This may be an explicit image section, or it +may be set to the special value "image". In the latter case the value given in +the image header is used. The image header value uses the entire overscan +strip without allowing any margin between the data section and the bias +section. Because Arcon uses a DC-coupled preamplifier the transition +between data and overscan is very sharp indeed. Nonetheless, we recommend that +you do skip the first few pixels of the overscan strip. To decide this issue +for yourself, use implot to plot the average of several lines from a high +exposure level image such as a flat field. Expand the transition region +between data and overscan and decide how many pixels of the overscan are +contaminated. + +In the case of multi-readout images, the way in which an explicit value for +\fBbiassec\fR must be set, is unfortunately somewhat non-intuitive. Currently, +the value recorded in the image header is that which would be appropriate had +the detector been read out using a single amplifier; an explicit image section +must be specified in the same way. \fBQuadproc\fR calculates the sections +to use for the sub-images corresponding to each readout based on such "single +readout" sections. To determine the section you must enter, use \fBimhead\fR +or \fBhselect\fR to determine the value of \fBbiassec\fR stored in the image +header. If this is, for instance, "[1025:1060,1:1028]" then setting +\fBbiassec\fR = "[1029:1060,1:1028]" would leave a margin of 4 pixels +(1029 - 1025). Note that if two readouts are used in the horizontal direction +(quad or serial-split dual readout) the overscan strip for each amplifier is +only half as wide as that in single readout mode. Thus in the example a 15 +pixel (36 / 2 - 3) wide strip is used for each readout. + +If an overscan or prescan correction is specified (\fIoverscan\fR +parameter) then the specified image section is averaged +along the readout axis (\fIreadaxis\fR parameter) to form a +correction vector. A function is fit to this vector and for each readout +line (image line or column) the function value for that line is +subtracted from the image line. The fitting function is generally +either a constant (polynomial of 1 term) or a high order function +which fits the large scale shape of the overscan vector. Bad pixel +rejection is also used to eliminate cosmic ray events. The function +fitting may be done interactively using the standard \fBicfit\fR +iteractive graphical curve fitting tool. Regardless of whether the fit +is done interactively, the overscan vector and the fit may be recorded +for later review in a metacode plot file named by the parameter +\fIquad.plotfile\fR. The mean value of the bias function is also recorded in +the image header and log file. + +The overscan subtraction performed by \fBquadproc\fR corrects the +amplifier-to-amplifier differences in the bias level, so that no +readout structure should be visible in processed zero images. However, you +will still see the chequer board structure in flatfield and object exposures +(unless the sky level is zero) because of gain difference between the +amplifiers. +.sh +4. Trim +When the parameter \fItrim\fR is set the input image will be trimmed to +the image section given by the parameter \fItrimsec\fR. This may be an explicit +image section, or it may be set to the special value "image". In the latter +case the value given in the image header is used. The image header value keeps +the entire imaging section of the CCD. + +In the case of multi-readout images, the way in which an explicit value for +\fBtrimsec\fR must be set, is unfortunately somewhat non-intuitive. Currently, +the value recorded in the image header is that which would be appropriate had +the detector been read out using a single amplifier; an explicit image section +must be specified in the same way. \fBQuadproc\fR calculates the sections +to use for the sub-images corresponding to each readout based on such "single +readout" sections. In addition one is currently restricted to trimming exactly +the same number of columns from each side of the CCD; there is no such +restriction on the number of lines which can be trimmed from the top and bottom +edges of the image. To determine the section you must enter, use \fBimhead\fR +or \fBhselect\fR to determine the value of \fBtrimsec\fR stored in the image +header. If this is, for instance, "[1:1024,1:1028]" then setting +\fBtrimsec\fR = "[10:1015,20:998]" would trim 9 columns from the left and right +edges and 19 and 29 lines from the bottom and top edges respectively. If you +need to perform an asymmetric trim in the horizontal direction this can be +done, after processing, by using \fBimcopy\fR to copy the required portion of +the image. + +The trim section used for science images should, of course, be the same as +that used for the calibration images. +.sh +5. Zerocor +After the readout bias is subtracted, as defined by the overscan or prescan +region, there may still be a zero level bias. This level may be two +dimensional or one dimensional (the same for every readout line). A +zero level calibration is obtained by taking zero length exposures; +generally many are taken and combined. To apply this zero +level calibration the parameter \fIzerocor\fR is set. In addition if +the zero level bias is only readout dependent then the parameter \fIreadcor\fR +is set to reduce two dimensional zero level images to one dimensional +images. The zero level images may be specified by the parameter \fIzero\fR +or given in the input image list (provided the CCD image type is defined). + +When the zero level image is needed to correct an input image it is checked +to see if it has been processed and, if not, it is processed automatically. +Processing of zero level images consists of bad pixel replacement, +overscan correction, trimming, and averaging to one dimension if the +readout correction is specified. +.sh +6. Darkcor +Dark counts are subtracted by scaling a dark count calibration image to +the same exposure time as the input image and subtracting. The +exposure time used is the dark time which may be different than the +actual integration or exposure time. A dark count calibration image is +obtained by taking a very long exposure with the shutter closed; i.e. +an exposure with no light reaching the detector. The dark count +correction is selected with the parameter \fIdarkcor\fR and the dark +count calibration image is specified either with the parameter +\fIdark\fR or as one of the input images. The dark count image is +automatically processed as needed. Processing of dark count images +consists of bad pixel replacement, overscan and zero level correction, +and trimming. +.sh +7. Flatcor +The relative detector pixel response is calibrated by dividing by a +scaled flat field calibration image. A flat field image is obtained by +exposure to a spatially uniform source of light such as an lamp or +twilight sky. Flat field images may be corrected for the spectral +signature in spectroscopic images (see \fBresponse\fR and +\fBapnormalize\fR), or for iillumination effects (see \fBmkillumflat\fR +or \fBmkskyflat\fR). For more on flat fields and iillumination corrections +see \fBflatfields\fR. The flat field response is dependent on the +wavelength of light so if different filters or spectroscopic wavelength +coverage are used a flat field calibration for each one is required. +The different flat fields are automatically selected by a subset +parameter (see \fBsubsets\fR). + +Flat field calibration is selected with the parameter \fBflatcor\fR +and the flat field images are specified with the parameter \fBflat\fR +or as part of the input image list. The appropriate subset is automatically +selected for each input image processed. The flat field image is +automatically processed as needed. Processing consists of bad pixel +replacement, overscan subtraction, zero level subtraction, dark count +subtraction, and trimming. Also if a scan mode is used and the +parameter \fIscancor\fR is specified then a scan mode correction is +applied (see below). The processing also computes the mean of the +flat field image which is used later to scale the flat field before +division into the input image. For scan mode flat fields the ramp +part is included in computing the mean which will affect the level +of images processed with this flat field. Note that there is no check for +division by zero in the interest of efficiency. If division by zero +does occur a fatal error will occur. The flat field can be fixed by +replacing small values using a task such as \fBimreplace\fR or +during processing using the \fIminreplace\fR parameter. Note that the +\fIminreplace\fR parameter only applies to flat fields processed by +\fBquadproc\fR. +.sh +8. Illumcor +CCD images processed through the flat field calibration may not be +completely flat (in the absence of objects). In particular, a blank +sky image may still show gradients. This residual nonflatness is called +the iillumination pattern. It may be introduced even if the detector is +uniformly illuminated by the sky because the flat field lamp +iillumination may be nonuniform. The iillumination pattern is found from a +blank sky, or even object image, by heavily smoothing and rejecting +objects using sigma clipping. The iillumination calibration image is +divided into the data being processed to remove the iillumination +pattern. The iillumination pattern is a function of the subset so there +must be an iillumination correction image for each subset to be +processed. The tasks \fBmkillumcor\fR and \fBmkskycor\fR are used to +create the iillumination correction images. For more on iillumination +corrections see \fBflatfields\fR. + +An alternative to treating the iillumination correction as a separate +operation is to combine the flat field and iillumination correction +into a corrected flat field image before processing the object +images. This will save some processing time but does require creating +the flat field first rather than correcting the images at the same +time or later. There are two methods, removing the large scale +shape of the flat field and combining a blank sky image iillumination +with the flat field. These methods are discussed further in the +tasks which create them; \fBmkillumcor\fR and \fBmkskycor\fR. +.sh +9. Fringecor +There may be a fringe pattern in the images due to the night sky lines. +To remove this fringe pattern a blank sky image is heavily smoothed +to produce an iillumination image which is then subtracted from the +original sky image. The residual fringe pattern is scaled to the +exposure time of the image to be fringe corrected and then subtracted. +Because the intensity of the night sky lines varies with time an +additional scaling factor may be given in the image header. +The fringe pattern is a function of the subset so there must be +a fringe correction image for each subset to be processed. +The task \fBmkfringecor\fR is used to create the fringe correction images. +.sh +10. Readcor +If a zero level correction is desired (\fIzerocor\fR parameter) +and the parameter \fIreadcor\fR is yes then a single zero level +correction vector is applied to each readout line or column. Use of a +readout correction rather than a two dimensional zero level image +depends on the nature of the detector or if the CCD is operated in +longscan mode (see below). The readout correction is specified by a +one dimensional image (\fIzero\fR parameter) and the readout axis +(\fIreadaxis\fR parameter). If the zero level image is two dimensional +then it is automatically processed to a one dimensional image by +averaging across the readout axis. Note that this modifies the zero +level calibration image. +.sh +11. Scancor +CCD detectors may be operated in several modes in astronomical +applications. The most common is as a direct imager where each pixel +integrates one point in the sky or spectrum. However, the design of most CCD's +allows the sky to be scanned across the CCD while shifting the +accumulating signal at the same rate. \fBQuadproc\fR provides for two +scanning modes called "shortscan" and "longscan". The type of scan +mode is set with the parameter \fIscanmode\fR. + +In "shortscan" mode the detector is scanned over a specified number of +lines (not necessarily at sideral rates). The lines that scroll off +the detector during the integration are thrown away. At the end of the +integration the detector is read out in the same way as an unscanned +observation. The advantage of this mode is that the small scale flat +field response is averaged in one dimension over the number of lines +scanned. A flat field may be observed in the same way in which case +there is no difference in the processing from unscanned imaging and the +parameter \fIscancor\fR should be no. However, one obtains an increase +in the statistical accuracy of the flat fields if they are not scanned +during the observation but digitally scanned during the processing. In +shortscan mode with \fIscancor\fR set to yes, flat field images are +digitally scanned, if needed, by the specified number of scan lines +(\fInscan\fR parameter). + +In "longscan" mode the detector is continuously read out to produce +an arbitrarily long strip. Provided data which has not passed over +the entire detector is thrown away, the flat field corrections will +be one dimensional. If \fIscancor\fR is specified and the +scan mode is "longscan" then a one dimensional flat field correction +will be applied. If the specified flat field (\fIflat\fR parameter) +is a two dimensional image then when the flat field image is processed +it will be averaged across the readout axis to form a one dimensional +correction image. +.sh +12. Outline of Processing Steps + +Because of the special handling required for multi-readout data +\fBquadproc\fR internally reduces the data in two stages. + +.ls Stage one +The operations which may be performed in the first stage are badpixel +correction, determination and subtraction of the readout bias level, and +trimming. This stage is only performed if one or more of the \fBfixpix\fR, +\fBoverscan\fR or \fBtrim\fR flags is set to yes. + +First, all the calibration images which will be needed are identified. Any +which were obtained in multi-readout mode AND which have not already been +trimmed are selected for processing during this stage. This is necessary to +ensure that the calibration images will be reduced properly. Similarly, the +input list is searched and all multi-readout images, which have not already +been trimmed are selected for processing. + +The images selected in this way are then processed sequentially. Each is split +into separate images one for each amplifier. The values of the trimsec and +biassec header keywords for each of these sub-images are set as required. +\fBccdproc\fR is then run to correct bad pixels, determine and subtract the +readout bias and trim each sub-image. Finaly, the pieces are glued back +together again to form the complete image and the header information is +tidied up. The resulting image is initialy created as a temporary image. +When stage one processing is complete the original image is deleted (or +renamed using the specified backup prefix) and the corrected image replaces +the original image. Using a temporary image protects the data in the +event of an abort or computer failure. Keeping the original image name +eliminates much of the record keeping and the need to generate new +image names. +.le +.ls Stage two +\fBCcdproc\fR is now run a second time to process ALL input images. For those +images which were NOT selected for processing during stage one all the selected +processing steps are carried out during this second pass. For those which were +selected in stage one only the remaining processing steps will be performed. +Again the output processed image is initialy created as a temporary image. +When stage two processing is complete the original image is deleted (or +renamed using the specified backup prefix) and the corrected image replaces +the original image. +.le + +The following difference in the behaviour of \fBquadproc\fB and \fBccdproc\fR +should be noted: +.ls +Because it is a script, and because it is reads and writes each image several +times during processing \fBquadproc\fR is not very efficiant. This will be +rectified when the present prototype code is replaced by the final version. +.le +.ls +If backups are enable then \fBquadproc\fR will produce two intermediate +images for every input image which is modified in both processing stages. +These backup images may quickly fill up the available disk space. +.le +.ls +Images may not be processed in the order they appear in the input list. Stage +one processing is performed (if necessary) on all calibration images, then on +all images in the input list. Any images which have already been trimmed, or +which were taken in mono-readout mode will be skipped. Stage two processing is +then done sequentially on all images in the input list. +.le +.sh +13. Processing Arithmetic +The \fBquadproc\fR task has two data paths, one for real image pixel datatypes +and one for short integer pixel datatype. In addition internal arithmetic +is based on the rules of FORTRAN. For efficiency there is +no checking for division by zero in the flat field calibration. +The following rules describe the processing arithmetic and data paths. + +.ls (1) +If the input, output, or any calibration image is of type real the +real data path is used. This means all image data is converted to +real on input. If all the images are of type short all input data +is kept as short integers. Thus, if all the images are of the same type +there is no datatype conversion on input resulting in greater +image I/O efficiency. +.le +.ls (2) +In the real data path the processing arithmetic is always real and, +if the output image is of short pixel datatype, the result +is truncated. +.le +.ls (3) +The overscan vector and the scale factors for dark count, flat field, +iillumination, and fringe calibrations are always of type real. Therefore, +in the short data path any processing which includes these operations +will be coerced to real arithmetic and the result truncated at the end +of the computation. +.le +.sh +14. In the Absence of Image Header Information +The tasks in the \fBquad\fR package are most convenient to use when +the CCD image type, subset, and exposure time are contained in the +image header. This is true for all data obtained with Arcon. The ability to +redefine which header parameters contain this information makes it possible +to use the package at many different observatories (see \fBinstruments\fR). +However, in the absence of any image header information the tasks may still +be used effectively. There are two ways to proceed. One way is to use +\fBccdhedit\fR to place the information in the image header. + +The second way is to specify the processing operations more explicitly +than is needed when the header information is present. The parameter +\fIccdtype\fR is set to "" or to "none". The calibration images are +specified explicitly by task parameter since they cannot be recognized +in the input list. Only one subset at a time may be processed. + +If dark count and fringe corrections are to be applied the exposure +times must be added to all the images. Alternatively, the dark count +and fringe images may be scaled explicitly for each input image. This +works because the exposure times default to 1 if they are not given in +the image header. +.ih +EXAMPLES +The user's \fBguide\fR presents a tutorial in the use of this task. + +1. In general all that needs to be done is to set the task parameters +and enter + + cl> quadproc *.imh & + +This will run in the background and process all images which have not +been processed previously. +.ih +SEE ALSO +quadformat, ccdproc, instruments, ccdtypes, flatfields, icfit, quad, guide, +mkillumcor, mkskycor, mkfringecor +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/quadreadout.hlp b/noao/imred/quadred/src/quad/doc/quadreadout.hlp new file mode 100644 index 00000000..355fdb61 --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/quadreadout.hlp @@ -0,0 +1,19 @@ +With Arcon it is possible to read out a CCD using more than one amplifier in +parallel, leading to a substantial reduction in readout time. Once properly +reduced, such data is indistinguishable from that obtained when reading out +through only a single amplifier. However, the raw images are rather different +and consequently must be processed somewhat differently. Firstly, each readout +will typically have a slightly different, zero level, gain, and readout noise, +and may differ slightly in its departures from perfect linearity. As a result +both zero frames and uniformly illuminated exposures will show a characteristic +chequer board pattern, the sections of the data read through each amplifier +having different levels. Secondly, there will be a separate overscan strip, +used to monitor the zero level, for each readout. The location of these overscan +strips in the raw frame depends on which amplifiers are used. When all four +amplifiers are used the four overscan regions form a vertical stripe down the +centre of the raw image. A CCD read out through two amplifiers can have its +two overscan strips, running side by side down the center of the picture; or +they may be one above the other, on the same side of the picture; or the strip +for the upper and lower halves of the CCD can be at opposited sides of the +image (in which case there will also be a horizontal displacement of the data +in the two sections). diff --git a/noao/imred/quadred/src/quad/doc/setinstrument.hlp b/noao/imred/quadred/src/quad/doc/setinstrument.hlp new file mode 100644 index 00000000..410dd20f --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/setinstrument.hlp @@ -0,0 +1,97 @@ +.help setinstrument Oct87 noao.imred.ccdred +.ih +NAME +setinstrument -- Set instrument parameters +.ih +USAGE +setinstrument instrument +.ih +PARAMETERS +.ls instrument +Instrument identification for instrument parameters to be set. If '?' +then a list of the instrument identifiers is printed. +.le +.ls site = "kpno" +Site ID. +.le +.ls directory = "ccddb$" +Instrument directory containing instrument files. The instrument files +are found in the subdirectory given by the site ID. +.le +.ls review = yes +Review the instrument parameters? If yes then \fBeparam\fR is run for +the parameters of \fBccdred\fR and \fBccdproc\fR. +.le +.ls query +Parameter query if initial instrument is not found. +.le +.ih +DESCRIPTION +The purpose of the task is to allow the user to easily set default +parameters for a new instrument. The default parameters are generally +defined by support personal in an instrument directory for a particular +site. The instrument directory is the concatenation of the specified +directory and the site. For example if the directory is "ccddb$" and +the site is "kpno" then the instrument directory is "ccddb$kpno/". +The user may have his own set of instrument files in a local directory. +The current directory is used by setting the directory and site to the +null string (""). + +The user specifies an instrument identifier. This instrument may +be specific to a particular observatory, telescope, instrument, and +detector. If the character '?' is specified or the instrument file is +not found then a list of instruments +in the instrument directory is produced by paging the file "instruments.men". +The task then performs the following operations: +.ls (1) +If an instrument translation file with the name given by the instrument +ID and the extension ".dat" is found then the instrument translation +file parameter, \fIccdred.instrument\fR, is set to this file. +If it does not exist then the user is queried again. Note that a +null instrument, "", is allowed to set no translation file. +.le +.ls (2) +If an instrument setup script with the name given by the instrument ID +and the extension ".cl" is found then the commands in the file are +executed (using the command \fIcl < script\fR. This script generally +sets default parameters. +.le +.ls (3) +If the review flag is set the task \fBeparam\fR is run to allow the user +to examine and modify the parameters for the package \fBccdred\fR and task +\fBccdproc\fR. +.le +.ih +EXAMPLES +1. To get a list of the instruments; + +.nf + cl> setinstrument ? + [List of instruments] + +2. To set the instrument and edit the processing parameters: + + cl> setinstrument ccdlink + [Edit CCDRED parameters] + [Edit CCDPROC parameters] + +3. To use your own instrument translation file and/or setup script in +your working directory. + + cl> setinst.site="" + cl> setinst.dir="" + cl> setinst myinstrument + +To make these files see help under \fBinstruments\fR. Copying and modifying +system files is also straightforward. + + cl> copy ccddb$kpno/fits.dat . + cl> edit fits.dat + cl> setinst.site="" + cl> setinst.dir="" + cl> setinst fits +.fi +.ih +SEE ALSO +instruments, ccdred, ccdproc +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/subsets.hlp b/noao/imred/quadred/src/quad/doc/subsets.hlp new file mode 100644 index 00000000..2823bd6d --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/subsets.hlp @@ -0,0 +1,97 @@ +.help subsets Jun87 noao.imred.ccdred +.ih +NAME +subsets -- Description of CCD subsets +.ih +DESCRIPTION +The \fBccdred\fR package groups observation into subsets. +The image header parameter used to identify the subsets is defined +in the instrument translation file (see help for \fBinstruments\fR). +For example to select subsets by the header parameter "filters" the +instrument translation file would contain the line: + + subset filters + +Observations are generally grouped into subsets based on a common +instrument configuration such as a filter, aperture mask, +grating setting, etc. This allows combining images from several +different subsets automatically and applying the appropriate +flat field image when processing the observations. For example +if the subsets are by filter then \fBflatcombine\fR will search +through all the images, find the flat field images (based on the +CCD type parameter), and combine the flat field images from +each filter separately. Then when processing the images the +flat field with the same filter as the observation is used. + +Each subset is assigned a short identifier. This is listed when +using \fBccdlist\fR and is appended to a root name when combining +images. Because the subset parameter in the image header may be +any string there must be a mapping applied to generate unique +identifiers. This mapping is defined in the file given by +the package parameter \fIccdred.ssfile\fR. The file consists of +lines with two fields: + + 'subset string' subset_id + +where the subset string is the image header string and the subset_id is +the identifier. A field must be quoted if it contains blanks. The +user may create this file but generally it is created by the tasks. The +tasks use the first word of the subset string as the default identifier +and a number is appended if the first word is not unique. The +following steps define the subset identifier: + +.ls (1) +Search the subset file, if present, for a matching subset string and +use the defined subset identifier. +.le +.ls (2) +If there is no matching subset string use the first word of the +image header subset string and, if it is not unique, +add successive integers until it is unique. +.le +.ls (3) +If the identifier is not in the subset file create the file and add an +entry if necessary. +.le +.ih +EXAMPLES +1. The subset file is "subsets" (the default). The subset parameter is +translated to "f1pos" in the image header (the old NOAO CCD parameter) +which is an integer filter position. After running a task, say +"ccdlist *.imh" to cause all filters to be checked, the subset file contains: + +.nf + '2' 2 + '5' 5 + '3' 3 +.fi + +The order reflects the order in which the filters were encountered. +Suppose the user wants to have more descriptive names then the subset +file can be created or edited to the form: + +.nf + '2' U + '3' B + '4' V +.fi + +(This is only an example and does not mean these are standard filters.) + +2. As another example suppose the image header parameter is "filter" and +contains more descriptive strings. The subset file might become: + +.nf + 'GG 385 Filter' GG + 'GG 495 Filter' GG1 + 'RG 610 Filter' RG + 'H-ALPHA' H_ALPHA +.fi + +In this case use of the first word was not very good but it is unique. +It is better if the filters are encoded with the thought that the first +word will be used by \fBccdred\fR; it should be short and unique. +.ih +SEE ALSO +instruments +.endhelp diff --git a/noao/imred/quadred/src/quad/doc/zerocombine.hlp b/noao/imred/quadred/src/quad/doc/zerocombine.hlp new file mode 100644 index 00000000..ad1b021d --- /dev/null +++ b/noao/imred/quadred/src/quad/doc/zerocombine.hlp @@ -0,0 +1,127 @@ +.help zerocombine Sep93 arcon.quad +.ih +NAME +zerocombine -- Combine and process zero level images +.ih +USAGE +zerocombine input +.ih +PARAMETERS +.ls input +List of zero level images to combine. The \fIccdtype\fR parameter +may be used to select the zero level images from a list containing all +types of data. +.le +.ls output = "Zero" +Output zero level root image name. +.le +.ls combine = "average" (average|median) +Type of combining operation performed on the final set of pixels (after +rejection). The choices are +"average" or "median". The median uses the average of the two central +values when the number of pixels is even. +.le +.ls reject = "minmax" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip) +Type of rejection operation. See \fBcombine\fR for details. +.le +.ls ccdtype = "zero" +CCD image type to combine. If no image type is given then all input images +are combined. +.le +.ls process = no +Process the input images before combining? +.le +.ls delete = no +Delete input images after combining? Only those images combined are deleted. +.le +.ls clobber = no +Clobber existing output images? +.le +.ls scale = "none" (none|mode|median|mean|exposure) +Multiplicative image scaling to be applied. The choices are none, scale +by the mode, median, or mean of the specified statistics section, or scale +by the exposure time given in the image header. +.le +.ls statsec = "" +Section of images to use in computing image statistics for scaling. +If no section is given then the entire region of the image is +sampled (for efficiency the images are sampled if they are big enough). +.le + +.ce +Algorithm Parameters +.ls nlow = 0, nhigh = 1 (minmax) +The number of low and high pixels to be rejected by the "minmax" algorithm. +.le +.ls nkeep = 1 +The minimum number of pixels to retain or the maximum number to reject +when using the clipping algorithms (ccdclip, crreject, sigclip, +avsigclip, or pclip). When given as a positive value this is the minimum +number to keep. When given as a negative value the absolute value is +the maximum number to reject. This is actually converted to a number +to keep by adding it to the number of images. +.le +.ls mclip = yes (ccdclip, crreject, sigclip, avsigcliip) +Use the median as the estimate for the true intensity rather than the +average with high and low values excluded in the "ccdclip", "crreject", +"sigclip", and "avsigclip" algorithms? The median is a better estimator +in the presence of data which one wants to reject than the average. +However, computing the median is slower than the average. +.le +.ls lsigma = 3., hsigma = 3. (ccdclip, crreject, sigclip, avsigclip, pclip) +Low and high sigma clipping factors for the "ccdclip", "crreject", "sigclip", +"avsigclip", and "pclip" algorithms. They multiply a "sigma" factor +produced by the algorithm to select a point below and above the average or +median value for rejecting pixels. The lower sigma is ignored for the +"crreject" algorithm. +.le +.ls rdnoise = "0.", gain = "1.", snoise = "0." (ccdclip, crreject) +CCD readout noise in electrons, gain in electrons/DN, and sensitivity noise +as a fraction. These parameters are used with the "ccdclip" and "crreject" +algorithms. The values may be either numeric or an image header keyword +which contains the value. +.le +.ls pclip = -0.5 (pclip) +Percentile clipping algorithm parameter. If greater than +one in absolute value then it specifies a number of pixels above or +below the median to use for computing the clipping sigma. If less +than one in absolute value then it specifies the fraction of the pixels +above or below the median to use. A positive value selects a point +above the median and a negative value selects a point below the median. +The default of -0.5 selects approximately the quartile point. +See \fBcombine\fR for further details. +.le +.ls blank = 0. +Output value to be used when there are no pixels. +.le +.ih +DESCRIPTION +The zero level images in the input image list are combined. +The input images may be processed first if desired. +The original images may be deleted automatically if desired. + +This task is a script which applies \fBquadproc\fR and \fBcombine\fR. The +parameters and combining algorithms are described in detail in the help for +\fBcombine\fR. This script has default parameters specifically set for +zero level images and simplifies the combining parameters. There are other +combining options not included in this task. For these additional +features, such as thresholding, offseting, masking, and projecting, use +\fBcombine\fR. + +The version of \fBzerocombine\fR in the \fBquad\fR package differs from that +in \fBccdred\fR in that \fBquadproc\fR rather than \fBccdproc\fR is used to +process the images if this is requested. The \fBquad\fR version MUST be +used if process=yes and the input list contains any multi-readout images which +have not been overscan corrected and trimmed. + +.ih +EXAMPLES +1. The image data contains four zero level images. +To automatically select them and combine them as a background job +using the default combining algorithm: + + cl> zerocombine ccd*.imh& +.ih +SEE ALSO +quadproc, combine +.endhelp |