diff options
Diffstat (limited to 'noao/twodspec/apextract/doc')
26 files changed, 7651 insertions, 0 deletions
diff --git a/noao/twodspec/apextract/doc/apall.hlp b/noao/twodspec/apextract/doc/apall.hlp new file mode 100644 index 00000000..c4e50072 --- /dev/null +++ b/noao/twodspec/apextract/doc/apall.hlp @@ -0,0 +1,557 @@ +.help apall Sep96 noao.twodspec.apextract +.ih +NAME +apall -- Extract one dimensional sums across the apertures +.ih +USAGE +apall input +.ih +PARAMETERS +.ls input +List of input images. +.le +.ls output = "" +List of output root names for extracted spectra. If the null +string is given or the end of the output list is reached before the end +of the input list then the input image name is used as the output root name. +This will not conflict with the input image since an aperture number +extension is added for onedspec format, the extension ".ms" for multispec +format, or the extension ".ec" for echelle format. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls format = "multispec" (onedspec|multispec|echelle|strip) +Format for output extracted spectra. "Onedspec" format extracts each +aperture to a separate image while "multispec" and "echelle" extract +multiple apertures for the same image to a single output image. +The "multispec" and "echelle" format selections differ only in the +extension added. The "strip" format produces a separate 2D image in +which each column or line along the dispersion axis is shifted to +exactly align the aperture based on the trace information. +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +Input images without/with a database entry are skipped silently. +.le +.ls profiles = "" +List of profile images for variance weighting or cleanning. If variance +weighting or cleanning a profile of each aperture is computed from the +input image unless a profile image is specified, in which case the +profile is computed from the profile image. The profile image must +have the same dimensions and dispersion and it is assumed that the +spectra have the same position and profile shape as in the object +spectra. Use of a profile image is generally not required even for +faint input spectra but the option is available for those who wish +to use it. +.le + +.ce +PROCESSING PARAMETERS +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing, trace +fitting, and extraction review are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le +.ls extract = yes +Extract the one dimensional aperture sums? +.le +.ls extras = yes +Extract the raw spectrum (if variance weighting is used), the sky spectrum +(if background subtraction is used), and sigma spectrum (if variance +weighting is used)? This information is extracted to the third dimension +of the output image. +.le +.ls review = yes +Review the extracted spectra? The \fIinteractive\fR parameter must also be +yes. +.le + +.ls line = INDEF, nsum = 10 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. A line of INDEF selects the middle of the +image along the dispersion axis. A positive nsum selects a sum of +lines and a negative selects a median of lines. +.le + +.ce +DEFAULT APERTURE PARAMETERS +.ls lower = -5., upper = 5. +Default lower and upper aperture limits relative to the aperture center. +These limits are used for apertures found with \fBapfind\fR and when +defining the first aperture in \fBapedit\fR. +.le +.ls apidtable = "" +Aperture identification table. This may be either a text file or an +image. A text file consisting of lines with an aperture number, beam +number, and aperture title or identification. An image will contain the +keywords SLFIBnnn with string value consisting of aperture number, beam +number, optional right ascension and declination, and aperture title. This +information is used to assign aperture information automatically in +\fBapfind\fR and \fBapedit\fR. +.le + +.ce +DEFAULT BACKGROUND PARAMETERS +.ls b_function = "chebyshev" +Default background fitting function. The fitting function types are +"chebyshev" polynomial, "legendre" polynomial, "spline1" linear spline, and +"spline3" cubic spline. +.le +.ls b_order = 1 +Default background function order. The order refers to the number of +terms in the polynomial functions or the number of spline pieces in the spline +functions. +.le +.ls b_sample = "-10:-6,6:10" +Default background sample. The sample is given by a set of colon separated +ranges each separated by either whitespace or commas. The string "*" refers +to all points. Note that the background coordinates are relative to the +aperture center and not image pixel coordinates so the endpoints need not +be integer. +.le +.ls b_naverage = -3 +Default number of points to average or median. Positive numbers +average that number of sequential points to form a fitting point. +Negative numbers median that number, in absolute value, of sequential +points. A value of 1 does no averaging and each data point is used in the +fit. +.le +.ls b_niterate = 0 +Default number of rejection iterations. If greater than zero the fit is +used to detect deviant fitting points and reject them before repeating the +fit. The number of iterations of this process is given by this parameter. +.le +.ls b_low_reject = 3., b_high_reject = 3. +Default background lower and upper rejection sigmas. If greater than zero +points deviating from the fit below and above the fit by more than this +number of times the sigma of the residuals are rejected before refitting. +.le +.ls b_grow = 0. +Default reject growing radius. Points within a distance given by this +parameter of any rejected point are also rejected. +.le + +.ce +APERTURE CENTERING PARAMETERS +.ls width = 5. +Width of spectrum profiles. This parameter is used for the profile +centering algorithm in this and other tasks. +.le +.ls radius = 10. +The profile centering error radius for the centering algorithm. +.le +.ls threshold = 0. +Centering threshold for the centering algorithm. The range of pixel intensities +near the initial centering position must exceed this threshold. +.le + +.ce +AUTOMATIC FINDING AND ORDERING PARAMETERS +.ls nfind +Maximum number of apertures to be defined. This is a query parameter +so the user is queried for a value except when given explicitly on +the command line. +.le +.ls minsep = 5. +Minimum separation between spectra. Weaker spectra or noise within this +distance of a stronger spectrum are rejected. +.le +.ls maxsep = 1000. +Maximum separation between adjacent spectra. This parameter +is used to identify missing spectra in uniformly spaced spectra produced +by fiber spectrographs. If two adjacent spectra exceed this separation +then it is assumed that a spectrum is missing and the aperture identification +assignments will be adjusted accordingly. +.le +.ls order = "increasing" +When assigning aperture identifications order the spectra "increasing" +or "decreasing" with increasing pixel position (left-to-right or +right-to-left in a cross-section plot of the image). +.le + +.ce +RECENTERING PARAMETERS +.ls aprecenter = "" +List of apertures to be used in shift calculation. +.le +.ls npeaks = INDEF +Select the specified number of apertures with the highest peak values +to be recentered. If the number is INDEF all apertures will be selected. +If the value is less than 1 then the value is interpreted as a fraction +of total number of apertures. +.le +.ls shift = yes +Use the average shift from recentering the apertures selected by the +\fIaprecenter\fR parameter to apply to the apertures selected by the +\fIapertures\fR parameter. The recentering is then a constant shift for +all apertures. +.le + +.ce +RESIZING PARAMETERS +.ls llimit = INDEF, ulimit = INDEF +Lower and upper aperture size limits. If the parameter \fIylevel\fR is +INDEF then these limits are assigned to all apertures. Otherwise +these parameters are used as limits to the resizing operation. +A value of INDEF places the aperture limits at the image edge (for the +dispersion line used). +.le +.ls ylevel = 0.1 +Data level at which to set aperture limits. If it is INDEF then the +aperture limits are set at the values given by the parameters +\fIllimit\fR and \fIulimit\fR. If it is not INDEF then it is a +fraction of the peak or an actual data level depending on the parameter +\fIpeak\fR. It may be relative to a local background or to zero +depending on the parameter \fIbkg\fR. +.le +.ls peak = yes +Is the data level specified by \fIylevel\fR a fraction of the peak? +.le +.ls bkg = yes +Subtract a simple background when interpreting the \fBylevel\fR parameter. +The background is a slope connecting the first inflection points +away from the aperture center. +.le +.ls r_grow = 0. +Change the lower and upper aperture limits by this fractional amount. +The factor is multiplied by each limit and the result added to limit. +.le +.ls avglimits = no +Apply the average lower and upper aperture limits to all apertures. +.le + +.ce +TRACING PARAMETERS +.ls t_nsum = 10 +Number of dispersion lines to be summed at each step along the dispersion. +.le +.ls t_step = 10 +Step along the dispersion axis between determination of the spectrum +positions. +.le +.ls t_nlost = 3 +Number of consecutive steps in which the profile is lost before quitting +the tracing in one direction. To force tracing to continue through +regions of very low signal this parameter can be made large. Note, +however, that noise may drag the trace away before it recovers. +.le +.ls t_function = "legendre" +Default trace fitting function. The fitting function types are +"chebyshev" polynomial, "legendre" polynomial, "spline1" linear spline, and +"spline3" cubic spline. +.le +.ls t_order = 2 +Default trace function order. The order refers to the number of +terms in the polynomial functions or the number of spline pieces in the spline +functions. +.le +.ls t_sample = "*" +Default fitting sample. The sample is given by a set of colon separated +ranges each separated by either whitespace or commas. The string "*" refers +to all points. +.le +.ls t_naverage = 1 +Default number of points to average or median. Positive numbers +average that number of sequential points to form a fitting point. +Negative numbers median that number, in absolute value, of sequential +points. A value of 1 does no averaging and each data point is used in the +.le +.ls t_niterate = 0 +Default number of rejection iterations. If greater than zero the fit is +used to detect deviant traced positions and reject them before repeating the +fit. The number of iterations of this process is given by this parameter. +.le +.ls t_low_reject = 3., t_high_reject = 3. +Default lower and upper rejection sigma. If greater than zero traced +points deviating from the fit below and above the fit by more than this +number of times the sigma of the residuals are rejected before refitting. +.le +.ls t_grow = 0. +Default reject growing radius. Traced points within a distance given by this +parameter of any rejected point are also rejected. +.le + +.ce +EXTRACTION PARAMETERS +.ls background = "none" (none|average|median|minimum|fit) +Type of background subtraction. The choices are "none" for no background +subtraction, "average" to average the background within the background +regions, "median" to use the median in the background regions, "minimum" to +use the minimum in the background regions, or "fit" to fit across the +dispersion using the background within the background regions. Note that +the "average" option does not do any medianing or bad pixel checking, +something which is recommended. The fitting option is slower than the +other options and requires additional fitting parameter. +.le +.ls skybox = 1 +Box car smoothing length for sky background when using background +subtraction. Since the background noise is often the limiting factor +for good extraction one may box car smooth the sky to improve the +statistics in smooth background regions at the expense of distorting +the subtraction near spectral features. This is most appropriate when +the sky regions are limited due to a small slit length. +.le +.ls weights = "none" (none|variance) +Type of extraction weighting. Note that if the \fIclean\fR parameter is +set then the weights used are "variance" regardless of the weights +specified by this parameter. The choices are: +.ls "none" +The pixels are summed without weights except for partial pixels at the +ends. +.le +.ls "variance" +The extraction is weighted by the variance based on the data values +and a poisson/ccd model using the \fIgain\fR and \fIreadnoise\fR +parameters. +.le +.le +.ls pfit = "fit1d" (fit1d|fit2d) +Profile fitting algorithm to use with variance weighting or cleaning. +When determining a profile the two dimensional spectrum is divided by +an estimate of the one dimensional spectrum to form a normalized two +dimensional spectrum profile. This profile is then smoothed by fitting +one dimensional functions, "fit1d", along the lines or columns most closely +corresponding to the dispersion axis or a special two dimensional +function, "fit2d", described by Marsh (see \fBapprofile\fR). +.le +.ls clean = no +Detect and replace deviant pixels? +.le +.ls saturation = INDEF +Saturation or nonlinearity level in data units. During variance weighted +extractions wavelength points having any pixels above this value are +excluded from the profile determination and the sigma spectrum extraction +output, if selected by the \fIextras\fR parameter, flags wavelengths with +saturated pixels with a negative sigma. +.le +.ls readnoise = 0. +Read out noise in photons. This parameter defines the minimum noise +sigma. It is defined in terms of photons (or electrons) and scales +to the data values through the gain parameter. A image header keyword +(case insensitive) may be specified to get the value from the image. +.le +.ls gain = 1 +Detector gain or conversion factor between photons/electrons and +data values. It is specified as the number of photons per data value. +A image header keyword (case insensitive) may be specified to get the value +from the image. +.le +.ls lsigma = 4., usigma = 4. +Lower and upper rejection thresholds, given as a number of times the +estimated sigma of a pixel, for cleaning. +.le +.ls nsubaps = 1 +During extraction it is possible to equally divide the apertures into +this number of subapertures. For multispec format all subapertures will +be in the same file with aperture numbers of 1000*(subap-1)+ap where +subap is the subaperture (1 to nsubaps) and ap is the main aperture +number. For echelle format there will be a separate echelle format +image containing the same subaperture from each order. The name +will have the subaperture number appended. For onedspec format +each subaperture will be in a separate file with extensions and +aperture numbers as in the multispec format. +.le +.ih +ADDITIONAL PARAMETERS +Dispersion axis and I/O parameters are taken from the package parameters. +.ih +DESCRIPTION +This task provides functions for defining, modifying, tracing, and +extracting apertures from two dimensional spectra. The functions +desired are selected using switch parameters. When the task is +run interactively queries are made at each step allowing additional +control of the operations performed on each input image. + +The functions, in the order in which they are generally performed, are +summarized below. +.ls o +Automatically find a specified number of spectra and assign default +apertures. Apertures may also be inherited from another image or +defined using an interactive graphical interface called the \fIaperture +editor\fR. +.le +.ls o +Recenter selected reference apertures on the image spectrum profiles. +.le +.ls o +Resize the selected reference apertures based on spectrum profile width. +.le +.ls o +Interactively define or adjust aperture definitions using a graphical +interface called the \fIaperture editor\fR. All function may also +be performed from this editor and, so, provides an alternative +method of processing and extracting spectra. +.le +.ls o +Trace the positions of the selected spectra profiles from a starting image line +or column to other image lines or columns and fit a smooth function. +The trace function is used to shift the center of the apertures +at each dispersion point in the image. +.le +.ls o +Extract the flux in the selected apertures into one dimensional spectra in +various formats. This includes possible background subtraction, variance +weighting, and bad pixel rejection. +.le + +Each of these functions has different options and parameters. In +addition to selecting any of these functions in this task, they may +also be selected using the aperture editor and as individual +commands (which themselves allow selection of other functions). When +broken down into individual tasks the parameters are also sorted by +their function though there are then some mutual parameter +interdependencies. This functional decomposition is what was available +prior to the addition of the \fBapall\fR task. It is recommended that +this task be used because it collects all the parameters in one +place eliminating confusion over where a particular parameter +is defined. However, documenting the various functions +is better organized in terms of the separate descriptions given for +each of the functions; namely under the help topics +\fBapdefault, apfind, aprecenter, apresize, apedit, +aptrace\fR, and \fBapsum\fR. +.ih +EXAMPLES +1. This example may be executed if desired. First we create an artificial +spectrum with four spectra and a background. After it is created you +can display or plot it. Next we define the dispersion axis and set the +verbose flag to better illustrate what is happening. The task APALL +is run with the default parameters except for background fitting and +subtracting added. The text beginning with # are comments of things to +try and do. + +.nf + ap> artdata + ar> unlearn artdata + ar> mk1dspec apdemo1d nl=50 + ar> mk2dspec apdemo2d model=STDIN + apdemo1d 1. gauss 3 0 20 .01 + apdemo1d .8 gauss 3 0 40 .01 + apdemo1d .6 gauss 3 0 60 .01 + apdemo1d .4 gauss 3 0 80 .01 + [EOF=Control D or Control Z] + ar> mknoise apdemo2d background=100. rdnoise=3. poisson+ + ar> bye + # Display or plot the spectrum + ap> dispaxis=2; verbose=yes + ap> unlearn apall + ap> apall apdemo2d back=fit + Searching aperture database ... + Find apertures for apdemo2d? (yes): + Finding apertures ... + Number of apertures to be found automatically (1): 4 + Jul 31 16:55: FIND - 4 apertures found for apdemo2d. + Resize apertures for apdemo2d? (yes): + Resizing apertures ... + Jul 31 16:55: RESIZE - 4 apertures resized for apdemo2d. + Edit apertures for apdemo2d? (yes): + # Get a list of commands with '?' + # See all the parameters settings with :par + # Try deleting and marking a spectrum with 'd' and 'm' + # Look at the background fitting parameters with 'b' (exit with 'q') + # Exit with 'q' + Trace apertures for apdemo2d? (yes): + Fit traced positions for apdemo2d interactively? (yes): + Tracing apertures ... + Fit curve to aperture 1 of apdemo2d interactively (yes): + # You can use ICFIT commands to adjust the fit. + Fit curve to aperture 2 of apdemo2d interactively (yes): n + Fit curve to aperture 3 of apdemo2d interactively (no): + Fit curve to aperture 4 of apdemo2d interactively (no): y + Jul 31 16:56: TRACE - 4 apertures traced in apdemo2d. + Write apertures for apdemo2d to apdemosdb (yes): + Jul 31 16:56: DATABASE - 4 apertures for apdemo2d written to database. + Extract aperture spectra for apdemo2d? (yes): + Review extracted spectra from apdemo2d? (yes): + Extracting apertures ... + Review extracted spectrum for aperture 1 from apdemo2d? (yes): + # Type 'q' to quit + Jul 31 16:56: EXTRACT - Aperture 1 from apdemo2d --> apdemo2d.ms + Review extracted spectrum for aperture 2 from apdemo2d? (yes): N + Jul 31 16:56: EXTRACT - Aperture 2 from apdemo2d --> apdemo2d.ms + Jul 31 16:56: EXTRACT - Aperture 3 from apdemo2d --> apdemo2d.ms + Jul 31 16:57: EXTRACT - Aperture 4 from apdemo2d --> apdemo2d.ms +.fi + +2. To extract a series of similar spectra noninteractively using a +reference for the aperture definitions, then recentering and resizing +but not retracing: + +.nf + ap> apall fib*.imh ref=flat inter- trace- +.fi + +Note that the interactive flag automatically turns off the edit, fittrace, +and review options and the reference image eliminates the find +(find only occurs if there are no initial apertures). +.ih +REVISIONS +.ls APALL V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". + +The aperture ID table information may now be contained in the +image header under the keywords SLFIBnnn. + +The "nsubaps" parameter now allows onedspec and echelle output formats. +The echelle format is appropriate for treating each subaperture as +a full echelle extraction. +.le +.ls APALL V2.10.3 +The dispersion axis parameter was moved to purely a package parameter. + +As a final step when computing a weighted/cleaned spectrum the total +fluxes from the weighted spectrum and the simple unweighted spectrum +(excluding any deviant and saturated pixels) are computed and a +"bias" factor of the ratio of the two fluxes is multiplied into +the weighted spectrum and the sigma estimate. This makes the total +fluxes the same. In this version the bias factor is recorded in the logfile +if one is kept. Also a check is made for unusual bias factors. +If the two fluxes disagree by more than a factor of two a warning +is given on the standard output and the logfile with the individual +total fluxes as well as the bias factor. If the bias factor is +negative a warning is also given and no bias factor is applied. +In the previous version a negative (inverted) spectrum would result. +.le +.ih +SEE ALSO +apdefault, apfind, aprecenter, apresize, apedit, aptrace, apsum +.endhelp diff --git a/noao/twodspec/apextract/doc/apbackground.hlp b/noao/twodspec/apextract/doc/apbackground.hlp new file mode 100644 index 00000000..93a49e42 --- /dev/null +++ b/noao/twodspec/apextract/doc/apbackground.hlp @@ -0,0 +1,79 @@ +.help apbackground Aug90 noao.twodspec.apextract + +.ce +Background Determination + + +Data from slit spectra allow the determination and subtraction +of the background sky using information from regions near the object +of interest. Background subtraction may also apply to cases of +scattered light though other techniques for scattered light removal +may be more appropriate. The APEXTRACT package provides for determining +the background level at each wavelength (line or column along the dispersion +axis) from a set of regions and extrapolating and subtracting the +background at each pixel extracted from the object profile. The +type of background used during extraction is specified by the parameter +\fIbackground\fR. If the value "none" is used then no background is +subtracted and any background parameters defined for an aperture are +ignored. If the value is "average", "median", "minimum" or "fit" then a +background is determined, including a variance estimate when using variance +weighted extraction (see \fIapvariance\fR), and the subtracted background +spectrum may be output if the \fIextras\fR parameter is set. + +The basic aperture definition structure used in the APEXTRACT package +includes associated background regions and fitting parameters. The +background regions are specified by a list of colon delimited ranges +defined relative to the center of the aperture. There are generally +two ranges, one on each side of the object, though one sided or more +complex sets may be used to avoid contaminated or missing parts +of the slit. The default ranges are defined by the parameter +\fIb_sample\fR. Often the ranges are better set graphically using a +cursor by invoking the 'b' option of the aperture editor. + +If the background type is "average", "median", or "minimum" then pixels +occupying these regions are averaged, medianed, or the minimum found to +produce a single background level for all object pixels at each wavelength. +Note that the "average" choice does not exclude any pixels which may +yield a background contaminated by cosmic rays. The "median" or "minimum" +is recommended instead. + +If the background type is "fit" then a function is fit to the pixels in the +background regions using the ICFIT options (see \fBicfit\fR). The +parameter \fIb_naverage\fR may be used to compute averages or medians of +groups or all of the points within each sample region. The fit is defined +by a function type \fIb_function\fR; one of legendre polynomial, chebyshev +polynomial, linear spline, or cubic spline, and function order +\fIb_order\fR (number of polynomial terms or spline pieces). An +interactive rejection of grossly deviant points from the fit may also be +used. The fitted function can define a constant, sloped, or higher order +background for the object pixels. + +Note that the background setting function, the 'b' key in \fBapedit\fR, +may be used to set the background regions for all the background options +but it will always show the result of a fit regardless of the background +type. + +After determining a background by averaging, medianing, minimizing, or +fitting, a box car smoothing step may be applied. The box car size is +given by the parameter \fIskybox\fR. When the number of available +background pixels is small, due to a small slit for instance, the noise +introduced to the extracted object spectrum may be unsatisfactorily large. +By smoothing the background one can reduce the noise when the background +consists of a smooth continuum. The trade-off, however, is that near sharp +features the smoothing will smear the features out and give a poorer +subtraction of these features. One could extract both the object and +background separately and apply a background smoothing separately using +other image processing tools. However, this is not possible for variance +weighted extraction because of the intimate connection between the +background levels, the profile determination, and the variance estimates +based on both. Thus, this smoothing feature is included. + +The background determined by the methods outlined above is actually +subtracted as a separate step during extraction. The background +is also used during profile fitting when cleaning or using variance +weighted extraction. See \fBapvariance\fR and \fBapprofile\fR for +further discussion. +.ih +SEE ALSO +approfile apvariance apdefault icfit apall apsum +.endhelp diff --git a/noao/twodspec/apextract/doc/apdefault.hlp b/noao/twodspec/apextract/doc/apdefault.hlp new file mode 100644 index 00000000..e17fe50d --- /dev/null +++ b/noao/twodspec/apextract/doc/apdefault.hlp @@ -0,0 +1,95 @@ +.help apdefault Jul95 noao.twodspec.apextract +.ih +NAME +apdefault -- Set default aperture parameters for the package +.ih +USAGE +apdefault +.ih +PARAMETERS +.ls lower = -5., upper = 5. +Default lower and upper aperture limits relative to the aperture center. +These limits are used for apertures found with \fBapfind\fR and when +defining the first aperture in \fBapedit\fR. +.le +.ls apidtable = "" +Aperture identification table. This may be either a text file or an +image. A text file consisting of lines with an aperture number, beam +number, and aperture title or identification. An image will contain the +keywords SLFIBnnn with string value consisting of aperture number, beam +number, optional right ascension and declination, and aperture title. This +information is used to assign aperture information automatically in +\fBapfind\fR and \fBapedit\fR. +.le + +.ce +Default Background Subtraction Parameters +.ls b_function = "chebyshev" +Default background fitting function. The fitting function types are +"chebyshev" polynomial, "legendre" polynomial, "spline1" linear spline, and +"spline3" cubic spline. +.le +.ls b_order = 1 +Default background function order. The order refers to the number of +terms in the polynomial functions or the number of spline pieces in the spline +functions. +.le +.ls b_sample = "-10:-6,6:10" +Default background sample. The sample is given by a set of colon separated +ranges each separated by either whitespace or commas. The string "*" refers +to all points. Note that the background coordinates are relative to the +aperture center and not image pixel coordinates so the endpoints need not +be integer. +.le +.ls b_naverage = -3 +Default number of points to average or median. Positive numbers +average that number of sequential points to form a fitting point. +Negative numbers median that number, in absolute value, of sequential +points. A value of 1 does no averaging and each data point is used in the +fit. +.le +.ls b_niterate = 0 +Default number of rejection iterations. If greater than zero the fit is +used to detect deviant fitting points and reject them before repeating the +fit. The number of iterations of this process is given by this parameter. +.le +.ls b_low_reject = 3., b_high_reject = 3. +Default background lower and upper rejection sigmas. If greater than zero +points deviating from the fit below and above the fit by more than this +number of times the sigma of the residuals are rejected before refitting. +.le +.ls b_grow = 0. +Default reject growing radius. Points within a distance given by this +parameter of any rejected point are also rejected. +.le +.ih +DESCRIPTION +This task sets the values of the default aperture parameters for the +tasks \fBapedit\fR and \fBapfind\fR which define new apertures. For a +description of the components of an aperture see the paper \fBThe +APEXTRACT Package\fR. In \fBapedit\fR the default aperture limits and +background parameters are only used if there are no other +apertures defined. The aperture identification table is used when +reordering the apertures with the 'o' key. When run the parameters are +displayed and modified using the \fBeparam\fR task. + +The aperture limits and background fitting sample regions are defined +relative to the center of the aperture. The background fitting parameters +are those used by the ICFIT package. They may be modified interactively +with the 'b' key in the task \fBapedit\fR. For more on background fitting +and subtracting see \fBapbackground\fR. +.ih +EXAMPLES +To review and modify the default aperture parameters: + + cl> apdefault +.ih +.ih +REVISIONS +.ls APDEFAULT V2.11 +The aperture ID table information may now be contained in the +image header under the keywords SLFIBnnn. +.le +SEE ALSO +apbackground, apedit, apfind, icfit +.endhelp diff --git a/noao/twodspec/apextract/doc/apedit.hlp b/noao/twodspec/apextract/doc/apedit.hlp new file mode 100644 index 00000000..324f6e5d --- /dev/null +++ b/noao/twodspec/apextract/doc/apedit.hlp @@ -0,0 +1,374 @@ +.help apedit Sep96 noao.twodspec.apextract +.ih +NAME +apedit -- Edit apertures +.ih +USAGE +apedit input +.ih +PARAMETERS +.ls input +List of input images for which apertures are to be edited. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +If the reference image list is shorter than the input image list the +last reference image is used for the remaining input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = no +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing is +disabled. +.le +.ls find = no +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = no +Recenter the apertures? +.le +.ls resize = no +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le + +.ls line = INDEF +The dispersion line (line or column perpendicular to the dispersion axis) to +be graphed. A value of INDEF uses the middle of the image. +.le +.ls nsum = 10 +Number of dispersion lines to be summed or medianed. The lines are taken +around the specified dispersion line. A positive nsum selects a sum of +lines and a negative selects a median of lines. +.le +.ls width = 5. +Width of spectrum profiles. This parameter is used for the profile +centering algorithm in this and other tasks. +.le +.ls radius = 5. +The profile centering error radius for the centering algorithm. +.le +.ls threshold = 0. +Centering threshold for the centering algorithm. The range of pixel intensities +near the initial centering position must exceed this threshold. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters are taken from the +task \fBapdefault\fR. Parameters for the various functions of finding, +recentering, and resizing are taken from the parameters for the +appropriate task. + +When this operation is performed from the task \fBapall\fR all parameters +except the package parameters are included in that task. +.ih +CURSOR KEYS +When editing the apertures interactively the following cursor keys are +available. + +.nf +? Print help +a Toggle the ALL flag +b an Set background fitting parameters +c an Center aperture(s) +d an Delete aperture(s) +e an Extract spectra (see APSUM) +f Find apertures up to the requested number (see APFIND) +g an Recenter aperture(s) (see APRECENTER) +i n Set aperture ID +j n Set aperture beam number +l ac Set lower limit of current aperture at cursor position +m Define and center a new aperture on the profile near the cursor +n Define a new aperture centered at the cursor +o n Enter desired aperture number for cursor selected aperture and + remaining apertures are reordered using apidtable and maxsep + parameters (see APFIND for ordering algorithm) +q Quit +r Redraw the graph +s an Shift the center(s) of the current aperture to the cursor + position +t ac Trace aperture positions (see APTRACE) +u ac Set upper limit of current aperture at cursor position +w Window the graph using the window cursor keys +y an Set aperture limits to intercept the data at the cursor y + position +z an Resize aperture(s) (see APRESIZE) +. n Select the aperture nearest the cursor for current aperture ++ c Select the next aperture (in ID) to be the current aperture +- c Select the previous aperture (in ID) to be the current aperture +I Interrupt task immediately. Database information is not saved. +.fi + +The letter a following the key indicates if all apertures are affected when +the ALL flag is set. The letter c indicates that the key affects the +current aperture while the letter n indicates that the key affects the +aperture whose center is nearest the cursor. +.ih +COLON COMMANDS + +.nf +:show [file] Print a list of the apertures (default STDOUT) +:parameters [file] Print current parameter values (default STDOUT) +:read [name] Read from database (default current image) +:write [name] Write to database (default current image) +.fi + +The remaining colon commands are task parameters and print the current +value if no value is given or reset the current value to that specified. +Use :parameters to see current parameter values. + +.nf +:apertures :apidtable :avglimits :b_function +:b_grow :b_high_reject :b_low_reject :b_naverage +:b_niterate :b_order :b_sample :background +:bkg :center :clean :database +:extras :gain :image :line +:llimit :logfile :lower :lsigma +:maxsep :minsep :npeaks :nsubaps +:nsum :order :parameters :peak +:plotfile :r_grow :radius :read +:readnoise :saturation :shift :show +:skybox :t_function :t_grow :t_high_reject +:t_low_reject :t_naverage :t_niterate :t_nsum +:t_order :t_sample :t_step :t_width +:threshold :title :ulimit :upper +:usigma :weights :width :write +:ylevel :t_nlost +.fi +.ih +DESCRIPTION +For each image in the input image list, apertures are defined and edited +interactively. The aperture editor is invoked when the parameters +\fIinteractive\fR and \fIedit\fR are both yes. When this is the case +the task will query whether to edit each image. The responses are +"yes", "no", "YES", and "NO", where the upper case responses suppress +queries for all following images. + +When the aperture editor is entered a graph of the image lines or +columns specified by the parameters \fIline\fR and \fInsum\fR is +drawn. In the \fBapextract\fR package a dispersion line is either a +line or column in the image at one point along the dispersion axis. +The dispersion axis may be defined in the image header under the +keyword DISPAXIS or by the package parameter \fIdispaxis\fR. The +parameter \fBnsum\fR determines how many dispersion lines surrounding +the specified dispersion line are summed or medianed. This improves the +signal in the profiles of weaker spectra. Once the graph is drawn an +interactive cursor loop is entered. The set of cursor keys and colon +commands is given above and may be printed when the task is running using +the '?' key. The CURSOR MODE keys and graph formatting options are also +available (see \fBcursor\fR and \fBgtools\fR). + +A status line, usually at the bottom of the graphics terminal, +indicates the current aperture and shows the ALL flag, 'a' key, if set. The +concept of the current aperture is used by several of the aperture +editing commands. Other commands operate on the aperture whose center +is nearest the cursor. It is important to know which commands operate +on the current aperture and which operate on the nearest aperture to +the cursor. + +The cursor keys and colon commands are used to define new apertures, +delete existing apertures, modify the aperture number, beam number, +title, center, and limits, set background fitting parameters, trace the +positions of the spectra in the apertures, and extract aperture +spectra. When creating new apertures default parameters are supplied +in two ways; if no apertures are defined then the default parameters +are taken from the task \fBapdefault\fR while if there is a current +aperture then a copy of its parameters are made. + +The keys for creating a new aperture are 'm' and 'n' and 'f'. The key +'m' marks a new aperture and centers the aperture on the profile +nearest the cursor. The centering algorithm is described under the +help topic \fBcenter1d\fR and the parameters controlling the centering are +\fIwidth\fR, \fIradius\fR, and \fIthreshold\fR. The key 'n' defines a +new aperture at the position of the cursor without centering. This is +used if there is no spectrum profile such as when defining sky apertures +or when defining apertures in extended profiles. The 'f' key finds new +apertures using the algorithm described in the task \fBapfind\fR. The +number of apertures found in this way is limited by the parameter +\fBnfind\fR and the number includes any previously defined +apertures. The new aperture number, beam number, and title are assigned using +the aperture assignment algorithm described in \fBapfind\fR. + +The aperture number for the aperture \fInearest\fR the cursor is changed +with the 'j' key and the beam number is changed with the 'k' key. The +user is prompted for a new aperture number or beam number. The +aperture title may be set or changed with the :title colon command. + +The 'o' key may be used to reorder or correct the aperture +identifications and beam numbers. This is useful if the aperture +numbers become disordered due to deletions and additions or if the +first spectrum is missing when using the automatic identification +algorithm. An aperture number is requested for the aperture pointed to +by the cursor. The remaining apertures are reordered relative to this +aperture number. There is a aperture number, beam number, and title +assignment algorithm which uses information about the maximum +separation between consecutive apertures, the direction of increasing +aperture numbers, and an optional aperture identification table. See +\fBapfind\fR for a description of the algorithm. + +After defining a new aperture it becomes the current aperture. The +current aperture is indicated on the status line and the '.', '+', and +'-' keys are used to select a new current aperture. + +Apertures are deleted with 'd' key. The aperture \fInearest\fR the +cursor is deleted. + +The aperture center may be changed with the 'c', 's', and 'g' keys and the +":center value" colon command. The 'c' key applies the centering algorithm +to the aperture \fInearest\fR the colon. The 's' key shifts the center +of the \fIcurrent\fR aperture to the position of the cursor. The 'g' +applies the \fBaprecenter\fR algorithm. The :center command sets the +center of the \fIcurrent\fR aperture to the value specified. Except +for the last option these commands may be applied to all apertures +if the ALL flag is set. + +The aperture limits are defined relative to the aperture center. The +limits may be changed with the 'l', 'u', 'y', and 'z' keys and with the +":lower value" and ":upper value" commands. The 'l' and 'u' keys set +the lower and upper limits of the \fIcurrent\fR aperture at the position +of the cursor. The colon commands allow setting the limits explicitly. +The 'y' key defines both limits for the \fInearest\fR aperture as +points at which the y cursor position intercepts the data profile. +This requires that the aperture include a spectrum profile and that +the y cursor value lie below the peak of the profile. The 'z' +key applies the \fBapresize\fR algorithm. Except for the colon +commands these commands may be applied to all apertures if the ALL +flag is set. + +The key 'b' modifies the background fitting parameters for the aperture +\fInearest\fR the cursor. The default background parameters are +specified by the task \fBapdefault\fR. Note that even though +background parameters are defined, background subtraction is not +performed during extraction unless specified. +When the 'b' key is used the \fBicfit\fR graphical interface is entered +showing the background regions and function fit for the current image +line. Note that the background regions are specified relative to +the aperture center and follows changes in the aperture position. + +The two types of +extraction which may be specified are to average all points within +a set of background regions or fit a function to the points in +the background regions. In the first case only the background sample +parameter is used. In the latter case the other parameters are +also used in conjunction with the \fBicfit\fR function fitting commands. +See \fBapbackground\fR for more on the background parameters. + +Each aperture may have different background +fitting parameters but newly defined apertures inherit the background +fitting parameters of the last current aperture. This will usually be +satisfactory since the background regions are defined relative to the +aperture center rather than in absolute coordinates. If the ALL flag +is set then all apertures will be given the same background +parameters. + +The algorithms used in the tasks \fBapfind, aprecenter, apresize, aptrace\fR, +and \fBapsum\fR are available from the editor with the keys 'f', 'g', 'z', +'t', and 'e' +respectively. Excluding finding, if the ALL flag is not set then the +nearest aperture +to the cursor is used. This allows selective recentering, resizing, +tracing and extracting. +If the ALL flag is set then all apertures are traced or extracted. +When extracting the output, rootname and profile name are queried. + +Some general purpose keys window the graph 'w' using the \fBgtools\fR +commands, redraw the graph 'r', and quit 'q'. + +The final cursor key is the 'a' key. The cursor keys which modify the +apertures were defined as operating on either the aperture nearest the +cursor or the current aperture. The 'a' key allows these keys to +affect all the apertures simultaneously. The 'a' key sets a flag which +is shown on the status line when it is set. When set, the operation on +one aperture is duplicated on the remaining apertures. The operations +which apply to all apertures are set background 'b', center 'c', delete +'d', extract 'e', recenter 'g', set lower limit 'l', shift 's', trace +'t', set upper limit 'u', set limits at the y cursor 'y', and resize +'z'. The 'b', 'l', 's', and 'u' keys first set the background, +aperture limits, or shift for the appropriate aperture and then are +applied to the other apertures relative to their centers. + +All the parameters used in any of the operations may be examined or +changed through colon commands. The :parameters command lists all +parameter values and :show lists the apertures. The :read and :write +are used to force an update or save the current apertures and to read +apertures for the current image or from some other image. The commands +all have optional arguments. For the commands which show information +the argument specifies a file to which the information is to be +written. The default is the standard output. The database read and +write and the change image commands take an image name. If an image +name is not given for the read and write commands the +current image name is used. The change image command default is to +print the current image name. The remaining commands take a value. If +a value is not given then the current value is printed. + +The aperture editor may be selected from nearly every task using the +\fBedit\fR parameter. +.ih +EXAMPLES +The aperture editor is a very flexible and interactive tool +for which it is impossible illustrate all likely uses. The following +give some simple examples. + +1. To define and edit apertures for image "n1.001": + + cl> apedit n1.001 + +2. To define apertures for one image and then apply them to several other +images: + +.nf + cl> apedit n1.* ref=n1.001 + Edit apertures for n1.001? (yes) + Edit apertures for n1.002? (yes) NO +.fi + +Answer "yes" to the first query for editing n1.001. To +the next query (for n1.002) respond with "NO". The remaining +images then will not be edited interactively. Note that after +defining the apertures for n1.001 they are recorded in the database +and subsequent images will be able to use them as reference apertures. + +3. Using the ":image name" and ":read image" colon commands and the +'f', 'g', 'z', 't' and 'e' keys the user can perform all the functions +available in the package without ever leaving the editor. The 'a' key +to set the ALL flag is very useful when dealing with many spectra in a +single image. +.ih +.ih +REVISIONS +.ls APEDIT V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". + +The aperture ID table information may now be contained in the +image header under the keywords SLFIBnnn. +.le +SEE ALSO +.nf +apdefault, apfind, aprecenter, apresize, aptrace, apsum, apall +center1d, cursor, gtools, icfit +.fi +.endhelp diff --git a/noao/twodspec/apextract/doc/apextract.hlp b/noao/twodspec/apextract/doc/apextract.hlp new file mode 100644 index 00000000..401d93e7 --- /dev/null +++ b/noao/twodspec/apextract/doc/apextract.hlp @@ -0,0 +1,365 @@ +.help package Feb94 noao.twodspec.apextract +.ih +NAME +apextract -- Identify, manipulate, and extract spectra in 2D images +.ih +USAGE +apextract +.ih +PARAMETERS +.ls dispaxis = 2 +Image axis along which the spectra dispersion run. The dispersion axis +is 1 when the dispersion is along lines so that spectra are horizontal +when displayed normally. The dispersion axis is 2 when the dispersion +is along columns so that spectra are vertical when displayed normally. +This parameter is superseded when the dispersion axis is defined in +the image header by the parameter DISPAXIS. +.le +.ls database = "database" +Database for storing aperture definitions. Currently the database is +a subdirectory of text files with prefix "ap" followed by the entry name, +usually the image name. +.le +.ls verbose = no +Print detailed processing and log information? The output is to the +standard output stream which is the user's terminal unless redirected. +.le +.ls logfile = "" +Text logfile of operations performed. If a file name is specified +log and history information produced by all the tasks in the package +is appended to the file. +.le +.ls plotfile = "" +Binary plot metacode file of aperture locations, traces, rejected points, +etc. If a file name is given metacode plots are appended. The contents +of the file may be manipulated with the tasks in the \fBplot\fR package. +The most common is \fBgkimosaic\fR. Special plotfile names may be used +to select only particular plots or plots not normally output. These are +debugall, debugfitspec, debugaps, debugspec, debugfits, debugtrace, +and debugclean which plot everything, the fitted spectrum, the apertures, +the extracted spectrum, profile fit plots, the trace, and the rejected +points during cleaned extraction. +.le +.ls version = "APEXTRACT V3.0: August 1990" +Version of the package. This is the third major version of the package. +.le +.ih +DESCRIPTION +The primary function of the \fBapextract\fR package is the extraction of +spectra from two dimensional formats to one dimensional formats. In +other words, the pixels at each wavelength are summed, possibly +subtracting a background or sky from other pixels at that wavelength, +to produce a vector of spectral fluxes as a function of wavelength. +It has become common to have many spectra in one two dimensional +image produced by instruments using echelles, fibers, and aperture +masks. Thus, the package provides many features for the efficient +extractions of multiple spectra as well as single spectra. There are +also some additional, special purpose tasks for modeling spectra +and using the aperture definitions, described below, +to create masks and modified flat field images. + +The package assumes that one of the image axes is the dispersion axis, +specified by the \fIdispaxis\fR package parameter or image header +parameter of the same name, and the other is the spatial axes. +This means that all pixels at the same column or line (the +orientation may be in either direction) are considered to be at the +same wavelength. Even if this is not exactly +true the resolution loss is generally quite small and the simplicity and +absence of interpolation problems justify this approach. The +alternatives are to rotate the image with \fBrotate\fR or use the more +complex \fBlongslit\fR package. Though extraction is strictly along +lines and columns the position of the spectrum along the spatial axis +is allowed to shift smoothly with wavelength. This accounts for small +misalignments and distortions. + +The two dimensional regions occupied by the spectra are defined by +digital apertures having a fixed width but with spatial position smoothly +varying with wavelength. The apertures have a number of attributes. +The aperture definitions are created and modified by the tasks in this +package and stored in a database specified by the parameter \fIdatabase\fR. +The database is currently a directory containing simple text files +in a human readable format. The elements of an aperture definition +are as follows. + + +.ce +Elements of an Aperture Definition +.ls aperture +An integer aperture identification number. The aperture number +must be unique within a set of apertures. The aperture number is +the primary means of referencing an aperture and the resulting +extracted spectra. The aperture numbers are part of the extracted +spectra image headers. The numbers may be any integer and in any order +but the most typical case is to have sequential numbers beginning +with 1. +.le +.ls beam +An integer beam number. The beam number need not be unique; i.e. +several apertures may have the same beam number. The beam numbers are +recorded in the image headers of the extracted spectra. The beam +number is often used to identify types of spectra such as object, +sky, arc, etc. +.le +.ls center +A pair of numbers specifying the center of the aperture along the spatial +and dispersion axes in the two dimensional image. The center along +the dispersion is usually defined as the middle of the image. The +rest of the aperture parameters are defined relative to the aperture +center making it easy to move apertures. +.le +.ls low, high +Pairs of numbers specifying the lower and upper limits of the +aperture relative to the center along the spatial and dispersion axes. +The lower limits are usually negative and the upper limits positive +but there is no actual restriction; i.e. the aperture can actually +be offset from the center position. Currently the dispersion +aperture limits are such that the entire length of the image along the +dispersion axis is used. In the future this definition can be +easily used for objective prism spectra. +.le +.ls curve, axis +An IRAF "curfit" function specifying a shift to be added to the center +position along the spatial axis, given by the axis parameter which is +the complement of the dispersion axis parameter \fIdispaxis\fR, as a +function of the dispersion coordinate. This trace function is one of +the standard IRAF \fBicfit\fR types; a legendre polynomial, a chebyshev +polynomial, a linear spline, or a cubic spline. +.le +.ls background +Background definition parameters. For the "average" background subtraction +option only the set of background sample regions (defined relative to +the aperture center) are used. For the "fit" option the parameters +are those used by the \fBicfit\fR package for fitting a function to +the points in the background sample regions. +.le + +This information as well as the image (or database entry) name are stored +in a text file, with name given by the prefix "ap" followed by the entry +name, in the database directory. An example with the special entry name +"last", stored in the file "database$aplast", is given below. The "begin" +line marks the beginning of an aperture definition. + + +.ce +Sample Aperture Database Entry + +.nf +# Fri 17:43:41 03-Aug-90 +begin aperture last 1 70.74564 256. + image last + aperture 1 + beam 1 + center 70.74564 256. + low -5. -255. + high 5. 256. + background + xmin -100. + xmax 100. + function chebyshev + order 1 + sample -10:-6,6:10 + naverage -3 + niterate 0 + low_reject 3. + high_reject 3. + grow 0. + axis 1 + curve 5 + 2. + 1. + 1. + 512. + 0. +.fi + +There are a number of logical functions which may be performed to +create, modify, and use the aperture definitions. These functions +are: +.ls o +Automatically find a specified number of spectra and assign default +apertures. Apertures may also be inherited from another image or +defined using an interactive graphical interface called the \fIaperture +editor\fR. +.le +.ls o +Recenter apertures on the image spectrum profiles. +.le +.ls o +Resize apertures based on spectrum profile width. +.le +.ls o +Interactively define or adjust aperture definitions using a graphical +interface called the \fIaperture editor\fR. All function may also +be performed from this editor and, so, provides an alternative +method of processing and extracting spectra. +.le +.ls o +Trace the positions of spectra profiles from a starting image line +or column to other image lines or columns and fit a smooth function. +The trace function is used to shift the center of the apertures +at each dispersion point in the image. +.le +.ls o +Extract the flux in the apertures into one dimensional spectra in various +formats. This includes possible background subtraction, variance +weighting, and bad pixel rejection. +.le + +The package is logically organized around these functions. Each +function has a task devoted to it. The description of the parameters +and algorithms for each function are organized according to these +tasks; namely under the help topics \fBapdefault, apfind, aprecenter, +apresize, apedit, aptrace\fR, and \fBapsum\fR. However, each task has +parameters to allow selecting some or all of the other functions, hence +it is not necessary to use the individual tasks and often it is more +convenient to use just the extraction task for all operations. It is +also possible to perform all the functions from within a graphical +interface called the aperture editor. This is usually only used to +define and modify aperture definitions but it also has the capability +to trace spectra and extract them. + +Each of the functions has many different options and parameters. When +broken down into individual tasks the parameters are also sorted by +their function though there are then some mutual interdependencies. +This parameter decomposition was what was available prior to the +addition of the task \fBapall\fR. This is the central task of the +package which performs any and all of the functions required for the +extraction of spectra and also collects all the parameters into one +parameter set. It is recommended that \fBapall\fR be used because it +collects all the parameters in one place eliminating confusion over +where a particular parameter is defined. + +In summary, the package consists of a number of logical functions which +are documented by the individual tasks named for that function, but the +functions are also integrated into each task and the aperture editor to +providing many different ways for the user to choose to perform the +functions. + +The package menu and help summary is shown below. + + +.ce +The APEXTRACT Package Tasks + +.nf + apall apedit apflatten aprecenter apsum + apdefault apfind apmask apresize aptrace + apdemos apfit apnormalize apscatter + + apall - Extract 1D spectra (all parameters in one task) + apdefault - Set the default aperture parameters and apidtable + apdemos - Various tutorial demonstrations + apedit - Edit apertures interactively + apfind - Automatically find spectra and define apertures + apfit - Fit 2D spectra and output the fit, difference, + or ratio + apflatten - Remove overall spectral and profile shapes from + flat fields + apmask - Create and IRAF pixel list mask of the apertures + apnormalize - Normalize 2D apertures by 1D functions + aprecenter - Recenter apertures + apresize - Resize apertures + apscatter - Fit and subtract scattered light + apsum - Extract 1D spectra + aptrace - Trace positions of spectra + + Additional topics + + apbackground - Background subtraction algorithms + apextract - Package parameters and general description of + package + approfiles - Profile determination algorithms + apvariance - Extractions, variance weighting, cleaning, and + noise model +.fi + +The extracted spectra are recorded in one, two, or three dimensional +images depending on the \fIformat\fR and \fIextras\fR parameters. If +the \fIextras\fR parameter is set to yes the formats are three +dimensional with each plane in the third dimension containing +associated information for the spectra in the first plane. See +\fBapsum\fR for further details. When \fIextras\fR=no only the +extracted spectra are output. + +If the format parameter is "onedspec" the output extractions are one +dimensional images with names formed from an output rootname and an +aperture number extension; i.e. root.0001 for aperture 1. There will +be as many output images as there are apertures for each input image, +all with the same output rootname but with different aperture +extensions. This format is provided to be compatible with the original +format used by the \fBonedspec\fR package. + +If the format parameter is "echelle" or "multispec" the output aperture +extractions are put into a two dimensional image with a name formed from +the output rootname and the extension ".ec" or ".ms". Each line in +the output image corresponds to one aperture. Thus in this format +there is one output image for each input image. These are the preferred +output formats for reasons of compactness, ease of handling, and efficiency. +These formats are compatible with the \fBonedspec\fR, \fBechelle\fR, and +\fBmsred\fR packages. The format is a standard IRAF image with +specialized image header keywords. Below is an example of the keywords. + + +.ce +MULTISPEC/ECHELLE Format Image Header Keywords + +.nf + ap> imhead test.ms + test.ms[512,2,4][real]: Title + BANDID1 = 'spectrum - background fit, weights variance, clean yes' + BANDID2 = 'spectrum - background fit, weights none, clean no' + BANDID3 = 'background - background fit' + BANDID4 = 'sigma - background fit, weights variance, clean yes' + APNUM1 = '1 1 87.11 94.79' + APNUM2 = '2 1 107.11 114.79' + APID1 = 'Galaxy center' + APID2 = 'Galaxy edge' + WCSDIM = 3 + CTYPE1 = 'PIXEL ' + CTYPE2 = 'LINEAR ' + CTYPE3 = 'LINEAR ' + CRVAL1 = 1. + CRPIX1 = 1. + CD1_1 = 1. + CD2_2 = 1. + CD3_3 = 1. + LTM1_1 = 1. + LTM2_2 = 1. + LTM3_3 = 1. + WAT0_001= 'system=equispec + WAT1_001= 'wtype=linear label=Pixel + WAT2_001= 'wtype=linear + WAT3_001= 'wtype=linear +.fi + +The BANDIDn keywords describe the various elements of the 3rd dimension. +Except for the first one the other bands only occur when \fIextras\fR is +yes and when sky subtraction and/or variance and cleaning are done. The +relation between the line and the aperture numbers is given by the header +parameters APNUMn where n is the line and the value gives extraction and +coordinate information about the spectrum. The first field is the aperture +number and the second is the beam number. After dispersion calibration of +echelle format spectra the beam number becomes the order number. The other +two numbers are the aperture limits at the line or column at which the +aperture was defined. +The APID keywords provide an optional title for each extracted spectrum +in addition to the overall image title. + +The rest of the keywords are part of the IRAF World Coordinate System +(WCS). If the image being extracted has been previously calibrated +(say with \fBlongslit.transform\fR) then the dispersion coordinates +will be carried in CRVAL1 and CD1_1. + +There is one other value for the format parameter, "strip". This produces +two dimensional extractions rather than one dimensional extractions. +Each aperture is output to a two dimensional image with a width set by the +nearest integer which includes the aperture. The output names are +generated in the same way as for "onedspec" format. The aperture is +shifted by interpolation so that it is exactly aligned with the image +columns. If not variance weighting the actual image data is output +with appropriate shifting while for variance weighting and/or cleaning +the profile model is output (similar to \fBapfit\fR except for being +aligned). This format is that provided in the previous version of +the package by the \fBapstrip\fR task. It is now relegated to a +special case. +.endhelp diff --git a/noao/twodspec/apextract/doc/apextractsys.hlp b/noao/twodspec/apextract/doc/apextractsys.hlp new file mode 100644 index 00000000..a93d9f56 --- /dev/null +++ b/noao/twodspec/apextract/doc/apextractsys.hlp @@ -0,0 +1,415 @@ +.help apextract Aug90 noao.twodspec.apextract + +.ce +APEXTRACT System Notes + + +\fBIntroduction\fR + +The \fBapextract\fR package is a complex package with a simple +purpose, the extraction of one dimensional spectra from two dimensional +images. The complexity arises from the many algorithms and parameters +involved. To manage the complexity of the algorithms, features, parameters, +functionality, and documentation the package has been organized in terms +of logical functions which may be invoked in a number of ways. The +logical functions are: +.ls o +Automatically find a specified number of spectra and assign default +apertures. Apertures may also be inherited from another image or +defined using an interactive graphical interface called the \fIaperture +editor\fR. +.le +.ls o +Recenter apertures on the image spectrum profiles. +.le +.ls o +Resize apertures based on spectrum profile width. +.le +.ls o +Interactively define or adjust aperture definitions using a graphical +interface called the \fIaperture editor\fR. All function may also +be performed from this editor and, so, provides an alternative +method of processing and extracting spectra. +.le +.ls o +Trace the positions of spectra profiles from a starting image line +or column to other image lines or columns and fit a smooth function. +The trace function is used to shift the center of the apertures +at each dispersion point in the image. +.le +.ls o +Extract the flux in the apertures into one dimensional spectra in various +formats. This includes possible background subtraction, variance +weighting, and bad pixel rejection. +.le + +The package is logically organized around these functions. Each +function has a task devoted to it. The description of the parameters +and algorithms for each function are organized according to these +tasks; namely under the help topics \fBapdefault, apfind, aprecenter, +apresize, apedit, aptrace\fR, and \fBapsum\fR. However, each task has +parameters to allow selecting some or all of the other functions, hence +it is not necessary to use the individual tasks and often it is more +convenient to use just the extraction task for all operations. It is +also possible to perform all the functions from within a graphical +interface called the aperture editor. This is usually only used to +define and modify aperture definitions but it also has the capability +to trace spectra and extract them. + +Each of the functions has many different options and parameters. When +broken down into individual tasks the parameters are also sorted by +their function though there are then some mutual interdependencies. +This parameter decomposition was what was available prior to the +addition of the task \fBapall\fR. This is the central task of the +package which performs any and all of the functions required for the +extraction of spectra and also collects all the parameters into one +parameter set. It is recommended that \fBapall\fR be used because it +collects all the parameters in one place eliminating confusion over +where a particular parameter is defined. + +In summary, the package consists of a number of logical functions which +are documented by the individual tasks named for that function, but the +functions are also integrated into each task and the aperture editor to +providing many different ways for the user to choose to perform the +functions. + +This document describes some of the implementation details and features +which are hidden from the normal user. + + +\fBParameters\fR + +The tasks actually use hidden parameter sets for almost all parameters. +To see all the parameter sets type + +.nf + ap> ?_ apextract +.fi + +The relation between the tasks and the hidden parameter sets is given below. + +.nf + PSET TASK + apparams - apdefault, apfind, aprecenter, apresize, + apedit, aptrace, apsum, apmask, apscatter + apall1 - apall + apfit1 - apfit + apflat1 - apflatten + apnorm1 - apnormalize +.fi + +The hidden parameter sets may be viewed in any of the normal ways +\fBeparam\fR, \fBlparam\fR, or just by typing their name, except +their names may not be abbreviated. Their purpose is to redirect +parameters to visible parameter sets, to hide some parameters which +are not meant to be changed by the user, and to include parameters +used for queries. + +Most of the redirected parameters go to a single visible parameter set +or to package parameters. +The interesting exception is \fBapparams\fR which provides the +parameter linkage between the various functional tasks like +\fBapfind\fR, \fBaptrace\fR, \fBapsum\fR, etc. Below is a reproduction +of this parameter set. + +.ce +APPARAMS Hidden Parameter Set + +.nf + I R A F + Image Reduction and Analysis Facility +PACKAGE = apextract + TASK = apparams + +(format = )_.format) Extracted spectra format +(extras = )apsum.extras) Extract sky, sigma, etc.? +(dbwrite= yes) Write to database? +(initial= yes) Initialize answers? +(verbose= )_.verbose) Verbose output? + + # DEFAULT APERTURE PARAMETERS + +(upper = )apdefault.upper) Upper aperture limit relative to center +(apidtab= )apdefault.apidtable) Aperture ID table (optional) + + # DEFAULT BACKGROUND PARAMETERS + +(b_funct= )apdefault.b_function) Background function +(b_order= )apdefault.b_order) Background function order +(b_sampl= )apdefault.b_sample) Background sample regions +(b_naver= )apdefault.b_naverage) Background average or median +(b_niter= )apdefault.b_niterate) Background rejection iterations +(b_low_r= )apdefault.b_low_reject) Background lower rejection sigma +(b_high_= )apdefault.b_high_reject) Background upper rejection sigma +(b_grow = )apdefault.b_grow) Background rejection growing radius + + # APERTURE CENTERING PARAMETERS + +(width = )apedit.width) Profile centering width +(radius = )apedit.radius) Profile centering radius +(thresho= )apedit.threshold) Detection threshold for profile centering + + # AUTOMATIC FINDING AND ORDERING PARAMETERS + +(nfind = )apfind.nfind) Number of apertures to be found automatically +(minsep = )apfind.minsep) Minimum separation between spectra +(maxsep = )apfind.maxsep) Maximum separation between spectra +(order = )apfind.order) Order of apertures + + # RECENTERING PARAMETERS + +(apertur= )aprecenter.apertures) Select apertures +(npeaks = )aprecenter.npeaks) Select brightest peaks +(shift = )aprecenter.shift) Use average shift instead of recentering? + + # RESIZING PARAMETERS + +(llimit = )apresize.llimit) Lower aperture limit relative to center +(ulimit = )apresize.ulimit) Upper aperture limit relative to center +(ylevel = )apresize.ylevel) Fraction of peak or intensity for automatic widt(peak = )apresize.peak) Is ylevel a fraction of the peak? +(bkg = )apresize.bkg) Subtract background in automatic width? +(r_grow = )apresize.r_grow) Grow limits by this factor +(avglimi= )apresize.avglimits) Average limits over all apertures? + + # EDITING PARAMETERS + +e_output= Output spectra rootname +e_profil= Profile reference image +(t_nsum = )aptrace.nsum) Number of dispersion lines to sum +(t_step = )aptrace.step) Tracing step +(t_width= )apedit.width) Centering width for tracing +(t_funct= )aptrace.function) Trace fitting function +(t_order= )aptrace.order) Trace fitting function order +(t_sampl= )aptrace.sample) Trace sample regions +(t_naver= )aptrace.naverage) Trace average or median +(t_niter= )aptrace.niterate) Trace rejection iterations +(t_low_r= )aptrace.low_reject) Trace lower rejection sigma +(t_high_= )aptrace.high_reject) Trace upper rejection sigma +(t_grow = )aptrace.grow) Trace rejection growing radius + + # EXTRACTION PARAMETERS + +(backgro= )apsum.background) Background to subtract (none|average|fit) +(skybox = )apsum.skybox) Box car smoothing length for sky +(weights= )apsum.weights) Extraction weights (none|variance) +(clean = )apsum.clean) Detect and replace bad pixels? +(niterat= 2) Number of profile fitting iterations +(saturat= )apsum.saturation) Saturation level +(readnoi= )apsum.readnoise) Read out noise sigma (photons) +(gain = )apsum.gain) Photon gain (photons/data number) +(lsigma = )apsum.lsigma) Lower rejection threshold +(usigma = )apsum.usigma) Upper rejection threshold +(maxtilt= 3) Maximum excursion for line/column fitting +(polysep= 0.95) Marsh algorithm polynomial spacing +(polyord= 10) Marsh algorithm polynomial order +(nsubaps= )apsum.nsubaps) Number of subapertures per aperture + + # ANSWER PARAMETERS + +(ansclob= no) +(ansclob= no) +(ansdbwr= yes) +(ansdbwr= yes) +(ansedit= yes) +(ansextr= yes) +(ansfind= yes) +(ansfit = yes) +(ansfits= yes) +(ansfits= yes) +(ansfits= yes) +(ansfits= yes) +(ansfitt= yes) +(ansfitt= yes) +(ansflat= yes) +(ansmask= yes) +(ansnorm= yes) +(ansrece= yes) +(ansresi= yes) +(ansrevi= yes) +(ansrevi= yes) +(ansscat= yes) +(anssmoo= yes) +(anstrac= no) +(mode = q) +.fi + +Note how the parameters are redirected to a variety of tasks. + + +\fBInvisible Parameters\fR + +The following algorithm parameters are not visible to the normal user +and are described only here. +.ls dbwrite = yes +Write to database? Writing to the database is a function just like +find, edit, extract, etc. When the task is interactive a query is +made whether to write to the database which may be answered with the +usual four values. When noninteractive the database writing is automatic. +This parameter provides the possibility of turning off database writing. +.le +.ls initialize = yes +Initialize default queries? Normally each invocation of a task results +in new queries independent of the last responses in a prior invocation +and based only on the functions selected; NO for those not selected and +yes for those selected. By setting this to no either the prior values +may be used or the response values may be set independently of the +function flags. This is used in scripts to tie together different +invocations of the task and to finely control the queries. +.le +.ls e_output, e_profile +These are query parameters used when extraction is invoked from the +aperture editor. +.le + +The following parameters are part of the variance weighted and cleaning +extractions. They are described further in \fBapprofiles\fR. +.ls niterate = 2 +Number of rejection iterations in the profile determination when cleaning. +Iteration of the profile is slow and the low order fitting function +is not very sensitive to deviant points. +.le +.ls maxtilt = 3 +Maximum excursion separating the two profile fitting algorithms. +.le +.ls polysep = 0.95 +Marsh algorithm polynomial spacing. +.le +.ls polyorder = 10 +Marsh algorithm polynomial order. +.le + + +\fBQuery Mechanism and Invisible Query Parameters\fR + +The querying mechanism of the \fBapextract\fR package is a nice feature +but has some complexities in implementation. At the bottom of the +mechanism are CL checks of the parameters described below. The parameter +is accessed first as a hidden parameter. If the value is YES or NO +then the appropriate function is performed or not. If the value is +lower case then the task supplies a prompt string, which varies by +including the image and/or aperture involved, the mode of the +parameter is changed to query, and the parameter is requested again +leading to a CL query of the user with the current default value. +Finally, the parameter is returned to hidden mode. + +If the \fIinitialize\fR parameter is no then the initial default +query values are those set before the task is invoked. This provides +very fine control of the query mechanism and linking different +invocations of the tasks to previous user responses. It is intended +only for complex scripts such as those in the spectroscopic \fBimred\fR +packages. Normally the initial values of the parameters are set +during task startup based on the function flags. If a flag is no +then the related query parameter is NO. If the function flag is yes +then when the task is interactive the initial value is yes otherwise +it is YES. The solely interactive functions, such as editing, are +set to NO when the task is noninteractive regardless of the function +selection. +.ls ansclobber, ansclobber1 +Used to define the action to be taken if an output image would be clobbered. +Normally the action is to query if interactive and not clobber if +noninteractive. The first parameter acts as the function switch and +the second as the actual query. +.le +.ls ansdbwrite, ansdbwrite1 +The second parameter is used by the task to mark whether any changes have +been made that might require a database update. The first parameter is +the actual query parameter for the \fIdbwrite\fR function flag. +.le +.ls ansedit +Query parameter for the interactive editing function. +.le +.ls ansextract +Query parameter for the extraction function. +.le +.ls ansfind +Query parameter for the find function. +.le +.ls ansfit +Query parameter for the fit function of \fBapfit\fR. +.le +.ls ansfitscatter +Query parameter for the interactive fitscatter function of \fBapscatter\fR. +.le +.ls ansfitsmooth +Query parameter for the interactive fitsmooth function of \fBapscatter\fR. +.le +.ls ansfitspec +Query parameter for the interactive fitspec function of \fBapflatten\fR +and \fBapnormalize\fR. This applies to each image. +.le +.ls ansfitspec1 +Query parameter for the interactive fitspec function of \fBapflatten\fR +and \fBapnormalize\fR. This applies to each aperture in an image. +.le +.ls ansfittrace +Query parameter for the interactive fittrace function. +This applies to each image. +.le +.ls ansfittrace1 +Query parameter for the interactive fittrace function. +This applies to each aperture in an image. +.le +.ls ansflat +Query parameter for the flatten function of \fBapflatten\fR. +.le +.ls ansmask +Query parameter for the mask function of \fBapmask\fR. +.le +.ls ansnorm +Query parameter for the normalize function of \fBapnormalize\fR. +.le +.ls ansrecenter +Query parameter for the recenter function. +.le +.ls ansresize +Query parameter for the resize function. +.le +.ls ansreview +Query parameter for the interactive extraction review function. +This applies to each image. +.le +.ls ansreview1 +Query parameter for the interactive extraction review function. +This applies to each aperture in an image. +.le +.ls ansscat +Query parameter for the subtract function of \fBapscatter\fR. +.le +.ls anssmooth +Query parameter for the smooth function of \fBapscatter\fR. +.le +.ls anstrace +Query parameter for the trace function. +.le + + +\fBTask Entry Points\fR + +Logical tasks in IRAF are organized as multiple procedures in one physical +task selected by the IRAF main. The \fBapextract\fR package extends +this concept to a lower level. All of the package tasks go through +one procedure, \fBapall\fR. This procedure handles all of the +startup details and breaks the logical task down into selected +functions which are implemented as other procedures. There are +a couple of interesting and unusual features of this organization. + +IRAF physical tasks may map multiple logical task names to the same +procedure. However, the procedure will not know under what name it +was called. In this package we want to know the logical task name +in order to select the appropriate hidden parameter set and to +make minor adjustments in what the tasks do while maintaining the +same basic logical flow and source code. To do this dummy entry +points are used whose only function is to call \fBapall\fR and +pass an indication of the task name. + +Based on the task name a named parameter set is opened with \fBclopset\fR +and then all CLIO calls use the returned pointer and can be blind to the +actual parameter set used. + +In addition to the tasks defined in the package and their associated +parameter sets there is one more task entry point called \fBapscript\fR +with parameter set \fBapscript\fR. It is intended for use in scripts +as it's name implies. For this reason it does not need an intermediate +hidden parameter set. For examples of it's use see the \fBimred\fR +packages such as \fBnessie\fR. +.endhelp diff --git a/noao/twodspec/apextract/doc/apextras.hlp b/noao/twodspec/apextract/doc/apextras.hlp new file mode 100644 index 00000000..36a51b26 --- /dev/null +++ b/noao/twodspec/apextract/doc/apextras.hlp @@ -0,0 +1,61 @@ +.help extras Sep95 noao.twodspec.apextract +.ih +NAME +extras -- Information about the extra bands in 3D output +.ih +DESCRIPTION +When one dimensional spectra are extracted by the tasks in the +\fBapextract\fR package the user may specify that additional +extra associated information be extracted at the same time. This +information is produced when the \fIextras\fR parameter is "yes". + +The associated information is recorded as additional "bands" (the IRAF term +for the third dimension of a three dimensional image) of the output +extracted spectral image. Extracted spectra are currently stored as IRAF +images with dispersion information given in the image header. The +image axes for such images are: + +.nf + 1 (columns) - dispersion axis + 2 (lines) - spectrum axis (each line is a separate spectrum) + 3 (bands) - extras axis (each band is associated data) +.fi + +The lengths of the second and third axes, that is the number of +lines and bands, may be one or more. If there is only one band +the image will be two dimensional and if there is only one line +and one band the image will be one dimensional. Note that the +\fIformat\fR parameter controls whether multiple apertures are +written to separate images or to a single image. Thus, if +the format is "onedspec" this means that the second dimension +will always be of length one and, if the \fIextras\fR parameter +is no, the output images will be one dimensional. + +The associated data in the image bands depends on which extraction +options are performed. The various types of data are: + +.nf + The primary spectrum flux values. + Simple aperture sum if variance weighting or cleaning was done. + Background spectrum if background subtraction was done. + Sigma spectrum if variance weighting or cleaning was done. +.fi + +The primary spectrum is always the first band and will be the cleaned +and/or variance weighted and/or background subtracted spectrum. The +simple aperture sum spectrum allows comparing against the results of the +variance weighting or pixel rejection options. When background +subtraction is performed the subtracted background is recorded in +one of the bands. When variance weighting or pixel rejection is +performed the software generates an estimate of the uncertainty +in the extracted flux as a sigma. + +The identity of the various bands is given by the image header +keywords BANDIDn (where n is the band number). This also serves +to document which extraction options were used. + +For more information get help under the topic "apextract.package". +.ih +SEE ALSO +apextract.package +.endhelp diff --git a/noao/twodspec/apextract/doc/apfind.hlp b/noao/twodspec/apextract/doc/apfind.hlp new file mode 100644 index 00000000..65260394 --- /dev/null +++ b/noao/twodspec/apextract/doc/apfind.hlp @@ -0,0 +1,180 @@ +.help apfind Sep96 noao.twodspec.apextract +.ih +NAME +apfind -- Find spectra and define apertures automatically +.ih +USAGE +apfind input +.ih +PARAMETERS +.ls input +List of input images in which spectra are to be identified and +apertures defined automatically. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = no +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing is +disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database and the +parameter \fInfind\fR must be greater than zero. +.le +.ls recenter = no +Recenter the apertures? +.le +.ls resize = no +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le + +.ls line = INDEF +The dispersion line (line or column perpendicular to the dispersion axis) to +be used in finding the spectra. A value of INDEF selects the middle of the +image. +.le +.ls nsum = 1 +Number of dispersion lines to be summed or medianed. The lines are taken +around the specified dispersion line. A positive value sums lines and +a negative value medians lines. +.le +.ls nfind = 1 +Maximum number of apertures to be defined. This is a query parameter +so the user is queried for a value except when given explicitly on +the command line. +.le +.ls minsep = 5. +Minimum separation between spectra. Weaker spectra or noise within this +distance of a stronger spectrum are rejected. +.le +.ls maxsep = 1000. +Maximum separation between adjacent spectra. This parameter +is used to identify missing spectra in uniformly spaced spectra produced +by fiber spectrographs. If two adjacent spectra exceed this separation +then it is assumed that a spectrum is missing and the aperture identification +assignments will be adjusted accordingly. +.le +.ls order = "increasing" +When assigning aperture identifications order the spectra "increasing" +or "decreasing" with increasing pixel position (left-to-right or +right-to-left in a cross-section plot of the image). +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters are taken from the +task \fBapdefault\fR, and parameters used for centering and editing the +apertures are taken from \fBapedit\fR. + +When this operation is performed from the task \fBapall\fR all parameters +except the package parameters are included in that task. +.ih +DESCRIPTION +For each image in the input image list spectra are identified and +default apertures defined. The automatic aperture finding is performed +only if 1) there are no apertures defined for the reference image, 2) +there are no apertures defined for the input image, 3) the parameter +\fIfind\fR is yes, and 4) the parameter \fInfind\fR is greater than +zero. + +The automatic finding algorithm uses the following steps. First, all local +maxima are found. The maxima are sorted by peak value and the weaker +of the peaks separated by less than the value given by the parameter +\fIminsep\fR are rejected. Finally, at most the \fInfind\fR strongests +peaks are kept. \fBNfind\fR is a query parameter, so if it is not +specified explicitly on the command line, the desired number of spectra +to be found is requested. After the peaks have been found the +\fBcenter1d\fR algorithm is used to refine the centers of the +profiles. Apertures having the default parameters set with the task +\fBapdefault\fR are defined at each center. This algorithm is also +available with the 'f' key in the task \fBapedit\fR with the change that +existing apertures are kept and count toward the maximum number +specified by \fBnfind\fR. + +The automatic assignment of aperture numbers, beam numbers, and titles +has several options. The simplest is when no aperture identification +table, parameter \fIapidtable\fR, is specified and the maximum separation +parameter, \fImaxsep\fR, is very large. In this case the aperture and +beam numbers are sequential starting from one and numbered either from +left-to-right or right-to-left depending on the \fIorder\fR parameter. +There are no aperture titles in this case. If two adjacent spectra are +separated by more than the specified maximum then the aperture numbers +jump by the integer part of the ratio of the separation to the +specified maximum separation. This is used when the image is expected +to have evenly spaced spectra, such as in multifiber spectrographs, in +which some may be missing due to broken fibers. Finally, the +aperture identification table (either a text file or an image +having a set of SLFIBnnn keyowrds) may contain lines with aperture number, +beam number, and (optional) title. The sequential numbers are then +indices into this table. Note that the skipping of missing spectra and +the ordering applies to entries in this table as well. + +The ways in which the automatic method can fail for evenly spaced +spectra with missing members are when the first spectrum is missing on +the side from which the ordering begins and when the expected rather +the actual number of spectra is used. In the first case one can use +the interactive 'o' key of the aperture editing facility to specify the +identity of any aperture and then all other apertures will be +appropriately reidentified. If more spectra are sought than actually +exist then noise spikes may be mistakenly found. This problem can be +eliminated by specifying the actual number of spectra or minimized by +using the threshold centering parameter. + +The \fIrecenter\fR parameter allows recentering apertures if defined by +a reference image. Since the purpose of this task is to find new +apertures it is usually the case that there are no reference images and +recentering is not done. The default apertures are of fixed width. +The \fIresize\fR parameter may be used to adjust the widths in a +variety of ways. The aperture positions and any other parameters may +also be edited with the aperture editing function if selected by the +\fIapedit\fR parameter and the task is run interactively. + +If the task is interactive the user is queried whether to perform +various steps on each image. The queries may be answered with one of +the four values "yes", "no", "YES" and "NO", where an upper case +response suppresses all further queries to this question. + +The aperture finding algorithm may be selected from nearly every task +in the package. +.ih +EXAMPLES + cl> apfind image nfind=10 +.ih +.ih +REVISIONS +.ls APFIND V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". + +The aperture ID table information may now be contained in the +image header under the keywords SLFIBnnn. +.le +SEE ALSO +center1d, apdefault, aprecenter, apresize, apedit, apall +.endhelp diff --git a/noao/twodspec/apextract/doc/apfit.hlp b/noao/twodspec/apextract/doc/apfit.hlp new file mode 100644 index 00000000..60dd9b4c --- /dev/null +++ b/noao/twodspec/apextract/doc/apfit.hlp @@ -0,0 +1,263 @@ +.help apfit Sep96 noao.twodspec.apextract +.ih +NAME +apfit -- Fit 2D spectra using APEXTRACT profile algorithms +.ih +USAGE +apfit input output fittype +.ih +PARAMETERS +.ls input +List of input images to be fit. +.le +.ls output = "" +List of output images to be created with the fitting results. If the null +string is given or the end of the output list is reached before the end +of the input list then the input image name is used and an extension +of ".fit", ".diff", or ".ratio" is added based on the type of fit. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and fit. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls fittype = "difference" +Type of fitted output. The choices are: +.ls "fit" +The fitted spectra are output. +.le +.ls "difference" +The difference (or residuals) of the data and the fit (data - fit). +.le +.ls "ratio" +The ratio of the data to the fit. If a fitted pixel goes below a specified +threshold the ratio is set to 1. +.le +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing and trace +fitting are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le +.ls fit = yes +Fit the spectra and produce a fitted output image? +.le + +The following two parameters are used in the finding, recentering, resizing, +editing, and tracing operations. +.ls line = INDEF +The starting dispersion line (line or column perpendicular to the dispersion +axis) for the tracing. A value of INDEF starts at the middle of the image. +.le +.ls nsum = 1 +Number of dispersion lines to be summed or medianed at each step along +the dispersion. For tracing only summing is done and the sign is +ignored. +.le + +.ls threshold = 10. +Division threshold for ratio fit type. If a pixel in the fitted spectrum +is less than this value then a ratio of 1 is output. +.le + +The following parameters control the profile and spectrum fitting. +.ls background = "none" +Type of background subtraction. The choices are "none" for no +background subtraction, "average" to average the background within the +background regions, or "fit" to fit across the dispersion using the +background within the background regions. Note that the "average" +option does not do any medianing or bad pixel checking; it is faster +than fitting however. Background subtraction also requires that the +background fitting parameters are properly defined. For the "average" +option only the background sample regions parameter is used. +.le +.ls pfit = "fit1d" (fit1d|fit2d) +Profile fitting algorithm to use with variance weighting or cleaning. +When determining a profile the two dimensional spectrum is divided by +an estimate of the one dimensional spectrum to form a normalized two +dimensional spectrum profile. This profile is then smoothed by fitting +one dimensional functions, "fit1d", along the lines or columns most closely +corresponding to the dispersion axis or a special two dimensional +function, "fit2d", described by Marsh (see \fBapprofile\fR). +.le +.ls clean = no +Detect and replace deviant pixels? +.le +.ls skybox = 1 +Box car smoothing length for sky background when using background +subtraction. Since the background noise is often the limiting factor +for good extraction one may box car smooth the sky to improve the +statistics in smooth background regions at the expense of distorting +the subtraction near spectral features. This is most appropriate when +the sky regions are limited due to a small slit length. +.le +.ls saturation = INDEF +Saturation or nonlinearity level. During variance weighted extractions +wavelength points having any pixels above this value are excluded from the +profile determination. +.le +.ls readnoise = 0. +Read out noise in photons. This parameter defines the minimum noise +sigma. It is defined in terms of photons (or electrons) and scales +to the data values through the gain parameter. A image header keyword +(case insensitive) may be specified to get the value from the image. +.le +.ls gain = 1 +Detector gain or conversion factor between photons/electrons and +data values. It is specified as the number of photons per data value. +A image header keyword (case insensitive) may be specified to get the value +from the image. +.le +.ls lsigma = 3., usigma = 3. +Lower and upper rejection thresholds, given as a number of times the +estimated sigma of a pixel, for cleaning. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. +.ih +DESCRIPTION +The two dimensional spectra within the defined apertures of the input +images are fit by a model and new output images are created with either +the model spectra, the difference between the input and model spectra, +or the ratio of input and model spectra. The type of output is +selected by the parameter \fIfittype\fR which may have one of the +values "fit", "difference", or "ratio". + +Aperture definitions may be inherited from those of other images by +specifying a reference image with the \fBreferences\fR parameter. +Images in the reference list are matched with those in the +input list in order. If the reference image list is shorter than the +number of input images, the last reference image is used for all +remaining input images. Thus, a single reference image may be given +for all the input images or different reference images may be given for +each input image. The special reference name "last" may be used to +select the last set apertures used in any of the \fBapextract\fR tasks. + +If an aperture reference image is not specified or no apertures are +found for the specified reference image, previously defined apertures +for the input image are sought in the aperture database. Note that +reference apertures supersede apertures for the input image. If no +apertures are defined they may be created automatically, the \fIfind\fR +option, or interactively in the aperture editor, if the +\fIinteractive\fR and \fIedit\fR options are set. + +The functions performed by the task are selected by a set of flag +parameters. The functions are an automatic spectrum finding and +aperture defining algorithm (see \fBapfind\fR) which is ignored if +apertures are already defined, automatic recentering and resizing +algorithms (see \fBaprecenter\fR and \fBapresize\fR), an interactive +aperture editing function (see \fBapedit\fR), a spectrum position tracing +and trace function fit (see \fBaptrace\fR), and the main function of +this task, two dimensional model fitting. + +Each function selection will produce a query for each input spectrum if +the \fIinteractive\fR parameter is set. The queries are answered by +"yes", "no", "YES", or "NO", where the upper case responses suppress +the query for following images. There are other queries associated +with tracing which first ask whether the operation is to be done +interactively and, if yes, lead to queries for each aperture. If the +\fIinteractive\fR parameter is not set then aperture editing and +interactive trace fitting are ignored. + +The two dimensional spectrum model consists of a smooth two dimensional +normalized profile multiplied by the variance weighted one dimensional +spectrum. The profile is computed by dividing the data within the aperture +by the one dimensional spectrum, smoothing with either low order function +fits parallel to the dispersion axis or a special two dimensional function +as selected by the \fIpfit\fR parameter. The smooth profile is then used +to improve the spectrum estimate using variance weighting and to eliminate +deviant or cosmic ray pixels by sigma tests. The profile algorithm is +described in detail in \fBapprofiles\fR and the variance weighted spectrum +is described in \fBapvariance\fR. + +The process of determining the profile and variance weighted spectrum, +and hence the two dimensional spectrum model, is identical to that used +for variance weighted extraction of the one dimensional spectra in the +tasks \fBapall\fR or \fBapsum\fR. Most of the parameters of in this +task are the same as those in the extraction tasks and so further +information about them may be found in the descriptions of those tasks. + +Because of the connection with variance weighted extraction and cleaning +of one dimensional spectra, this task is useful as a diagnostic tool for +understanding and evaluating the variance weighting algorithm. +For example the "difference" image provides the residuals in a +two dimensional visual form. + +The "fit" output image does not include any background determination; +i.e the fit is background subtracted. Pixels outside the modeled +spectra are set to zero. + +The "difference" output image is simply the difference between the +background subtracted "fit" and the data. Thus the difference within +the apertures should approximate the background and outside the +apertures the difference will be identical with the input image. + +The "ratio" output image does include any background in the model +before taking the ratio of the data and model. If a model pixel +is less than the given \fIthreshold\fR parameter the output ratio +is set to one. This is used to avoid division by zero and set a +limit to noise in ratio image. Outside of the apertures the ratio +output pixels are set to one. +.ih +EXAMPLES +1. To compute the residuals of a model fit where the image already has +aperture defined: + + cl> apfit ls1 inter- rec- res- trace- read=3 gain=1 back=fit + +.ih +REVISIONS +.ls APFIND V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +apbackground, approfile, apvariance, +apdefault, apfind, aprecenter, apresize, apedit, aptrace, apsum, apall +.endhelp diff --git a/noao/twodspec/apextract/doc/apflatten.hlp b/noao/twodspec/apextract/doc/apflatten.hlp new file mode 100644 index 00000000..f7e1b8c0 --- /dev/null +++ b/noao/twodspec/apextract/doc/apflatten.hlp @@ -0,0 +1,304 @@ +.help apflatten Sep96 noao.twodspec.apextract +.ih +NAME +apflatten -- Create flat fields for fiber or narrow aperture spectra +.ih +USAGE +apflatten input output +.ih +PARAMETERS +.ls input +List of input flat field observations. +.le +.ls output = "" +List of output flat field images. If no output name is given then the +input name is used as a root with the extension ".flat". +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and flatten. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing and trace +fitting are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le +.ls flatten = yes +Remove the profile shape and flat field spectrum leaving only +sensitivity variations? +.le +.ls fitspec = yes +Fit normalization spectrum interactively? The \fIinteractive\fR +parameter must also be yes. +.le + +.ls line = INDEF, nsum = 1 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. For tracing this is the starting line and +the same number of lines are summed at each tracing point. A line of +INDEF selects the middle of the image along the dispersion axis. +A positive nsum sums the lines and a negative value takes the median. +However, for tracing only sums are allowed and the absolute value +is used. +.le +.ls threshold = 10. +Division threshold. If a pixel in the two dimensional normalization spectrum +is less than this value then a flat field value of 1 is output. +.le + +The following parameters control the profile and spectrum fitting. +.ls pfit = "fit1d" (fit1d|fit2d) +Profile fitting algorithm to use with variance weighting or cleaning. +When determining a profile the two dimensional spectrum is divided by +an estimate of the one dimensional spectrum to form a normalized two +dimensional spectrum profile. This profile is then smoothed by fitting +one dimensional functions, "fit1d", along the lines or columns most closely +corresponding to the dispersion axis or a special two dimensional +function, "fit2d", described by Marsh (see \fBapprofile\fR). +.le +.ls clean = no +Detect and replace deviant pixels? +.le +.ls saturation = INDEF +Saturation or nonlinearity level. During variance weighted extractions +wavelength points having any pixels above this value are excluded from the +profile determination. +.le +.ls readnoise = 0. +Read out noise in photons. This parameter defines the minimum noise +sigma. It is defined in terms of photons (or electrons) and scales +to the data values through the gain parameter. A image header keyword +(case insensitive) may be specified to get the value from the image. +.le +.ls gain = 1 +Detector gain or conversion factor between photons/electrons and +data values. It is specified as the number of photons per data value. +A image header keyword (case insensitive) may be specified to get the value +from the image. +.le +.ls lsigma = 3., usigma = 3. +Lower and upper rejection thresholds, given as a number of times the +estimated sigma of a pixel, for cleaning. +.le + +The following parameters are used to fit the normalization spectrum using +the ICFIT routine. +.ls function = "legendre" +Fitting function for the normalization spectra. The choices are "legendre" +polynomial, "chebyshev" polynomial, linear spline ("spline1"), and +cubic spline ("spline3"). +.le +.ls order = 1 +Number of polynomial terms or number of spline pieces for the fitting function. +.le +.ls sample = "*" +Sample regions for fitting points. Intervals are separated by "," and an +interval may be one point or a range separated by ":". +.le +.ls naverage = 1 +Number of points within a sample interval to be subaveraged or submedianed to +form fitting points. Positive values are for averages and negative points +for medians. +.le +.ls niterate = 0 +Number of sigma clipping rejection iterations. +.le +.ls low_reject = 3. , high_reject = 3. +Lower and upper sigma clipping rejection threshold in units of sigma determined +from the RMS sigma of the data to the fit. +.le +.ls grow = 0. +Growing radius for rejected points (in pixels). That is, any rejected point +also rejects other points within this distance of the rejected point. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. +.ih +DESCRIPTION +It is sometimes the case that it is undesirable to simply divide +two dimensional format spectra taken through fibers, aperture masks +with small apertures such as holes and slitlets, or small slits in +echelle formats by a flat field observation of a lamp. This is due +to the sharp dropoff of the flat field and object profiles and +absence of signal outside of the profile. Slight shifts or changes +in profile shape introduce bad edge effects, unsightly "grass" is +produced where there is no signal (which may also confuse extraction +programs), and the division will also remove the characteristic +profile of the object which might be needed for tracking the +statistical significance, variance weighted extraction, and more. +A straight flat field division also has the problem of changing the +shape of the spectrum in wavelength, again compromising the +poisson statistics and artificially boosting low signal regions. + +There are three approaches to consider. First, the +flat field correction can be done after extraction to one dimension. +This is valid provided the flat field and object profiles don't shift +much. However, for extractions that depend on a smooth profile, +such as the variance weighting algorithms of this package, the sensitivity +corrections must remain small; i.e. no large fringes or other +small scale variations that greatly perturb the true photon profile. +The second approach is to divide out the overall spectral shape of +the flat field spectrum, fill regions outside of the signal with +one and leave the profile shape intact. This will still cause profile +division problems described earlier but is mentioned here since it +implemented in a related task called \fBapnormalize\fR. The last +approach is to model both the profile and overall spectrum shape and +remove it from the flat field leaving only the sensitivity variations. +This is what the task \fBapflatten\fR does. + +The two dimensional flat field spectra within the defined apertures of +the input images are fit by a model having the profile of the data and +a smooth spectral shape. This model is then divided into the flat +field image within the aperture, replacing points of low signal, set +with the \fIthreshold\fR parameter, within the aperture and all points +outside the aperture by one to produce an output sensitivity variation +only flat field image. + +A two dimensional normalized profile is computed by dividing the data +within the aperture by the one dimensional spectrum and smoothing with +low order function fits parallel to the dispersion axis if the aperture +is well aligned with the axis or parallel to the traced aperture center +if the trace is tilted relative to the dispersion axis. The smooth +profile is then used to improve the spectrum estimate using variance +weighting and to eliminate deviant or cosmic ray pixels by sigma +tests. The profile algorithm is described in detail in +\fBapprofiles\fR and the variance weighted spectrum is described in +\fBapvariance\fR. + +The process of determining the profile and variance weighted spectrum, +and hence the two dimensional spectrum model, is identical to that used +for variance weighted extraction of the one dimensional spectra in the +tasks \fBapall\fR or \fBapsum\fR and in making a two dimensional +spectrum model in the task \fBapfit\fR. Most of the parameters in +this task are the same in those tasks and so further information about +them may be found in their descriptions. In fact, up to this point the +task is the same as \fBapfit\fR and, if the flat field were normalized +by this model it would produce the "ratio" output of that task. + +This task deviates from \fBapfit\fR in that the final variance weighted +one dimensional spectrum of the flat field is subjected to a smoothing +operation. This is done by fitting a function to the spectrum using +the \fBicfit\fR routine. This may be done interactively or +noninteractively depending on the \fBinteractive\fR parameter. The +default fitting parameters are part of this task. The goal of the +fitting is to follow the general spectral shape of the flat field light +(usually a lamp) but not the small bumps and wiggles which are the one +dimensional projection of sensitivity variations. When the fitted +function is multiplied into the normalize profile and then the two +dimensional model divided into the data the sensitivity variations not +part of the fitted spectrum are what is left in the final output flat +field. + +The remainder of this description covers the basic steps defining the +apertures to be used. These steps and parameter are much the same as +in any of the other \fBapextract\fR tasks. + +Aperture definitions may be inherited from those of other images by +specifying a reference image with the \fBreferences\fR parameter. +Images in the reference list are matched with those in the input list +in order. If the reference image list is shorter than the number of +input images, the last reference image is used for all remaining input +images. Thus, a single reference image may be given for all the input +images or different reference images may be given for each input +image. The special reference name "last" may be used to select the +last set apertures used in any of the \fBapextract\fR tasks. + +If an aperture reference image is not specified or no apertures are +found for the specified reference image, previously defined apertures +for the input image are sought in the aperture database. Note that +reference apertures supersede apertures for the input image. If no +apertures are defined they may be created automatically, the \fIfind\fR +option, or interactively in the aperture editor, if the +\fIinteractive\fR and \fIedit\fR options are set. + +The functions performed by the task are selected by a set of flag +parameters. The functions are an automatic spectrum finding and +aperture defining algorithm (see \fBapfind\fR) which is ignored if +apertures are already defined, automatic recentering and resizing +algorithms (see \fBaprecenter\fR and \fBapresize\fR), an interactive +aperture editing function (see \fBapedit\fR), a spectrum position tracing +and trace function fit (see \fBaptrace\fR), and the main function of +this task, the flat field profile and spectral shape modeling and removal. + +Each function selection will produce a query for each input spectrum if +the \fIinteractive\fR parameter is set. The queries are answered by +"yes", "no", "YES", or "NO", where the upper case responses suppress +the query for following images. There are other queries associated +with tracing which first ask whether the operation is to be done +interactively and, if yes, lead to queries for each aperture. If the +\fIinteractive\fR parameter is not set then aperture editing +interactive trace fitting, and interactive spectrum shape fitting are ignored. +.ih +REVISIONS +.ls APFLATTEN V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +EXAMPLES +1. To make a two dimensional flat field from a lamp observation: + +.nf + cl> apflatten fiber1 flat read=3 gain=1 back=fit + Yes find + No resize + No edit + Yes trace + Yes trace interactively + NO + Yes flatten + Yes fit interactively +.fi +.ih +SEE ALSO +apbackground, approfile, apvariance, apfit, icfit, +apdefault, apfind, aprecenter, apresize, apedit, aptrace, apsum +.endhelp diff --git a/noao/twodspec/apextract/doc/apmask.hlp b/noao/twodspec/apextract/doc/apmask.hlp new file mode 100644 index 00000000..78d775f9 --- /dev/null +++ b/noao/twodspec/apextract/doc/apmask.hlp @@ -0,0 +1,123 @@ +.help apmask Sep96 noao.twodspec.apextract +.ih +NAME +apmask -- Make pixel mask from apertures definitions +.ih +USAGE +apfind input +.ih +PARAMETERS +.ls input +List of input images with aperture definitions. +.le +.ls output +List of output mask names. As a convention the extension ".pl" (pixel +list) should be used. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and create a mask. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = no +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing is +disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database and the +parameter \fInfind\fR must be greater than zero. +.le +.ls recenter = no +Recenter the apertures? +.le +.ls resize = no +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace apertures? +.le +.ls fittrace = yes +Fit the traced points interactively? The \fIinteractive\fR parameter +must also be yes. +.le +.ls mask = yes +Create mask images? +.le + +.ls line = INDEF +The dispersion line (line or column perpendicular to the dispersion axis) to +be used in finding, recentering, resizing, editing, and starting to +trace spectra. A value of INDEF selects the middle of the image. +.le +.ls nsum = 1 +Number of dispersion lines to be summed or medianed. The lines are taken +around the specified dispersion line. A positive value takes the +sum and a negative value selects a median. +.le +.ls buffer = 0. +Buffer to add to aperture limits. One use for this is to increase +the width of the apertures when a mask is used to fit data between +the apertures. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. +.ih +DESCRIPTION +Pixel list masks are created from the aperture definitions in the input +images. Pixel list masks are a compact way to define arbitrary +regions of an image. The masks may be used directly as an image with values +of 1 (in an aperture) and 0 (outside an aperture). Alternatively, +some tasks may use a mask to define regions to be operated upon. +When this task was written there were no such tasks though eventually +some tasks will be converted to use this general format. The intent +of making an aperture mask is to someday allow using it with the task +\fBimsurfit\fR to fit a background or scattered light surface. +(See \fBapscatter\fR for an alternative method). +.ih +EXAMPLES +1. To replace all data outside the apertures by zero: + +.nf + cl> apmask image image.pl nfind=10 + cl> imarith image * image.pl image1 +.fi +.ih +REVISIONS +.ls APMASK V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +apdefault, aprecenter, apresize, apedit, aptrace, apall +.endhelp diff --git a/noao/twodspec/apextract/doc/apnoise.hlp b/noao/twodspec/apextract/doc/apnoise.hlp new file mode 100644 index 00000000..a4f69f83 --- /dev/null +++ b/noao/twodspec/apextract/doc/apnoise.hlp @@ -0,0 +1,231 @@ +.help apnoise Sep96 noao.twodspec.apextract +.ih +NAME +apnoise -- Compute and examine noise characteristics of spectra +.ih +USAGE +apnoise input dmin dmax nbins +.ih +PARAMETERS +.ls input +List of input spectra to examine. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls dmin, dmax, nbins +The noise sigma is computed in a set of bins over the specified +range of image data numbers. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing and trace +fitting are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le + +.ls line = INDEF, nsum = 1 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. For tracing this is the starting line and +the same number of lines are summed at each tracing point. A line of +INDEF selects the middle of the image along the dispersion axis. +A positive nsum sums the lines and a negative value takes the median. +However, for tracing only sums are allowed and the absolute value +is used. +.le +.ls threshold = 10. +Division threshold. If a pixel in the two dimensional normalization spectrum +is less than this value then a flat field value of 1 is output. +.le + +The following parameters control the profile and spectrum fitting. +.ls background = "none" +Type of background subtraction. The choices are "none" for no +background subtraction, "average" to average the background within the +background regions, or "fit" to fit across the dispersion using the +background within the background regions. Note that the "average" +option does not do any medianing or bad pixel checking; it is faster +than fitting however. Background subtraction also requires that the +background fitting parameters are properly defined. For the "average" +option only the background sample regions parameter is used. +.le +.ls pfit = "fit1d" (fit1d|fit2d) +Profile fitting algorithm to use with variance weighting or cleaning. +When determining a profile the two dimensional spectrum is divided by +an estimate of the one dimensional spectrum to form a normalized two +dimensional spectrum profile. This profile is then smoothed by fitting +one dimensional functions, "fit1d", along the lines or columns most closely +corresponding to the dispersion axis or a special two dimensional +function, "fit2d", described by Marsh (see \fBapprofile\fR). +.le +.ls clean = no +Detect and replace deviant pixels? +.le +.ls skybox = 1 +Box car smoothing length for sky background when using background +subtraction. Since the background noise is often the limiting factor +for good extraction one may box car smooth the sky to improve the +statistics in smooth background regions at the expense of distorting +the subtraction near spectral features. This is most appropriate when +the sky regions are limited due to a small slit length. +.le +.ls saturation = INDEF +Saturation or nonlinearity level. During variance weighted extractions +wavelength points having any pixels above this value are excluded from the +profile determination. +.le +.ls readnoise = "0." +Read out noise in photons. This parameter defines the minimum noise +sigma. It is defined in terms of photons (or electrons) and scales +to the data values through the gain parameter. A image header keyword +(case insensitive) may be specified to get the value from the image. +.le +.ls gain = "1." +Detector gain or conversion factor between photons/electrons and +data values. It is specified as the number of photons per data value. +A image header keyword (case insensitive) may be specified to get the value +from the image. +.le +.ls lsigma = 3., usigma = 3. +Lower and upper rejection thresholds, given as a number of times the +estimated sigma of a pixel, for cleaning. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. +.ih +CURSOR COMMANDS +The following cursor keys and colon commands are available during the +display of the noise sigmas and noise model. See \fBapedit\fR for +the commands for that mode. + +.nf +? Print command help +q Quit +r Redraw +w Window the graph (see :/help) +I Interupt immediately + +:gain <value> Check or set the gain model parameter +:readnoise <value> Check or set the read noise model parameter + +Also see the CURSOR MODE commands (:.help) and the windowing commands +(:/help). +.fi +.ih +DESCRIPTION +\fBApnoise\fR computes the noise sigma as a function of data value +using the same profile model used for weighted extraction and +cosmic ray cleanning. In particular, the residuals used in computing the +noise sigma are the same as those during cleanning. By looking +at the noise sigma as a function of data value as compared to that +predicted by the noise model based on the read out noise and gain +parameters one can then better refine these values for proper +rejection of cosmic rays without rejection of valid data. +So this task can be used to check or deduce these values and also +to adjust them to include additional sources of error such as +flat field noise and, especially, an additional source of noise due +to the accuracy of the profile modeling. + +The first part of this task follows the standard model of allowing +one to define apertures by finding, recentering, editing, and +tracing. If one has previously defined apertures then these +steps can be skipped. Once the apertures are defined the apertures +are internally extracted using the profile modeling (see \fBapprofile\fR) +with the optional background subtraction, cleanning, and choices of +profile fitting algorithm, "fit1d" or "fit2d". But rather than +outputing the extracted spectrum as in \fBapsum\fR or \fBapall\fR +or various functions of the data and profile model as in \fBapfit\fR, +\fBapnormalize\fR, or \fBapflatten\fR, the task computes the +residuals for all points in all apertures (essentially the same +as the difference output of \fBapfit\fR) and determines the +sigma (population corrected RMS) as a function of model data value +in the specified bins. The bins are defined by a minimum and +maximum data value (found using \fBminmax\fR, \fBimplot\fR, or +\fBimexamine\fR) and the number of bins. + +The noise sigma values, with their estimated uncertainties, are then +plotted as a function of data numer. A curve representing the specified +read out noise and gain is also plotted. The user then has the +option of varying these two parameters with colon commands. The +aim of this is to find a noise model which either represents the +measure noise sigmas or at least exceeds them so that only valid +outliers such as cosmic rays will be rejected during cleanning. +The interactive graphical mode only has this function. The other +keys and colon commands are the standard ones for redrawing, windowing, +and quitting. +.ih +EXAMPLES +1. To check that the read noise and gain parameters are reasonable for +cleaning \fBapnoise\fR is run. In this case it is assumed that the +apertures have already been defined and traced. + +.nf + cl> minmax lsobj + lsobj -2.058870315551758 490.3247375488282 + cl> apnoise lsobj 0 500 50 rece- resi- edit- trace- + A graph of the noise sigma for data between 0 and 500 + data numbers is given with a line showing the + expected value for the current read noise and gain. + The read noise and gain may be varied if desired. + Exit with 'q' +.fi +.ih +REVISIONS +.ls APNOISE V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +apbackground, approfile, apvariance, apfit, icfit, minmax, +apdefault, apfind, aprecenter, apresize, apedit, aptrace, apsum +.endhelp diff --git a/noao/twodspec/apextract/doc/apnormalize.hlp b/noao/twodspec/apextract/doc/apnormalize.hlp new file mode 100644 index 00000000..fda3fd31 --- /dev/null +++ b/noao/twodspec/apextract/doc/apnormalize.hlp @@ -0,0 +1,324 @@ +.help apnormalize Sep96 noao.twodspec.apextract +.ih +NAME +apnormalize -- Normalize 2D apertures by 1D functions +.ih +USAGE +apnormalize input output +.ih +PARAMETERS +.ls input +List of input images to be normalized. +.le +.ls output +List of output image names for the normalized input images. If no output +name is given then the input name is used as a root with the extension +".norm" added. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and normalize. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing and trace +fitting are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le +.ls normalize = yes +Normalize the aperture spectra by a one dimensional function? +.le +.ls fitspec = yes +Fit normalization spectrum interactively? The \fIinteractive\fR +parameter must also be yes. +.le + +.ls line = INDEF, nsum = 1 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. For tracing this is the starting line and +the same number of lines are summed at each tracing point. A line of +INDEF selects the middle of the image along the dispersion axis. +A negative nsum selects a median rather than a sum except that +tracing always uses a sum. +.le +.ls cennorm = no +Normalize to the aperture center rather than the mean? +.le +.ls threshold = 10. +All pixels in the normalization spectrum less than this value are replaced +by this value. +.le + +The following parameters control the normalization spectrum extraction. +.ls background = "none" +Type of background subtraction. The choices are "none" for no +background subtraction, "average" to average the background within the +background regions, or "fit" to fit across the dispersion using the +background within the background regions. Note that the "average" +option does not do any medianing or bad pixel checking; it is faster +than fitting however. Background subtraction also requires that the +background fitting parameters are properly defined. For the "average" +option only the background sample regions parameter is used. +.le +.ls weights = "none" +Type of extraction weighting. Note that if the \fIclean\fR parameter is +set then the weights used are "variance" regardless of the weights +specified by this parameter. The choices are: +.ls "none" +The pixels are summed without weights except for partial pixels at the +ends. +.le +.ls "variance" +The extraction is weighted by estimated variances of the pixels using +a poisson noise model. +.le +.le +.ls pfit = "fit1d" (fit1d|fit2d) +Profile fitting algorithm to use with variance weighting or cleaning. +When determining a profile the two dimensional spectrum is divided by +an estimate of the one dimensional spectrum to form a normalized two +dimensional spectrum profile. This profile is then smoothed by fitting +one dimensional functions, "fit1d", along the lines or columns most closely +corresponding to the dispersion axis or a special two dimensional +function, "fit2d", described by Marsh (see \fBapprofile\fR). +.le +.ls clean = no +Detect and replace deviant pixels? +.le +.ls skybox = 1 +Box car smoothing length for sky background when using background +subtraction. Since the background noise is often the limiting factor +for good extraction one may box car smooth the sky to improve the +statistics in smooth background regions at the expense of distorting +the subtraction near spectral features. This is most appropriate when +the sky regions are limited due to a small slit length. +.le +.ls saturation = INDEF +Saturation or nonlinearity level. During variance weighted extractions +wavelength points having any pixels above this value are excluded from the +profile determination. +.le +.ls readnoise = 0. +Read out noise in photons. This parameter defines the minimum noise +sigma. It is defined in terms of photons (or electrons) and scales +to the data values through the gain parameter. A image header keyword +(case insensitive) may be specified to get the value from the image. +.le +.ls gain = 1 +Detector gain or conversion factor between photons/electrons and +data values. It is specified as the number of photons per data value. +A image header keyword (case insensitive) may be specified to get the value +from the image. +.le +.ls lsigma = 3., usigma = 3. +Lower and upper rejection thresholds, given as a number of times the +estimated sigma of a pixel, for cleaning. +.le + +The following parameters are used to fit the normalization spectrum using +the ICFIT routine. +.ls function = "legendre" +Fitting function for the normalization spectra. The choices are "legendre" +polynomial, "chebyshev" polynomial, linear spline ("spline1"), and +cubic spline ("spline3"). +.le +.ls order = 1 +Number of polynomial terms or number of spline pieces for the fitting function. +.le +.ls sample = "*" +Sample regions for fitting points. Intervals are separated by "," and an +interval may be one point or a range separated by ":". +.le +.ls naverage = 1 +Number of points within a sample interval to be subaveraged or submedianed to +form fitting points. Positive values are for averages and negative points +for medians. +.le +.ls niterate = 0 +Number of sigma clipping rejection iterations. +.le +.ls low_reject = 3. , high_reject = 3. +Lower and upper sigma clipping rejection threshold in units of sigma determined +from the RMS sigma of the data to the fit. +.le +.ls grow = 0. +Growing radius for rejected points (in pixels). That is, any rejected point +also rejects other points within this distance of the rejected point. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. +.ih +DESCRIPTION +For each image in the input image list the two dimensional spectra +defined by a set of apertures are normalized by a one dimensional +normalization function derived by extracting and smoothing the spectrum +by fitting a function with the \fBicfit\fR procedure. The value of the +fitting function at each point along the dispersion, divided by the +aperture width to form a mean or scaled to the same mean as the center +pixel of the aperture depending on the \fIcennorm\fR parameter, is +divided into the two dimensional input aperture. All points outside +the apertures are set to unity. + +The purpose of this task is to remove a general shape from the aperture +spectra. If low order (order = 1 for instance) functions are used then +only the amplitudes of the spectra are affected, shifting each aperture +to approximately unit intensity per pixel. If high order functions are +used only the small spatial scale variations are preserved. This +is useful for making flat field images with the spectral signature of the +continuum source removed or for producing two dimensional normalized +spectra similar to the task \fBonedspec.continuum\fR. For flat fields +this algorithm retains the profile shape which may be useful for +removing the profile response in short slit data. However, often +one does not want the profile of the flat fielded observation to be +modified in which case the task \fBapflatten\fR should be used. + +The normalization spectrum is first extracted in the same way as is +the one dimensional extraction in \fBapsum\fR or \fBapall\fR. In +particular the same parameters for selecting weighting and cleaning +are available. After extraction the spectrum is fit using the +\fBicfit\fR routine. This may be done interactively or noninteractively +depending on the \fIinteractive\fR parameter. The default fitting +parameters are part of this task. The goal of the fitting depends +on the application. One may be trying to simply continuum normalize, +in which case one wants to iteratively reject and grow the rejected +points to exclude the lines and fit the continuum with a +moderate order function (see \fBcontinuum\fR for more discussion). +If one wants to simply normalize all spectra to a common flux, say to +remove a blaze function in echelle data, then an order of 1 will +normalize by a constant. For flat field and profile correction of +small slits one wants to fit the large scale shape of the +spectrum but not fit the small bumps and wiggles due to sensitivity +variations and fringing. + +The smoothed extracted spectrum represents the total flux within the +aperture. There are two choices for scaling to a normalization per +pixel. One is to divide by the aperture width, thus computing an average +flux normalization. In this case the peak of the spectrum will be +greater than unity. This is done when \fIcennorm\fR = no. When this +parameter has the value yes then the mean of the normalization spectrum +is scaled to the mean of the aperture center, computed by linearly +interpolating the two pixels about the traced center. This will give +values near one for the pixels at the center of the aperture in the +final output image. + +Before division of each pixel by the appropriate dispersion point in +the normalization spectrum, all pixels below the value specified by the +\fIthreshold\fR parameter in the normalization spectrum are replaced by +the threshold value. This suppresses division by very small numbers. +Finally, the pixels within the aperture are divided by the normalization +function and the pixels outside the apertures are set to 1. + +The remainder of this description covers the basic steps defining the +apertures to be used. These steps and parameter are much the same as +in any of the other \fBapextract\fR tasks. + +Aperture definitions may be inherited from those of other images by +specifying a reference image with the \fBreferences\fR parameter. +Images in the reference list are matched with those in the input list +in order. If the reference image list is shorter than the number of +input images, the last reference image is used for all remaining input +images. Thus, a single reference image may be given for all the input +images or different reference images may be given for each input +image. The special reference name "last" may be used to select the +last set apertures used in any of the \fBapextract\fR tasks. + +If an aperture reference image is not specified or no apertures are +found for the specified reference image, previously defined apertures +for the input image are sought in the aperture database. Note that +reference apertures supersede apertures for the input image. If no +apertures are defined they may be created automatically, the \fIfind\fR +option, or interactively in the aperture editor, if the +\fIinteractive\fR and \fIedit\fR options are set. + +The functions performed by the task are selected by a set of flag +parameters. The functions are an automatic spectrum finding and +aperture defining algorithm (see \fBapfind\fR) which is ignored if +apertures are already defined, automatic recentering and resizing +algorithms (see \fBaprecenter\fR and \fBapresize\fR), an interactive +aperture editing function (see \fBapedit\fR), a spectrum position tracing +and trace function fit (see \fBaptrace\fR), and the main function of +this task, the one dimensional normalization of the aperture +profiles. + +Each function selection will produce a query for each input spectrum if +the \fIinteractive\fR parameter is set. The queries are answered by +"yes", "no", "YES", or "NO", where the upper case responses suppress +the query for following images. There are other queries associated +with tracing which first ask whether the operation is to be done +interactively and, if yes, lead to queries for each aperture. If the +\fIinteractive\fR parameter is not set then aperture editing, +interactive trace fitting, and interactive spectrum shape fitting are ignored. +.ih +EXAMPLES +To make a flat field image which leaves the total counts of the object +images approximately unchanged from a quartz echelle or slitlet image: + +.nf + cl> apnormalize qtz001,qtz002 flat001,flat002 + Yes find + No resize + No edit + Yes trace + Yes trace interactively + NO + Yes flatten + Yes fit interactively +.fi +.ih +REVISIONS +.ls APNORMALIZE V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +apbackground, approfile, apvariance, apfit, icfit, +apdefault, apfind, aprecenter, apresize, apedit, aptrace, apsum +.endhelp diff --git a/noao/twodspec/apextract/doc/approfiles.hlp b/noao/twodspec/apextract/doc/approfiles.hlp new file mode 100644 index 00000000..43ae774a --- /dev/null +++ b/noao/twodspec/apextract/doc/approfiles.hlp @@ -0,0 +1,131 @@ +.help approfiles Feb93 noao.twodspec.apextract + +.ce +Spectrum Profile Determinations + + +The foundation of variance weighted or optimal extraction, cosmic ray +detection and removal, two dimensional flat field normalization, and +spectrum fitting and modeling is the accurate determination of the +spectrum profile across the dispersion as a function of wavelength. +The previous version of the APEXTRACT package accomplished this by +averaging a specified number of profiles in the vicinity of each +wavelength after correcting for shifts in the center of the profile. +This technique was sensitive to perturbations from cosmic rays +and the exact choice of averaging parameters. The current version of +the package uses two different algorithm which are much more stable. + +The basic idea is to normalize each profile along the dispersion to +unit flux and then fit a low order function to sets of unsaturated +points at nearly the same point in the profile parallel to the +dispersion. The important point here is that points at the same +distance from the profile center should have the nearly the same values +once the continuum shape and spectral features have been divided out. +Any variations are due to slow changes in the shape of the profile with +wavelength, differences in the exact point on the profile, pixel +binning or sampling, and noise. Except for the noise, the variations +should be slow and a low order function smoothing over many points will +minimize the noise and be relatively insensitive to bad pixels such as +cosmic rays. Effects from bad pixels may be further eliminated by +chi-squared iteration and clipping. Since there will be many points +per degree of freedom in the fitting function the clipping may even be +quite aggressive without significantly affecting the profile +estimates. Effects from saturated pixels are minimized by excluding +from the profile determination any profiles containing one or more +saturated pixels as defined by the \fIsaturation\fR parameter. + +The normalization is, in fact, the one dimensional spectrum. Initially +this is the simple sum across the aperture which is then updated by the +variance weighted sum with deviant pixels possibly removed. This updated +one dimensional spectrum is what is meant by the profile normalization +factor in the discussion below. The two dimensional spectrum model or +estimate is the product of the normalization factor and the profile. This +model is used for estimating the pixel intensities and, thence, the +variances. + +There are two important requirements that must be met by the profile fitting +algorithm. First it is essential that the image data not be +interpolated. Any interpolation introduces correlated errors and +broadens cosmic rays to an extent that they may be confused with the +spectrum profile, particularly when the profile is narrow. This was +one of the problems limiting the shift and average method used +previously. The second requirement is that data fit by the smoothing +function vary slowly with wavelength. This is what precludes, for +instance, fitting profile functions across the dispersion since narrow, +marginally sampled profiles require a high order function using only a +very few points. One exception to this, which is sometimes useful but +of less generality, is methods which assume a model for the profile +shape such as a gaussian. In the methods used here there is no +assumption made about the underlying profile other than it vary +smoothly with wavelength. + +These requirements lead to two fitting algorithms which the user +selects with the \fIpfit\fR parameter. The primary method, "fit1d", +fits low order, one dimensional functions to the lines or columns +most nearly parallel to the dispersion. While this is intended for +spectra which are well aligned with the image axes, even fairly large +excursions or tilts can be adequately fit in this +way. When the spectra become strongly tilted then single lines or +columns may cross the actual profile relatively quickly causing the +requirement of a slow variation to be violated. One thought is to use +interpolation to fit points always at the same distance from the +profile. This is ruled out by the problems introduced by +image interpolation. However, there is a clever method which, in +effect, fits low order polynomials parallel to the direction defined by +tracing the spectrum but which does not interpolate the image data. +Instead it weights and couples polynomial coefficients. This +method was developed by Tom Marsh and is described in detail in the +paper, "The Extraction of Highly Distorted Spectra", PASP 101, 1032, +Nov. 1989. Here we refer to this method as the Marsh or "fit2d" +algorithm and do not attempt to explain it further. + +The choice of when to use the one dimensional or the two dimensional +fitting is left to the user. The "fit1d" algorithm is preferable since it +is faster, easier to understand, and has proved to be very robust. The +"fit2d" algorithm usually works just as well but is slower and has been +seen to fail on some data. The user may simply try both to achieve the +best results. + +What follows are some implementation details of the preceding ideas in the +APEXTRACT package. For column/line fitting, the fitting function is a +cubic spline. A base number of spline pieces is set by rounding up the +maximum trace excursion; an excursion of 1.2 pixels would use a spline of 2 +pieces. To this base number is added the number of coefficients in the +trace function in excess of two; i.e. the number of terms in excess of a +linear function. This is done because if the trace wiggles a large amount +then a higher order function will be needed to fit a line or column as the +profile shifts under it. Finally the number of pieces is doubled +because experience shows that for low tilts it doesn't matter but for +large tilts this improves the results dramatically. + +For the Marsh algorithm there are two parameters to be set, the +polynomial order parallel to the dispersion and the spacing between +parallel, coupled polynomials. The algorithm requires that the spacing +be less than a pixel to provide sufficient sampling. The spacing is +arbitrarily set at 0.95 pixels. Because the method always fits +polynomials to points at the same position of the profile the order +should be 1 except for variations in the profile shape with +wavelength. To allow for this the profile order is set at 10; i.e. a +9th order function. A final parameter in the algorithm is the number +of polynomials across the profile but this is obviously determined +from the polynomial spacing and the width of the aperture including an +extra pixel on either side. + +Both fitting algorithms weight the pixels by their variance as computed +from the background and background variance if background subtraction +is specified, the spectrum estimate from the profile and the spectrum +normalization, and the detector noise parameters. A poisson +plus constant gaussian readout noise model is used. The noise model is +described further in \fBapvariance\fR. + +As mentioned earlier, the profile fitting can be iterated to remove +deviant pixels. This is done by rejecting pixels greater than a +specified number of sigmas above or below the expected value based +on the profile, the normalization factor, the background, the +detector noise parameters, and the overall chi square of the residuals. +Rejected points are removed from the profile normalization and +from the fits. +.ih +SEE ALSO +apbackground apvariance apall apsum apfit apflatten +.endhelp diff --git a/noao/twodspec/apextract/doc/aprecenter.hlp b/noao/twodspec/apextract/doc/aprecenter.hlp new file mode 100644 index 00000000..5a05cb36 --- /dev/null +++ b/noao/twodspec/apextract/doc/aprecenter.hlp @@ -0,0 +1,148 @@ +.help aprecenter Sep96 noao.twodspec.apextract +.ih +NAME +aprecenter -- Recenter apertures automatically +.ih +USAGE +aprecenter input +.ih +PARAMETERS +.ls input +List of input images in which apertures are to be recentered. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le +.ls interactive = no +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing is +disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = no +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le + +.ls line = INDEF +The dispersion line (line or column perpendicular to the dispersion axis) to +be used in recentering the spectra. A value of INDEF selects the middle of the +image. +.le +.ls nsum = 1 +Number of dispersion lines to be summed or medianed. The lines are taken +around the specified dispersion line. A positive value takes a sum +and a negative values selects a median. +.le +.ls aprecenter = "" +List of apertures to be used in shift calculation. +.le +.ls npeaks = INDEF +Select the specified number of apertures with the highest peak values +to be recentered. If the number is INDEF all apertures will be selected. +If the value is less than 1 then the value is interpreted as a fraction +of total number of apertures. +.le +.ls shift = yes +Use the median shift from recentering the selected apertures to apply to +all apertures. The recentering is then a constant shift for all apertures. +The median is the average of the two central values for an even number +of apertures. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters are taken from the +task \fBapdefault\fR, automatic aperture finding parameters are taken +from \fBapfind\fR, and parameters used for centering and editing the +apertures are taken from \fBapedit\fR. + +When this operation is performed from the task \fBapall\fR all parameters +except the package parameters are included in that task. +.ih +DESCRIPTION +For each image in the input image list, the aperture center positions +are redefined by centering at the specified dispersion line using the +\fBcenter1d\fR algorithm with centering parameters from \fBapedit\fR. +Normally this is done when inheriting apertures from an aperture +reference image. The recentering does not change the "trace" of the +aperture but simple adds a shift across the dispersion axis. + +There are a several recentering options. Each selected aperture may be +recentered independently. However, if some or all of the spectra are +relatively weak this may actually be worse than using the reference +apertures defined by strong spectra or flat fields in the case of +fibers or aperture masks. One may select a subset of apertures to be +used in calculating shift. This is done with a the \fIaprecenter\fR +list of aperture numbers (see +\fBranges\fR for the syntax) and/or by selecting a specific number or +fraction of the apertures with the strongest peak values. The list +selection is done first and the strongest remaining apertures are used +to satisfy the \fBnpeaks\fR value. Though some or all of the apertures +may be recentered independently the most common case of recentering +reference apertures is to account for detector shifts. In this case +one expects that any shift should be common to all apertures. The +\fIshift\fR parameter allows using the new centers for all selected +apertures to compute a median shift to be added to ALL apertures. Using +a median shift for all apertures is the default. + +The \fIfind\fR parameter allows automatically finding apertures if none +are defined for the image or by a reference image. Since the purpose +of this task is to recenter reference apertures it is usually the case +that reference images are used and apertures are not defined by this +task. One case in which the apertures from the image itself might be +recentered is if one wants to use a different dispersion line. The +\fIresize\fR parameter may be used to adjust the widths in a variety +of ways based on the spectra profiles specific to each image. The +aperture positions and any other parameters may also be edited with the +aperture editing function if selected by the \fIapedit\fR parameter and +the task is run interactively. The recentering algorithm may be run +from the aperture editor using the 'g' keystroke. + +If the task is interactive the user is queried whether to perform +various steps on each image. The queries may be answered with one of +the four values "yes", "no", "YES" and "NO", where an upper case +response suppresses all further queries to this question. + +The aperture recentering algorithm may be selected from nearly every task +in the package. +.ih +EXAMPLES + cl> aprecenter newimage reference=flat +.ih +REVISIONS +.ls APRECENTER V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +center1d, ranges, apfind, apresize, apedit, apall +.endhelp diff --git a/noao/twodspec/apextract/doc/apresize.hlp b/noao/twodspec/apextract/doc/apresize.hlp new file mode 100644 index 00000000..d8ab4774 --- /dev/null +++ b/noao/twodspec/apextract/doc/apresize.hlp @@ -0,0 +1,201 @@ +.help apresize Sep96 noao.twodspec.apextract +.ih +NAME +apresize -- Resize apertures automatically +.ih +USAGE +apresize input +.ih +PARAMETERS +.ls input +List of input images in which apertures are to be resized. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = no +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing is +disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = no +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le + +.ls line = INDEF +The dispersion line (line or column perpendicular to the dispersion axis) to +be used in resizing the spectra. A value of INDEF selects the middle of the +image. +.le +.ls nsum = 1 +Number of dispersion lines to be summed or medianed. The lines are taken +around the specified dispersion line. A positive value takes a +sum and a negative value selects a median. +.le +.ls llimit = INDEF, ulimit = INDEF +Lower and upper aperture size limits. If the parameter \fIylevel\fR is +INDEF then these limits are assigned to all apertures. Otherwise +these parameters are used as limits to the resizing operation. +A value of INDEF places the aperture limits at the image edge (for the +dispersion line used). +.le +.ls ylevel = 0.1 +Data level at which to set aperture limits. If it is INDEF then the +aperture limits are set at the values given by the parameters +\fIllimit\fR and \fIulimit\fR. If it is not INDEF then it is a +fraction of the peak or an actual data level depending on the parameter +\fIpeak\fR. It may be relative to a local background or to zero +depending on the parameter \fIbkg\fR. +.le +.ls peak = yes +Is the data level specified by \fIylevel\fR a fraction of the peak? +.le +.ls bkg = yes +Subtract a simple background when interpreting the \fBylevel\fR parameter. +The background is a slope connecting the first minima +away from the aperture center. +.le +.ls r_grow = 0. +Change the lower and upper aperture limits by this fractional amount. +The factor is multiplied by each limit and the result added to limit. +.le +.ls avglimits = no +Apply the average lower and upper aperture limits to all apertures. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters are taken from the +task \fBapdefault\fR, automatic aperture finding parameters are taken +from \fBapfind\fR, and parameters used for centering and editing the +apertures are taken from \fBapedit\fR. + +When this operation is performed from the task \fBapall\fR all parameters +except the package parameters are included in that task. +.ih +DESCRIPTION +For each image in the input image list, the aperture limits are +redefined to be either specified values or by finding the points at +which the spectrum profile, linearly interpolated, first crosses a +specified value moving away from the aperture center at the specified +dispersion line. In the latter case the limits may then be increased +or decreased by a specified percentage, a maximum lower and upper limit, +may be imposed, and the independent limits may be averaged and the +single values applied to all the apertures. + +The simplest resizing choice is to reset all the aperture limits to +the values specified by \fIllimit\fR and \fIulimit\fR. This option +is selected if the parameter \fIylevel\fR is INDEF. + +There are several options for specifying a data level at which an +aperture is sized. The most common method (the default) is to specify +a fraction of the peak value since this is data independent and physically +reasonable. This is done by setting the fraction with the parameter +\fIylevel\fR and the parameter \fIpeak\fR to yes. If the peak parameter +is no then the level is a data value. + +The levels may be relative to zero, as might be used with fibers or +high dispersion / high signal-to-noise data, or relative to a local +linear background, as would be appropriate for slit data having a +significant background. A background is found and used if the +parameter \fIbkg\fR is set. The background determination is very +simple. Starting at the peak two background points are found, one in +each direction, which are inflection points; i.e. the first pixels +which are less than their two neighbors. A linear slope is fit and +subtracted for the purposes of measuring the peak and setting the +aperture limits. Note that if the slope is significant the actual +limits may not correspond to the intercepts of a line at constant data +value. + +Once aperture limits, a distance relative to the center, are determined +they are increased or decreased by a percentage, expressed as a fraction, +given by the parameter \fIr_grow\fR. To illustrate the operation, +if xlow is the initial lower limit then the final lower limit will be: + + xlow final = xlow * (1 + r_grow) + +A value of zero leaves the aperture limits unchanged. + +After the aperture limits are found, based on the above steps, a fixed lower +limit, given by the parameter \fIllimit\fR, is applied to the lower +aperture points and, similarly, a fixed upper limit is applied to the +upper aperture points. This feature protects against absurdly wide apertures. + +Finally, if the parameter \fIavglimits\fR is set the individual aperture +limits are averaged to form an average aperture. This average aperture +is then assigned to all apertures. This option allows keeping common +aperture sizes but allowing variation due to seeing changes. + +The resizing algorithm is available in the interactive aperture editor. +Here one may select individual apertures or all apertures using the +'a' switch. The resizing algorithm described above is selected using +the 'z' key. An simple alternative is the 'y' key which resizes +apertures to the y level marked by the cursor. + +If the task is interactive the user is queried whether to perform +various steps on each image. The queries may be answered with one of +the four values "yes", "no", "YES" and "NO", where an upper case +response suppresses all further queries to this question. + +The aperture resizing algorithm may be selected from nearly every task +in the package with the \fIresize\fR parameter. +.ih +EXAMPLES +1. To resize all apertures to the range -4 to 4: + + cl> apresize image llimit=-4 ulimit=4 ylevel=INDEF + +2. To resize all aperture to a point which is 5% of the peak relative +to a local background: + + cl> apresize image ylevel=.05 peak+ bkg+ + +3. To resize all apertures to the point where the data exceeds 100 +data units: + + cl> apresize image ylevel=100 peak- bkg- + +4. To resize all apertures to default values of the task except +averaging all the results at the end: + + cl> apresize image avg+ +.ih +REVISIONS +.ls APRESIZE V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +center1d, ranges, apfind, aprecenter, apedit, apall +.endhelp diff --git a/noao/twodspec/apextract/doc/apscatter.hlp b/noao/twodspec/apextract/doc/apscatter.hlp new file mode 100644 index 00000000..902c57a8 --- /dev/null +++ b/noao/twodspec/apextract/doc/apscatter.hlp @@ -0,0 +1,253 @@ +.help apscatter Sep96 noao.twodspec.apextract +.ih +NAME +apscatter -- Fit and subtract scattered light +.ih +USAGE +apscatter input output +.ih +PARAMETERS +.ls input +List of input images in which to determine and subtract scattered light. +.le +.ls output +List of output scattered light subtracted images. If no output images +are specified or the end of the output list is reached before the end +of the input list then the output image will overwrite the input image. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. All apertures are +used to define the scattered light region. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls scatter = "" +List of scattered light images. This is the scattered light subtracted +from the input image. If no list is given or the end of the list is +reached before the end of the input list then no scattered light image +is created. +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing, trace +fitting, and interactive scattered light fitting are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = yes +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le +.ls subtract = yes +Subtract the scattered light from the input images? +.le +.ls smooth = yes +Smooth the cross-dispersion fits along the dispersion? +.le +.ls fitscatter = yes +Fit the scattered light across the dispersion interactively? +The \fIinteractive\fR parameter must also be yes. +.le +.ls fitsmooth = yes +Smooth the cross-dispersion fits along the dispersion? +The \fIinteractive\fR parameter must also be yes. +.le + +.ls line = INDEF, nsum = 1 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. For tracing this is the starting line and +the same number of lines are summed at each tracing point. This is +also the initial line for interactive fitting of the scattered light. +A line of INDEF selects the middle of the image along the dispersion +axis. A positive nsum takes a sum and a negative value selects a +median except that tracing always uses a sum. +.le + +.ls buffer = 1. +Buffer distance from the aperture edges to be excluded in selecting the +scattered light pixels to be used. +.le +.ls apscat1 = "" +Fitting parameters across the dispersion. This references an additional +set of parameters for the ICFIT package. The default is the "apscat1" +parameter set. See below for additional information. +.le +.ls apscat2 = "" +Fitting parameters along the dispersion. This references an additional +set of parameters for the ICFIT package. The default is the "apscat2" +parameter set. See below for additional information. +.le +.ih +ICFIT PARAMETERS FOR FITTING THE SCATTERED LIGHT +There are two additional parameter sets which define the parameters used +for fitting the scattered light across the dispersion and along the +dispersion. The default parameter sets are \fBapscat1\fR and \fBapscat2\fR. +The parameters may be examined and edited by either typing their names +or by typing ":e" when editing the main parameter set with \fBeparam\fR +and with the cursor pointing at the appropriate parameter set name. +These parameters are used by the ICFIT package and a further +description may be found there. + +.ls function = "spline3" (apscat1 and apscat2) +Fitting function for the scattered light across and along the dispersion. +The choices are "legendre" polynomial, "chebyshev" polynomial, +linear spline ("spline1"), and cubic spline ("spline3"). +.le +.ls order = 1 (apscat1 and apscat2) +Number of polynomial terms or number of spline pieces for the fitting function. +.le +.ls sample = "*" (apscat1 and apscat2) +Sample regions for fitting points. Intervals are separated by "," and an +interval may be one point or a range separated by ":". +.le +.ls naverage = 1 (apscat1 and apscat2) +Number of points within a sample interval to be subaveraged or submedianed to +form fitting points. Positive values are for averages and negative points +for medians. +.le +.ls niterate = 5 (apscat1), niterate = 0 (apscat2) +Number of sigma clipping rejection iterations. +.le +.ls low_reject = 5. (apscat1) , low_reject = 3. (apscat2) +Lower sigma clipping rejection threshold in units of sigma determined +from the RMS sigma of the data to the fit. +.le +.ls high_reject = 2. (apscat1) , high_reject = 3. (apscat2) +High sigma clipping rejection threshold in units of sigma determined +from the RMS sigma of the data to the fit. +.le +.ls grow = 0. (apscat1 and apscat2) +Growing radius for rejected points (in pixels). That is, any rejected point +also rejects other points within this distance of the rejected point. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. +.ih +DESCRIPTION +The scattered light outside the apertures defining the two dimensional +spectra is extracted, smoothed, and subtracted from each input image. The +approach is to first select the pixels outside the defined apertures +and outside a buffer distance from the edge of any aperture at each +point along the dispersion independently. A one dimensional function +is fit using the \fBicfit\fR package. This fitting uses an iterative +algorithm to further reject high values and thus fit the minima between +the spectra. (This even works reasonably well if no apertures are +defined). Because each fit is done independently the scattered light +thus determined will not be smooth along the dispersion. If desired +each line along the dispersion in the scattered light surface may then +be smoothed by again fitting a one dimensional function using the +\fBicfit\fR package. The final scattered light surface is then +subtracted from the input image to form the output image. The +scattered light surface may be output if desired. + +The reason for using two one dimensional fits as opposed to a surface fit +is that the actual shape of the scattered light is often not easily modeled +by a simple two dimensional function. Also the one dimensional function +fitting offers more flexibility in defining functions and options as +provided by the \fBicfit\fR package. + +The organization of the task is like the other tasks in the package +which has options for defining apertures using a reference image, +defining apertures through an automatic finding algorithm (see +\fBapfind\fR), automatically recentering or resizing the apertures (see +\fBaprecenter\fR and \fBapresize\fR), interactively editing the +apertures (see \fBapedit\fR), and tracing the positions of the spectra +as a function of dispersion position (see \fBaptrace\fR). Though +unlikely, the actual scattered light subtraction operation may be +suppressed when the parameter \fIsubtract\fR is no. If the scattered +light determination and fitting is done interactively (the +\fIinteractive\fR parameter set to yes) then the user is queried +whether or not to do the fitting and subtraction for each image. The +responses are "yes", "no", "YES", or "NO", where the upper case +queries suppress this query for the following images. When the task is +interactive there are further queries for each step of the operation +which may also be answered both individually or collectively for all +other input images using the four responses. + +When the scattered light operation is done interactively the user may +set the fitting parameters for the scattered light functions both +across and along the dispersion interactively. Initially the central +line or column is used but after exiting (with 'q') a prompt is given +for selecting additional lines or columns and for changing the buffer +distance. Note that the point of the interactive stage is to set the +fitting parameters. When the entire image is finally fit the last set +of fitting parameters are used for all lines or columns. + +The default fitting parameters are organized as separate parameter sets +called \fBapscat1\fR for the first fits across the dispersion and +\fBapscat2\fR for the second smoothing fits along the dispersion. +Changes to these parameters made interactively during execution of +this task are updated in the parameter sets. The general idea for +these parameters is that when fitting the pixels from between the +apertures the iteration and rejection thresholds are set to eliminate +high values while for smoothing along the dispersion a simple smooth +function is all that is required. +.ih +EXAMPLES +1. To subtract the scattered light from a set of images to form a +new set of images: + + cl> apscatter raw* %raw%new%* + +This example uses a substitution in the names from raw to new. +By default this would be done interactively + +2. To subtract the scattered light in place and save the scattered light +images: + + cl> apscatter im* "" scatter="s//im*" ref=im1 interact- + +The prefix s is added to the original names for the scattered light. +This operation is done noninteractively using a reference spectrum +to define the apertures. +.ih +REVISIONS +.ls APSCATTER V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +apfind, aprecenter, apresize, apedit, aptrace, apsum, apmask, icfit +.endhelp diff --git a/noao/twodspec/apextract/doc/apsum.hlp b/noao/twodspec/apextract/doc/apsum.hlp new file mode 100644 index 00000000..6fa7ad0e --- /dev/null +++ b/noao/twodspec/apextract/doc/apsum.hlp @@ -0,0 +1,402 @@ +.help apsum Sep96 noao.twodspec.apextract +.ih +NAME +apsum -- Extract one dimensional sums across the apertures +.ih +USAGE +apsum input +.ih +PARAMETERS +.ls input +List of input images containing apertures to be extracted. +.le +.ls output = "" +List of output rootnames for the extracted spectra. If the null +string is given or the end of the output list is reached before the end +of the input list then the input image name is used as the output rootname. +This will not conflict with the input image since an aperture number +extension is added for onedspec format, the extension ".ms" for multispec +format, or the extension ".ec" for echelle format. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls format = "multispec" (onedspec|multispec|echelle|strip) +Format for output extracted spectra. "Onedspec" format extracts each +aperture to a separate image while "multispec" and "echelle" extract +multiple apertures for the same image to a single output image. +The "multispec" and "echelle" format selections differ only in the +extension added. The "strip" format produces a separate 2D image in +which each column or line along the dispersion axis is shifted to +exactly align the aperture based on the trace information. +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le +.ls profiles = "" +List of profile images for variance weighting or cleanning. If variance +weighting or cleanning a profile of each aperture is computed from the +input image unless a profile image is specified, in which case the +profile is computed from the profile image. The profile image must +have the same dimensions and dispersion and it is assumed that the +spectra have the same position and profile shape as in the object +spectra. Use of a profile image is generally not required even for +faint input spectra but the option is available for those who wish +to use it. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing, trace +fitting, and extraction review are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = no +Recenter the apertures? +.le +.ls resize = no +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le +.ls extract = yes +Extract the one dimensional aperture sums? +.le +.ls extras = no +Extract the raw spectrum (if variance weighting is used), the sky spectrum +(if background subtraction is used), and variance spectrum (if variance +weighting is used)? This information is extracted to the third dimension +of the output image. +.le +.ls review = yes +Review the extracted spectra? The \fIinteractive\fR parameter must also be +yes. +.le + +.ls line = INDEF, nsum = 10 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. For tracing this is the starting line and +the same number of lines are summed at each tracing point. A line of +INDEF selects the middle of the image along the dispersion axis. +A positive nsum takes a sum while a negative value selects a median +except that tracing always uses a sum. +.le + +.ls background = "none" (none|average|median|minimum|fit) +Type of background subtraction. The choices are "none" for no background +subtraction, "average" to average the background within the background +regions, "median" to use the median in the background regions, "minimum" to +use the minimum in the background regions, or "fit" to fit across the +dispersion using the background within the background regions. Note that +the "average" option does not do any medianing or bad pixel checking, +something which is recommended. The fitting option is slower than the +other options and requires additional fitting parameter. +.le +.ls weights = "none" +Type of extraction weighting. Note that if the \fIclean\fR parameter is +set then the weights used are "variance" regardless of the weights +specified by this parameter. The choices are: +.ls "none" +The pixels are summed without weights except for partial pixels at the +ends. +.le +.ls "variance" +The extraction is weighted by the variance based on the data values +and a poisson/ccd model using the \fIgain\fR and \fIreadnoise\fR +parameters. +.le +.le +.ls pfit = "fit1d" (fit1d|fit2d) +Profile fitting algorithm to use with variance weighting or cleaning. +When determining a profile the two dimensional spectrum is divided by +an estimate of the one dimensional spectrum to form a normalized two +dimensional spectrum profile. This profile is then smoothed by fitting +one dimensional functions, "fit1d", along the lines or columns most closely +corresponding to the dispersion axis or a special two dimensional +function, "fit2d", described by Marsh (see \fBapprofile\fR). +.le +.ls clean = no +Detect and replace deviant pixels? +.le +.ls skybox = 1 +Box car smoothing length for sky background when using background +subtraction. Since the background noise is often the limiting factor +for good extraction one may box car smooth the sky to improve the +statistics in smooth background regions at the expense of distorting +the subtraction near spectral features. This is most appropriate when +the sky regions are limited due to a small slit length. +.le +.ls saturation = INDEF +Saturation or nonlinearity level in data units. During variance weighted +extractions wavelength points having any pixels above this value are +excluded from the profile determination and the sigma spectrum extraction +output, if selected by the \fIextras\fR parameter, flags wavelengths with +saturated pixels with a negative sigma. +.le +.ls readnoise = 0. +Read out noise in photons. This parameter defines the minimum noise +sigma. It is defined in terms of photons (or electrons) and scales +to the data values through the gain parameter. A image header keyword +(case insensitive) may be specified to get the value from the image. +.le +.ls gain = 1 +Detector gain or conversion factor between photons/electrons and +data values. It is specified as the number of photons per data value. +A image header keyword (case insensitive) may be specified to get the value +from the image. +.le +.ls lsigma = 4., usigma = 4. +Lower and upper rejection thresholds, given as a number of times the +estimated sigma of a pixel, for cleaning. +.le +.ls nsubaps = 1 +During extraction it is possible to equally divide the apertures into +this number of subapertures. For multispec format all subapertures will +be in the same file with aperture numbers of 1000*(subap-1)+ap where +subap is the subaperture (1 to nsubaps) and ap is the main aperture +number. For echelle format there will be a separate echelle format +image containing the same subaperture from each order. The name +will have the subaperture number appended. For onedspec format +each subaperture will be in a separate file with extensions and +aperture numbers as in the multispec format. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, parameters used for centering and +editing the apertures from \fBapedit\fR, and tracing parameters from +\fBaptrace\fR. + +When this operation is performed from the task \fBapall\fR all +parameters except the package parameters are included in that task. +.ih +DESCRIPTION +For each image in the input image list, the two dimensional spectra are +extracted to one dimensional spectra by summing the pixels across the +dispersion axis at each wavelength along the dispersion axis within a +set of defined apertures. The extraction apertures consist of an +aperture number, a beam number, a title, a center, limits relative to +the center, a curve describing shifts of the aperture center across the +dispersion axis as a function of the wavelength, and parameters for +background fitting and subtraction. See \fBapextract\fR for a more +detailed discussion of the aperture structures. + +The extracted spectra are recorded in one, two, or three dimensional +images depending on the \fIformat\fR and \fIextras\fR parameters. The +output image rootnames are specified by the \fIoutput\fR list. If the +list is empty or shorter than the input list the missing names are +taken to be the same as the input image names. Because the rootnames +have extensions added it is common to default to the input names in +order to preserve a naming relation between the input two dimensional +spectra and the extracted spectra. + +When the parameter \fIextras\fR=no only the extracted spectra are +output. If the format parameter \fIformat\fR="onedspec" the output +aperture extractions are one dimensional images with names formed from +the output rootname and a numeric extension given by the aperture +number; i.e. root.0001 for aperture 1. Note that there will be as many +output images as there are apertures for each input image, all with the +same output rootname but with different aperture extensions. The +aperture beam number associated with each aperture is recorded in the +output image under the keyword BEAM-NUM. The output image name format +and the BEAM-NUM entry in the image are chosen to be compatible with +the \fBonedspec\fR package. + +If the format parameter is "echelle" or "multispec" the output aperture +extractions are put into a two dimensional image with a name formed from +the output rootname and the extension ".ech" or ".ms". Each line in +the output image corresponds to one aperture. Thus in this format +there is one output image for each input image. These are the preferred +output formats for reasons of compactness and ease of handling. These +formats are compatible with the \fBonedspec\fR, \fBechelle\fR, and +\fBmsred\fR packages. The relation between the line and the aperture +numbers is given by the header parameter APNUMn where n is the line and +the value is the aperture number and other numeric information. + +If the \fIextras\fR parameter is set to yes then the above formats +become three dimensional. Each plane in the third dimension contains +associated information for the spectra in the first plane. If variance +weighted extractions are done the unweighted spectra are recorded. If +background subtraction is done the background spectra are recorded. If +variance weighted extractions are done the sigma spectrum (the +estimated sigma of each spectrum pixel based on the individual +variances of the pixels summed) is recorded. The order of the +additional information is as given above. For example, an unweighted +extraction with background subtraction will have one additional plane +containing the sky spectra while a variance weighted extraction with +background subtractions will have the variance weighted spectra, the +unweighted spectra, the background spectra, and the sigma spectra in +consecutive planes. + +Aperture definitions may be inherited from those of other images by +specifying a reference image with the \fBreferences\fR parameter. +Images in the reference list are matched with those in the +input list in order. If the reference image list is shorter than the +number of input images, the last reference image is used for all +remaining input images. Thus, a single reference image may be given +for all the input images or different reference images may be given for +each input image. The special reference name "last" may be used to +select the last set apertures used in any of the \fBapextract\fR tasks. + +If an aperture reference image is not specified or no apertures are +found for the specified reference image, previously defined apertures +for the input image are sought in the aperture database. Note that +reference apertures supersede apertures for the input image. If no +apertures are defined they may be created automatically, the \fIfind\fR +option, or interactively in the aperture editor, if the +\fIinteractive\fR and \fIedit\fR options are set. + +The functions performed by the task are selected by a set of flag +parameters. The functions are an automatic spectrum finding and +aperture defining algorithm (see \fBapfind\fR) which is ignored if +apertures are already defined, automatic recentering and resizing +algorithms (see \fBaprecenter\fR and \fBapresize\fR), an interactive +aperture editing function (see \fBapedit\fR), a spectrum position tracing +and trace function fit (see \fBaptrace\fR), and the main function of +this task, one dimensional spectrum extraction. + +Each function selection will produce a query for each input spectrum if +the \fIinteractive\fR parameter is set. The queries are answered by +"yes", "no", "YES", or "NO", where the upper case responses suppress +the query for following images. There are other queries associated +with tracing and extracted spectrum review which first ask whether the +operation is to be done interactively and, if yes, lead to queries for +each aperture. The cursor keys available during spectrum review are +minimal, only the CURSOR MODE keys for expanding and adjusting the +graph are available and the quit key 'q'. If the \fIinteractive\fR +parameter is not set then aperture editing, interactive trace fitting, +and spectrum review are ignored. + +Background sky subtraction is done during the extraction based on +background regions and parameters defined by the default parameters or +changed during the interactive setting of the apertures. The background +subtraction options are to do no background subtraction, subtract the +average, median, or minimum of the pixels in the background regions, or to +fit a function and subtract the function from under the extracted object +pixels. The background regions are specified in pixels from +the aperture center and follow changes in center of the spectrum along the +dispersion. The syntax is colon separated ranges with multiple ranges +separated by a comma or space. The background fitting uses the \fBicfit\fR +routines which include medians, iterative rejection of deviant points, and +a choice of function types and orders. Note that it is important to use a +method which rejects cosmic rays such as using either medians over all the +background regions (\fIbackground\fR = "median") or median samples during +fitting (\fIb_naverage\fR < -1). The background subtraction algorithm and +options are described in greater detail in \fBapsum\fR and +\fBapbackground\fR. + +Since the background noise is often the limiting factor for good +extraction one may box car smooth the sky to improve the statistics in +smooth background regions at the expense of distorting the subtraction +near spectra features. This is most appropriate when the sky region is +limited due to small slit length. The smoothing length is specified by +the parameter \fIskybox\fR. + +For a more extended discussion about the background determination see +\fBapbackground\fR. + +The aperture extractions consists of summing all the background +subtracted pixel values at a given wavelength within the aperture +limits. The aperture limits form a fixed width aperture but the center +varies smoothly to follow changes in the position of the spectrum +across the dispersion axis. At the ends of the aperture partial pixels +are used. + +The pixels in the sum may be weighted as specified by the \fIweights\fR +parameter. If the weights parameter is "none" and the \fIclean\fR +parameter is no then the simple sum of the pixels (with fractional +endpoints) is extracted. If the weights parameter is "variance" or if +the \fBclean\fR parameter is yes the pixels are weighted by their +estimated variance derived from a noise model based on the \fIgain\fR +and \fIreadnoise\fR parameters and a smooth profile function. Normally +the profile function is determined from the data being extracted. +However, one may substitute a "profile" image as specified by the +\fIprofiles\fR parameter for computing the profile. This requires that +the profile image have spectra of identical position and profile as +the image being extracted. For example, this would likely be the case +with fiber spectra and an off-telescope spectrograph and a strong flat +field or object spectrum could be used for weak spectra. Note that +experience has shown that even for very weak spectra there is little +improvement with using a separate profile image but the user is free +to experiment. + +When the \fIclean\fR parameter is set pixels deviating by more than a +specified number of sigma from the profile function are excluded from the +variance weighted sum. Note that the \fIclean\fR parameter always selects +variance weights. For a more complete discussion of the extraction sums, +variance weighting, cleaning, the noise model, and profile function +determination see \fBapvariance\fR and \fBapprofiles\fR. +.ih +EXAMPLES +1. To simply extract the spectra from a multislit observation: + + cl> apsum multislit1 + +The positions of the slits are defined using either automatic finding +or with the aperture editor. The positions of the slits are traced if +necessary and then the apertures are extracted to the image +"multslit1.ms". The steps of defining the slit positions and tracing +can be done as part of this command or previously using the other tasks +in the \fBapextract\fR package. +.ih +REVISIONS +.ls APSUM V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". + +The "nsubaps" parameter now allows onedspec and echelle output formats. +The echelle format is appropriate for treating each subaperture as +a full echelle extraction. + +The dispersion axis parameter was moved to purely a package parameter. + +As a final step when computing a weighted/cleaned spectrum the total +fluxes from the weighted spectrum and the simple unweighted spectrum +(excluding any deviant and saturated pixels) are computed and a +"bias" factor of the ratio of the two fluxes is multiplied into +the weighted spectrum and the sigma estimate. This makes the total +fluxes the same. In this version the bias factor is recorded in the logfile +if one is kept. Also a check is made for unusual bias factors. +If the two fluxes disagree by more than a factor of two a warning +is given on the standard output and the logfile with the individual +total fluxes as well as the bias factor. If the bias factor is +negative a warning is also given and no bias factor is applied. +In the previous version a negative (inverted) spectrum would result. +.le +.ih +SEE ALSO +apbackground, apvariance, approfile, +apdefault, apfind, aprecenter, apresize, apedit, aptrace, apall +.endhelp diff --git a/noao/twodspec/apextract/doc/aptrace.hlp b/noao/twodspec/apextract/doc/aptrace.hlp new file mode 100644 index 00000000..3b9ddd38 --- /dev/null +++ b/noao/twodspec/apextract/doc/aptrace.hlp @@ -0,0 +1,354 @@ +.help aptrace Sep96 noao.twodspec.apextract +.ih +NAME +aptrace -- Trace spectra for aperture extraction +.ih +USAGE +.nf +aptrace images +.fi +.ih +PARAMETERS +.ls input +List of input images to be traced. +.le +.ls apertures = "" +Apertures to recenter, resize, trace, and extract. This only applies +to apertures read from the input or reference database. Any new +apertures defined with the automatic finding algorithm or interactively +are always selected. The syntax is a list comma separated ranges +where a range can be a single aperture number, a hyphen separated +range of aperture numbers, or a range with a step specified by "x<step>"; +for example, "1,3-5,9-12x2". +.le +.ls references = "" +List of reference images to be used to define apertures for the input +images. When a reference image is given it supersedes apertures +previously defined for the input image. The list may be null, "", or +any number of images less than or equal to the list of input images. +There are three special words which may be used in place of an image +name. The word "last" refers to the last set of apertures written to +the database. The word "OLD" requires that an entry exist +and the word "NEW" requires that the entry not exist for each input image. +.le + +.ls interactive = yes +Run this task interactively? If the task is not run interactively then +all user queries are suppressed and interactive aperture editing and trace +fitting are disabled. +.le +.ls find = yes +Find the spectra and define apertures automatically? In order for +spectra to be found automatically there must be no apertures for the +input image or reference image defined in the database. +.le +.ls recenter = no +Recenter the apertures? +.le +.ls resize = yes +Resize the apertures? +.le +.ls edit = yes +Edit the apertures? The \fIinteractive\fR parameter must also be yes. +.le +.ls trace = yes +Trace the apertures? +.le +.ls fittrace = yes +Interactively fit the traced positions by a function? The \fIinteractive\fR +parameter must also be yes. +.le + +.ls line = INDEF, nsum = 1 +The dispersion line (line or column perpendicular to the dispersion +axis) and number of adjacent lines (half before and half after unless +at the end of the image) used in finding, recentering, resizing, +and editing operations. For tracing this is the starting line and +the same number of lines are summed at each tracing point. A line of +INDEF selects the middle of the image along the dispersion axis. +A positive nsum selects the number of lines to sum while a negative +value selects a median. Tracing always uses a sum. +.le +.ls step = 10 +Step along the dispersion axis between determination of the spectrum +positions. +.le +.ls nlost = 3 +Number of consecutive steps in which the profile is lost before quitting +the tracing in one direction. To force tracing to continue through +regions of very low signal this parameter can be made large. Note, +however, that noise may drag the trace away before it recovers. +.le + +The following parameters are the defaults used to fit the traced positions +by a function of the dispersion line. These parameters are those used by +the ICFIT package. +.ls function = "legendre" +Default trace fitting function. The fitting function types are +"chebyshev" polynomial, "legendre" polynomial, "spline1" linear spline, and +"spline3" cubic spline. +.le +.ls order = 2 +Default trace function order. The order refers to the number of +terms in the polynomial functions or the number of spline pieces in the spline +functions. +.le +.ls sample = "*" +Default fitting sample. The sample is given by a set of colon separated +ranges each separated by either whitespace or commas. The string "*" refers +to all points. +.le +.ls naverage = 1 +Default number of points to average or median. Positive numbers +average that number of sequential points to form a fitting point. +Negative numbers median that number, in absolute value, of sequential +points. A value of 1 does no averaging and each data point is used in the +.le +.ls niterate = 0 +Default number of rejection iterations. If greater than zero the fit is +used to detect deviant traced positions and reject them before repeating the +fit. The number of iterations of this process is given by this parameter. +.le +.ls low_reject = 3., high_reject = 3. +Default lower and upper rejection sigma. If greater than zero traced +points deviating from the fit below and above the fit by more than this +number of times the sigma of the residuals are rejected before refitting. +.le +.ls grow = 0. +Default reject growing radius. Traced points within a distance given by this +parameter of any rejected point are also rejected. +.le +.ih +ADDITIONAL PARAMETERS +I/O parameters and the default dispersion axis are taken from the +package parameters, the default aperture parameters from +\fBapdefault\fR, automatic aperture finding parameters from +\fBapfind\fR, recentering parameters from \fBaprecenter\fR, resizing +parameters from \fBapresize\fR, and parameters used for centering and +editing the apertures from \fBapedit\fR. + +When this operation is performed from the task \fBapall\fR all parameters +except the package parameters are included in that task. +.ih +DESCRIPTION +For each image in the input image list the position of the spectrum +within each aperture are determined at a number of points along the +dispersion axis and a smooth function is fit to these positions. The +fitted curve defines a shift to be added to the aperture center at each +wavelength. Other options allow defining apertures using a reference +image, defining apertures through an automatic finding algorithm (see +\fBapfind\fR), automatically recentering apertures (see +\fBaprecenter\fR), automatically resizing apertures (see +\fBapresize\fR), and interactively editing the apertures prior to +tracing (see \fBapedit\fR). Tracing is selected with the parameter +\fItrace\fR. If the tracing is done interactively (the +\fIinteractive\fR parameter set to yes) then the user is queried +whether or not to trace each image. The responses are "yes", "no", +"YES", or "NO", where the upper case queries suppress this query +for the following images. + +The tracing begins with the specified dispersion line. A dispersion +line is a line or column of the image perpendicular to the dispersion +axis. The dispersion axis is defined in the image header or by the +package parameter \fIdispaxis\fR. If the starting dispersion line is +INDEF then the middle dispersion line of the image is used. The +positions of the spectra are determined using the \fBcenter1d\fR +algorithm and the centering parameters from the \fBapedit\fR task. +(See help under \fBcenter1d\fR for a description of the one dimensional +position measuring algorithm.) The positions are redetermined at other +points along the dispersion axis by stepping from the starting line in +steps specified by the user. A number of dispersion lines around each +dispersion line to be measured may be summed to improve the position +determinations, particularly for weak profiles. This number usually is +set equal to the tracing step. + +It is important to understand how to set the step size and the +relationship between the step size and the centering error radius. +Larger steps reduce the computational time, which is an important +consideration. However, if the step is too large then the tracing may +fail to follow the systematic changes in the positions of the +spectrum. The centering error radius, \fIradius\fR, is used to limit +the maximum position change between two successive steps. If the +positions of a spectrum changes by more than the specified amount or +the data contrast falls below the \fIthreshold\fR parameter then +the position is marked as lost. + +The centering radius should be large enough to follow changes in the +spectrum positions from point to point but small enough to detect an error +in the tracing by a sudden abrupt change in position, such as caused by +crowding with other spectra or by the disappearance of the spectrum. The +\fInlost\fR parameter determines how many consecutive steps the position +may fail to be found before tracing in that direction is stopped. If this +parameter is small the trace will stop quickly upon loss of the profile +while if it is very large it will continue to try and recover the profile. + +The parameter \fIthreshold\fR checks for the vanishing of a spectrum by +requiring a minimum range in the data used for centering. If the +tracing fails when the spectra are strong and well defined the problem +is usually that the step size is too large and/or the centering error +radius is too small. + +The traced positions of a spectrum include some measurement variation +from point to point. Since the actual position of the spectrum in the +image should be a smooth curve, a function of the dispersion line is fit +to the measured points. The fitted function is stored as part of the +aperture description. It is an offset to be added to the aperture's +center as a function of the dispersion line. Even if the fitting is not +done interactively plots of the trace and the fit are recorded in the +plot file or device specified by the parameter \fIplotfile\fR. + +Fitting the traced spectrum positions with a smooth function may be +performed interactively when parameters \fIfittrace\fR and +\fIinteractive\fR are yes. This allows changing the default fitting +parameters. The function fitting is done with the interactive curve +fitting tools described under the help topic \fBicfit\fR. There are +two levels of queries when fitting the spectrum positions +interactively; prompts for each image and prompts for each aperture in +an image. These prompts may be answered individually with the lower +case responses "yes" or "no" or answered for all further prompts with +the responses "YES" or "NO". Responding with "yes" or "YES" to the +image prompt allows interactive fitting of the traced positions for the +spectra. Prompts are then given for each aperture in the image. When +an spectrum is not fit interactively the last set of fitting parameters +are used (initially the default function and order given by the task +parameters). Note that answering "YES" or "NO" to a aperture prompt +applies to all further aperture in the current image only. Responding +with "no" or "NO" to the image prompt fits the spectrum positions for +all apertures in all images with the last set of fitting parameters. + +The tracing may also be done from the interactive aperture editor with +the 't' key. The aperture tracing algorithm may be selected from many +of the tasks in the package with the \fItrace\fR parameter. +.ih +APTRACE DATABASE COEFFICIENTS +The path of an aperture is described by a function that gives an additive +offset relative to the aperture center as stored under the database keyword +center. The function is saved in the database as a series of +coefficients. The section containing the coefficients starts with the +keyword "curve" and the number of coefficients. + +The first four coefficients define the type of function, the order +or number of spline pieces, and the range of the independent variable +(the line or column coordinate along the dispersion). The first +coefficient is the function type code with values: + +.nf + Code Type + 1 Chebyshev polynomial + 2 Legendre polynomial + 3 Cubic spline + 4 Linear spline +.fi + +The second coefficient is the order (actually the number of terms) of +the polynomial or the number of pieces in the spline. + +The next two coefficients are the range of the independent variable over +which the function is defined. These values are used to normalize the +input variable to the range -1 to 1 in the polynomial functions. If the +independent variable is x and the normalized variable is n, then + +.nf + n = (2 * x - (xmax + xmin)) / (xmax - xmin) +.fi + +where xmin and xmax are the two coefficients. + +The spline functions divide the range into the specified number of +pieces. A spline coordinate s and the nearest integer below s, +denoted as j, are defined by + +.nf + s = (x - xmin) / (xmax - xmin) * npieces + j = integer part of s +.fi + +where npieces are the number of pieces. + +The remaining coefficients are those for the appropriate function. +The number of coefficients is either the same as the function order +for the polynomials, npieces+1 for the linear spline, or npieces + 3 +for the cubic spline. + +1. Chebyshev Polynomial + +The polynomial can be expressed as the sum + +.nf + y = sum from i=1 to order {c_i * z_i} +.fi + +where the c_i are the coefficients and the z_i are defined +interactively as: + +.nf + z_1 = 1 + z_2 = n + z_i = 2 * n * z_{i-1} - z_{i-2} +.fi + +2. Legendre Polynomial + +The polynomial can be expressed as the sum + +.nf + y = sum from i=1 to order {c_i * z_i} +.fi + +where the c_i are the coefficients and the z_i are defined +interactively as: + +.nf + z_1 = 1 + z_2 = n + z_i = ((2*i-3) * n * z_{i-1} - (i-2) * z_{i-2}) / (i - 1) +.fi + +3. Linear Spline + +The linear spline is evaluated as + +.nf + y = c_j * a + c_{j+1} * b +.fi + +where j is as defined earlier and a and b are fractional difference +between s and the nearest integers above and below + +.nf + a = (j + 1) - s + b = s - j +.fi + +4. Cubic Spline + +The cubic spline is evaluated as + +.nf + y = sum from i=0 to 3 {c_{i+j} * z_i} +.fi + +where j is as defined earlier. The term z_i are computed from +a and b, as defined earlier, as follows + +.nf + z_0 = a**3 + z_1 = 1 + 3 * a * (1 + a * b) + z_2 = 1 + 3 * b * (1 + a * b) + z_3 = b**3 +.fi +.ih +EXAMPLES +.ih +REVISIONS +.ls APTRACE V2.11 +The "apertures" parameter can be used to select apertures for resizing, +recentering, tracing, and extraction. This parameter name was previously +used for selecting apertures in the recentering algorithm. The new +parameter name for this is now "aprecenter". +.le +.ih +SEE ALSO +apdefault, apfind, aprecenter, apresize, apedit, apall, +center1d, icfit, gtools +.endhelp diff --git a/noao/twodspec/apextract/doc/apvariance.hlp b/noao/twodspec/apextract/doc/apvariance.hlp new file mode 100644 index 00000000..6ff1e073 --- /dev/null +++ b/noao/twodspec/apextract/doc/apvariance.hlp @@ -0,0 +1,159 @@ +.help apvariance Aug90 noao.twodspec.apextract + +.ce +Variance Weighted and Cleaned Extractions + + +There are two types of aperture extraction (estimating the background +subtracted flux across a fixed width aperture at each image line or +column) in the APEXTRACT package. One is a simple sum of pixel values +across an aperture. It is selected by specifying "none" for the +\fIweights\fR parameter. The second type weights each pixel in the sum +by it's estimated variance based on a spectrum model and detector noise +parameters. This type of extraction is selected by specifying +"variance" for the weighting parameter. These two extractions are +defined by the following equations. + +.nf + none: S = sum { I - B } + variance: S = sum { (P**2 / V) (I - B) / P } / sum { P**2 / V } +.fi + +S is the one dimensional spectrum flux at a particular wavelength (line +or column along the dispersion axis). The sum is over all pixels at +that wavelength within the aperture limits. If the aperture endpoints +occupy only a fraction of a pixel then the pixel value above the +background is multiplied by the fraction. I is the pixel value and B +is the estimated background at that pixel (see \fBapbackground\fR), P +is estimated normalized profile value for that pixel (see +\fBapprofile\fR), and V is the estimated variance of the pixel based on +the noise model described below. Note that the quantity (I-B)/P is an +independent estimate of the total flux from one pixel since the +integral of P is one and it is these estimates that are variance +weighted. + +Variance weighting is often called "optimal" extraction since it +produces the best unbiased signal-to-noise estimate of the flux in the +two dimensional profile. The theory and application of this type of +weighting has been described in several papers. The ones which were +closely examined and used as a model for the algorithms in this +software are "An Optimal Extraction Algorithm for CCD Spectroscopy", +PASP 98, 609, 1986, by Keith Horne and "The Extraction of Highly +Distorted Spectra", PASP 100, 1032, 1989, by Tom Marsh. + +The noise model for the image data used in the variance weighting, +cleaning, and profile fitting consists of a constant gaussian noise and +a photon count dependent poisson noise. The signal is related to the +number of photons detected in a pixel by a \fRgain\fR parameter given +as the number of photons per data number. The gaussian noise is given +by a \fIreadnoise\fR parameter which is a defined as a sigma in +photons. The poisson noise is approximated as gaussian with sigma +given by the number of photons. + +Some additional effects which should be considered in principle, and +which are possibly important in practice, are that the variance +estimate should be based on the actual number of photons detected before +correction for pixel sensitivity; i.e. before flat field correction. +Furthermore the uncertainty in the flat field should also be included +in the weighting. However, the profile must be determined free of +sensitivity effects including rapid larger scale variations such as +fringing. Thus, ideally one should input the unflat-fielded +observation and the flat field data and carry out the extractions with +the above points in mind. However, due to the complexity often +involved in basic CCD reductions and special steps required for +producing spectroscopic flat fields this level of sophistication is not +provided by the current package. The package does provide, however, +for propagation of an approximate uncertainty in the background +estimate when using background subtraction. + +The noise model is described by the following equations. + +.nf + (1) V = max (VMIN, (R**2 + I + VB) / G**2) + max (VMIN, (R**2 + S * P + B + VB) / G**2) + + (2) VB = 0. if (B = 0) + = B / (N - 1) if (B > 0) + + (3) VMIN = 1 / G**2 if (R = 0) + R**2 / G**2 if (R > 0) +.fi + +V is the desired variance of a pixel to use for variance weighting. R +is the photon read out noise specified by the parameter \fIreadnoise\fR +and G is the photon per data value gain specified by the parameter +\fIgain\fR. There are two forms to (1). The first is used in the +initial pass of estimating the spectrum flux S and the actual pixel +value I (which includes any background) is used for the poisson term. +The other form is used in a second pass (and further passes if +cleaning) using the estimated data value based on the normalized +profile P scaled to the estimated total flux plus the estimated +background B; i.e. I estimated = S * P + B. + +The background variance VB is computed using the poisson noise model +based on the estimated background counts. If no background subtraction +is done then both B and VB are set to zero. If a background is +determined the background is either an average or function fit to +pixels in defined background regions. If a fit is used B need not be a +constant. Because the background estimate is based on a finite number of +pixels, the poisson variance estimate is divided by the number N (minus +one) of pixels used in determining the background. The number of +pixels used includes any box car smoothing. Thus, the larger the +number of background pixels the smaller the background noise +contribution to the variance weighting. This method is only +approximate since no correction is made for the number of degrees of +freedom and correlations when using the fitting method of background +estimation. + +VMIN is a minimum variance need to avoid generating zero or negative +variances from the data. The definition of VMIN is such that if a zero +read out noise is specified (which is certainly possible such as with +photon counting detectors) then a minimum of 1 photon is imposed. +Otherwise the minimum is set by the read out noise even if the poisson +count part is (unphysically) negative. + +One deviation from the linear photon response mode which is considered +is saturation. A data level specified by the parameter +\fIsaturation\fR is used to exclude data from the profile fitting. +During extraction the saturated pixels are not treated any differently +than unsaturated pixels except that dispersion points with saturated +pixels are flagged by reversing the sign of the final estimated sigma; +the sigma output is enabled with the \fIextras\fR parameter. Exclusion +of saturated pixels from the extraction, as is done with deviant +pixels, was tried but this resulted in higher noise in the spectrum. + +If removal of cosmic rays and other deviant pixels is desired, called +cleaning and selected with a \fIclean\fR parameter, they are +iteratively rejected based on the estimated variance and excluded from +the weighted sum. Note that a cleaned extraction is always variance +weighted regardless of the value of the \fIweights\fR parameter. This +makes sense since the detector noise parameters must be specified and +the spectrum profile computed, so all of the computational effort must +be done anyway, and the variance weighting is as good or superior to a +simple unweighted extraction. + +The detection and removal of deviant pixels is straightforward. Based +on the noise model described earlier, pixels deviating by more than a +specified number of sigma (square root of the variance) above or below +the model are removed from the weighted sum. A new spectrum estimate +is made and the rejection is repeated. The rejections are made one at +a time starting with the most deviant and up to half the pixels in the +aperture may be rejected. The total number of rejected pixels in the +spectrum is recorded in the logfile and a profile plot of data and +model profile is recorded in the plotfile. + +As a final step when computing a weighted/cleaned spectrum the total +fluxes from the weighted spectrum and the simple unweighted spectrum +(excluding any deviant and saturated pixels) are computed and a +"bias" factor of the ratio of the two fluxes is multiplied into +the weighted spectrum and the sigma estimate. This makes the total +fluxes the same. The bias factor is recorded in the logfile +if one is kept. Also a check is made for unusual bias factors. +If the two fluxes disagree by more than a factor of two a warning +is given on the standard output and the logfile with the individual +total fluxes as well as the bias factor. If the bias factor is +negative a warning is also given and no bias factor is applied. +.ih +SEE ALSO +apbackground approfiles apall apsum +.endhelp diff --git a/noao/twodspec/apextract/doc/dictionary b/noao/twodspec/apextract/doc/dictionary new file mode 100644 index 00000000..1046499c --- /dev/null +++ b/noao/twodspec/apextract/doc/dictionary @@ -0,0 +1,282 @@ +ADU +APALL +APAXIS +APEDIT +APEXTRACT +APFIND +APFIT +APFLATTEN +APFORMAT +APID +APID2 +APIO +APMASK +APNORMALIZE +APNUM2 +APNUMn +APPARAMS +APRECENTER +APRESIZE +APSCATTER +APSTRIP +APSUM +APTRACE +CCD +CL +DISPAXIS +ECHELLE +EOF +EPARAM +FIT1D +FLAT1D +Fri +Horne +Horne's +ICFIT +IMARITH +IMREPLACE +IMSURFIT +INDEF +IRAF +Jul +Jul90 +Nfind +P.O +PASP +PSET +RMS +SETDISP +SN +STDIN +STDOUT +Slitlet +VB +VMIN +Valdes +ansclob +ansclobber +ansclobber1 +ansdbwr +ansdbwrite +ansdbwrite1 +ansedit +ansextr +ansextract +ansfind +ansfit +ansfits +ansfitscatter +ansfitsmooth +ansfitspec +ansfitspec1 +ansfitt +ansfittrace +ansfittrace1 +ansflat +ansmask +ansnorm +ansrece +ansrecenter +ansresi +ansresize +ansrevi +ansreview +ansreview1 +ansscat +anssmoo +anssmooth +anstrac +anstrace +ap +apall +apall1 +apbackground +apdefault +apdefault.apidtable +apdefault.b +apdefault.lower +apdefault.upper +apdemo1d +apdemo2d +apdemo2d.ms +apdemos +apdemosdb +apedit +apedit.radius +apedit.threshold +apedit.width +apertur +apextract +apextractsys +apfind +apfind.maxsep +apfind.minsep +apfind.nfind +apfind.order +apfit +apfit1 +apflat1 +apflatten +apidtab +apidtable +apio +aplast +apmask +apnorm1 +apnormalize +apparams +approfile +approfiles +aprecenter +aprecenter.apertures +aprecenter.npeaks +aprecenter.shift +apresize +apresize.avglimits +apresize.bkg +apresize.llimit +apresize.peak +apresize.r +apresize.ulimit +apresize.ylevel +apscat1 +apscat2 +apscatter +apscript +apstrip +apsum +apsum.background +apsum.clean +apsum.extras +apsum.gain +apsum.lsigma +apsum.nsubaps +apsum.readnoise +apsum.saturation +apsum.skybox +apsum.usigma +apsum.weights +aptrace +aptrace.function +aptrace.grow +aptrace.high +aptrace.low +aptrace.naverage +aptrace.niterate +aptrace.nsum +aptrace.order +aptrace.sample +aptrace.step +apvariance +artdata +avg +avglimi +avglimits +backgro +bkg +ccd +cennorm +center1d +chebyshev +cl +clopset +computerese +curfit +dbwrite +dispaxi +dispaxis +dropoff +ech +ech001 +echelle +echelles +elp +eparam +fiber1 +fitscatter +fitsmooth +fitspec +fittrace +fittype +flat001,flat002 +funct +gaussian +gkimosaic +gtools +icfit +im +im1 +image.pl +image1 +imarith +imh +imred +imred.generic.flat1d +imsurfit +keystroke +legendre +llimit +logfile +longslit +lparam +ls1 +lsigma +maxsep +maxtilt +minsep +mk1dspec +mk2dspec +mknoise +msred +multislit1 +multspec.ms +naverage +ndhelp +nessie +newimage +nfind +niter +niterat +niterate +nl +noao.twodspec.apextract +npeaks +nsubaps +nsum +onedspec +onedspec.continuum +pl +plotfile +poisson +polyord +polysep +pset +psets +qtz001,qtz002 +rdnoise +readnoi +readnoise +rec +ref +res +root.0001 +rootname +rootnames +sampl +saturat +setdisp +skybox +slitlet +slitlets +spline1 +spline3 +thresho +twodspec +ulimit +usigma +whitespace +widt +xlow +xmax +xmin +ylevel diff --git a/noao/twodspec/apextract/doc/old/Tutorial.hlp b/noao/twodspec/apextract/doc/old/Tutorial.hlp new file mode 100644 index 00000000..fd0ff8e8 --- /dev/null +++ b/noao/twodspec/apextract/doc/old/Tutorial.hlp @@ -0,0 +1,278 @@ +.help Tutorial Sep86 "Apextract Tutorial" +.ih +TOPICS +The APEXTRACT tutorial consists of a number of topics. The topics are brief +and describe the simplest operations. More sophisticated discussions are +available for the tasks in the printed documentation and through the on-line +\fBhelp\fR facility; i.e. "help taskname". To obtain information +on a particular topic type "tutor topic" where the topic is one of the +following: + +.nf + TOPICS + + topics - List of topics + overview - An overview of the \fBapextract\fR tasks + organization - How the package is organized + apertures - Definition of apertures + defining - How to define apertures + references - Using reference images to define apertures + queries - Description of interactive queries + cosmic - Problems with cosmic ray removal + all - Print all of this tutorial +.fi +.ih +OVERVIEW +The \fBapextract\fR tasks extract spectra from two dimensional images. +One image axis is the dispersion axis and the other image axis is the +aperture axis. The user defines apertures whose position along the +aperture axis is a function of position along the dispersion axis and +whose width is fixed. There are two types of aperture extractions. +\fIStrip\fR extraction produces two dimensional images in which the +center of the aperture is exactly centered along one of the lines or +columns of the image and the edges of the image just include the +edges of the aperture. \fISum\fR extraction sums the pixels across +the aperture at each point along the dispersion to produce a one +dimensional spectrum. The extraction algorithms include +fitting and subtracting a background, modeling the profiles across the +dispersion, detecting and removing deviant pixels which do not fit the +model profiles, and weighting the pixels in the sum extraction according +to the signal-to-noise. + +To extract spectra one must define the dispersion axis by placing the +parameter DISPAXIS in the image headers using the task \fBsetdisp\fR. +Then apertures are defined either automatically, interactively, or by +reference to an image in which apertures have been previously defined. +Initially the apertures are aligned parallel to the dispersion axis +but if the spectra are not aligned with the dispersion axis and have +profiles which can be traced then the position of the aperture along +the aperture axis can be made a function of position along the dispersion +axis. Finally, the extraction operation is performed for each aperture. +.ih +ORGANIZATION +The tasks in the \fBapextract\fR package are highly integrated. This +means that tasks call each other. For example, the aperture +editing task may be called from the finding, tracing, or extraction +tasks. Also from within the aperture editor the finding, tracing, and +extraction tasks may be run on selected apertures. This organization +provides the flexibility to process images either step-by-step, +image-by-image, or very interactively from the aperture editor. For +example, one may defined apertures for all the images, trace all the +images, and then extract all the images or, alternatively, define, +trace, and extract each image individually. + +This organization also implies that parameters from many tasks are used +during the execution of a single task. For example, the editing +parameters are used in any of the tasks which may enter the interactive +editing task. Two tasks, \fBapio\fR and \fBapdefault\fR, only set +parameters but these parameters are package parameters which affect all +the other tasks. There are two effects of this parameter +organization. First, only parameters from the task being executed may +be specified on the command line or with menu mode. However, the +parameters are logically organized and the parameter list for any +particular task is not excessively long or complex. For example, the +number of parameters potentially used by the task \fBapsum\fR is 57 +parameters instead of just the parameters logically related to the +extraction itself. + +Another feature of the package organization is the ability to +control the flow and interactivity of the tasks. The parameter +\fIinteractive\fR selects whether the user will be queried about various +operations and if the aperture editor, trace fitting, and extraction +review will be performed. The parameters \fBdbwrite, +find, recenter, edit, trace, fittrace, sum, review\fR, and +\fBstrip\fR select which operations may be performed by a particular +task. When a task is run interactively the user is queried about +whether to perform each operation on each image. A query may be answered +individually or as a group. In the latter case the query will not be +repeated for other images but will always take the specified action. +This allows the user to begin interactively and then reduce +the interactivity as the images are processed and parameters are refined. +For additional discussion of these parameters see the topic QUERIES. + +Finally, the package has attempted to provide good logging facilities. +There are log files for both time stamped text output and plots. +The text log is still minimal but the plot logging is complete +and allows later browsing and hardcopy review of batch processing. +See \fBapio\fR for further discussion. + +This package organization is somewhat experimental. Let us know what +you think. +.ih +APERTURES +An aperture consists of the following elements: + +.ls id +An integer aperture identification number. The identification number +must be unique. The aperture number is used as the default extension +of the extracted spectra. +.le +.ls beam +An integer beam number. The beam number need not be unique; i.e. +several apertures may have the same beam number. The beam number will +be recorded in the image header of the extracted spectrum. Note that +the \fBonedspec\fR package restricts the beam numbers to the range 0 to +49. +.le +.ls cslit, cdisp +The center of the aperture along the slit and dispersion axes in the two +dimensional image. +.le +.ls lslit, ldisp +The lower limits of the aperture, relative to the aperture center, +along the slit and dispersion axes. The lower limits need not be less +than the center. +.le +.ls uslit, udisp +The upper limits of the aperture, relative to the aperture center, +along the slit and dispersion axes. The upper limits need not be greater +than the center. +.le +.ls curve +An shift to be added to the center position for the slit axis which is +a function of position along the dispersion axis. The function is one +of the standard IRAF \fBicfit\fR types; a legendre polynomial, a chebyshev +polynomial, a linear spline, or a cubic spline. +.le +.ls background +Background fitting parameters used by the \fBicfit\fR package for background +subtraction. Background parameters need not be used if background +subtraction is not needed. The background sample regions are specified +relative to aperture center. +.le + +The aperture center is the only absolute coordinate relative to the +image or image section. The size and shape of the aperture are +specified relative to the aperture center. The center and aperture +limits in image coordinates along the slit axis are functions of the +dispersion coordinate, lambda, given by + +.nf + center(lambda) = cslit + curve(lambda) + lower(lambda) = center(lambda) + lslit + upper(lambda) = center(lambda) + uslit +.fi + +Note that both the lower and upper constants are added to the center +defined by the aperture center and the curve offset. The aperture limits +along the dispersion axis are constant, + +.nf + center(s) = cdisp + lower(s) = center(s) + ldisp + upper(s) = center(s) + udisp +.fi + +Usually the aperture size along the dispersion is equal to the entire image. +.ih +DEFINING APERTURES +If a reference image is specified the \fBapextract\fR tasks first search +the database for it's apertures. Note that this supercedes any apertures +previously defined for the input image. If no reference apertures are +found then the apertures for the input image are sought. +If no apertures are defined at this point then apertures +may be defined automatically, interactively, or, by default, in the center +of the image. The automatic method, \fBapfind\fR, locates spectra as peaks +across the dispersion and then defines default apertures given by the +parameters from \fBapdefault\fR. The algorithm is controlled +by specifying the number of apertures and a minimum separation between +spectra. Only the strongest peaks are selected. + +The interactive method, \fBapedit\fR, allows the user to mark the positions +of apertures and to adjust the aperture parameters such as the limits. +The aperture editor may be used edit apertures defined by any of the +other methods. + +If no apertures are defined when tracing or extraction is begun, that is +following the optional editing, then a default aperture is defined +centered along the aperture axis of the image. This ultimate default +may be useful for spectra defined by image sections; i.e. the image +section is a type of aperture. Image sections are sometimes used with +multislit spectra. +.ih +REFERENCE IMAGES +The \fBapextract\fR tasks define apertures for an input image by +first searching the database for apertures recorded under the name +of the reference image. Use of a reference image implies +superceding the input image apertures. Reference images are specified +by an image list which is paired with +the input image list. If the number of reference images +is less than the number of input images then the last reference image +is used for all following images. Generally, the reference image list +consists of the null string if reference images are not to be used, +a single image which is applied to all input images, or a list +which exactly matches the input list. The special reference image +name "last" may be used to refer to the last apertures written to +the database; usually the previous input image. + +The task parameter \fIrecenter\fR specifies whether the +reference apertures are to be recentered on the spectra in the input +image. If recentering is desired the \fBcenter1d\fR centering algorithm +is used with centering parameters taken from the task \fBapedit\fR. +The spectra in the image must all have well defined profiles for the +centering. It does not make sense to center an aperture defined for +a region of sky or background or for an arc spectrum. + +Recentering is used when the only change between two spectra is +a shift along the aperture axis. This can reduce the number of +images which must be traced if tracing is required by using a +traced reference image and just recentering on the next spectra. +Recentering of a traced reference image is also useful when +the spectra are too weak to be traced reliably. Recentering would be +most commonly used with echelle or multiaperture spectra. + +Recentering is not used when extracting sky or arc calibration spectra +from long slit or multislit images. This is because it is desirable +to extract from the same part of the detector as the object spectra and +because recentering does not make sense when there is no profile across +the aperture. Centering or recentering is also not used when dealing +with apertures covering parts of extended objects in long slit spectra. +.ih +QUERIES +When the interactive parameter is specified as yes in a task then the user +is queried at each step of the task. The queries refer to either a +particular image or a particular aperture in an image. The acceptable +responses to the queries are the strings "yes", "no", "YES", and "NO". +The lower case answers refer only to the specific query. The upper +case answers apply to all repetitions of query for other images and +apertures. The upper case reponses then suppress the query and take +the specified action every time. This allows tasks to be highly interactive +by querying at each step and for each image or to skip or perform +each step for all images without queries. + +The two steps of fitting a function to traced positions and reviewing +one dimensional extracted spectra, selected with the parameters +\fIaptrace.fittrace\fR and \fIapsum.review\fR have two levels of queries. +First a query is made for the image being traced or extracted. If +the answer is "yes" or "YES" then a query is made for each aperture. +A response of "YES" or "NO" applies only to the remaining apertures +and not to apertures of a later image. +.ih +COSMIC RAYS +The cleaning and modeling features available during aperture extraction +are fairly good. They are described in the documentation for the +tasks. It can only go so far towards discriminating cosmic rays +because of problems described below. Further work on the algorithm may +improve the performance but it is best, when feasible, to first +eliminate at least the strongest cosmic rays from the data before +extracting. One recommended method is to use \fBlineclean\fR with a +high rejection threshold and a high order. + +There are two difficult problems encountered in using the +\fBapextract\fR tasks for cosmic ray detection. First, the spectral +profiles are first interpolated to a common center before comparison +with the average profile model. The interpolation often splits single +strong spikes into two high points of half the intensity, as is +intuitively obvious. Furthermore, for very strong spikes there is +ringing in the interpolator which makes the interpolated profile depart +significantly from the original profile. The fact that the +interpolated profile now has two or more deviant points makes it much +harder to decide which points are in the profile. This leads to the +second problem. The average profile model is scaled to fit the +spectrum profile. When there are several high points it is very +difficult to discriminate between a higher scale factor and bad +points. The algorithm has been enhanced to initially exclude the point which +most pulls the scale factor to higher values. If there are two high +points due to the interpolator splitting a strong spike then this helps +but does not eliminate errors in the extracted spectra. +.endhelp diff --git a/noao/twodspec/apextract/doc/old/apextract.ms b/noao/twodspec/apextract/doc/old/apextract.ms new file mode 100644 index 00000000..3e71890b --- /dev/null +++ b/noao/twodspec/apextract/doc/old/apextract.ms @@ -0,0 +1,725 @@ +.EQ +delim $$ +define sl '{s lambda}' +.EN +.RP +.TL +The IRAF APEXTRACT Package +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +.AB +The IRAF \fBapextract\fR package provides tools for the extraction of +one and two dimensional spectra from two dimensional images +such as echelle, long slit, multi-fiber, and multi-slit spectra. +Apertures of fixed width along the spatial define the regions of +the two dimensional images to be extracted at each point along the +dispersion axis. Apertures may follow changes in the positions of +the spectra as a function of position along the dispersion axis. +The spatial and dispersion axes may be oriented along either image axis. +Extraction to one dimensional spectra consists of a weighted sum of the pixels +within the apertures at each point along the dispersion axis. The +weighting options provide the simple sum of the pixel values and a +weighting by the expected uncertainty of each pixel. Two dimensional +extractions interpolate the spectra in the spatial axis to produce +image strips with the position of the spectra exactly aligned with one +of the image dimensions. The extractions also include optional +background subtraction, modeling, and bad pixel detection and replacement. +The tasks are flexible in their ability to define and edit apertures, +operate on lists of images, use apertures defined for reference +images, and operate both very interactively or noninteractively. +The extraction tasks are efficient and require only one pass through +the data. This paper describes the tasks, the algorithms, the data +structures, as well as some examples and possible future developments. +.AE +.NH +Introduction +.PP +The IRAF \fBapextract\fR package provides tools for the extraction of +one and two dimensional aperture spectra from two dimensional format +images such as those produced by echelle, long slit, multi-fiber, and +multi-slit spectrographs. This type of data is becoming increasingly +popular because of the efficiency of data collection and recent +technological improvements such as fibers and digital detectors. +The trend is also to greater and greater numbers of spectra per +image. Extraction is one of the fundamental operations performed +on these types of two dimensional spectral images, so a great deal of effort +has gone into the design and development of this package. +.PP +The tasks are flexible and have many options. To make the best use of +them it is important to understand how they work. This paper provides +a general description of the tasks, the algorithms, the data structures, +as well as some examples of usage. Specific descriptions of parameters +and usage may be found in the IRAF help pages for the tasks included +as appendices to this paper. The image reduction "cookbooks" also +provide complete examples of usage for specific instruments or types +of instruments. +.PP +The tasks in the \fBapextract\fR pacakge are summarized below. + +.ce +The \fBApextract\fR Package +.TS +center; +n. +apdefault \&- Set the default aperture parameters +apedit \&- Edit apertures interactively +apfind \&- Automatically find spectra and define apertures +apio \&- Set the I/O parameters for the APEXTRACT tasks +apnormalize \&- Normalize 2D apertures by 1D functions +apstrip \&- Extract two dimensional aperture strips +apsum \&- Extract one dimensional aperture sums +aptrace \&- Trace positions of spectra +.TE + +The tasks are highly integrated so that one task may call another tasks +or use its parameters. Thus, these tasks reflect the logical organization +of the package rather than a set of disparate tools. One reason for +this organization is group the parameters by function into easy to manage +\fIparameter sets (psets)\fR. The tasks \fBapdefault\fR and \fBapio\fR +are just psets for specifying the default aperture parameters and the +I/O parameters of the package; in other words, they do nothing but +provide a grouping of parameters. Executing these tasks is a shorthand +for the command "eparam apdefault" or "eparam apio". The other tasks +provide both a logical grouping of parameters and function. For +example the task \fBaptrace\fR traces the positions of the spectra +in the images and has the parameters related to tracing. The task +\fBapsum\fR, however, may trace the spectra as part of the overall +extraction process and it uses the functionality and parameters of +the \fBaptrace\fR task without requiring all the tracing parameters +be included as part of its parameter set. As we examine each task +in detail it will become more apparent how this integration of function +and parameters works. +.PP +The \fBapextract\fR package identifies the image axes with the spatial +and dispersion axes. Thus, during extraction pixels of constant +wavelength are those along a line or column. In this paper the terms +\fIslit\fR or \fIspatial\fR axis and \fIdispersion\fR or \fIwavelength\fR +axis are used to refer to the image axes corresponding to the spatial +and dispersion axes. Often a small degree of misalignment between the +image axes and the true dispersion and spatial axes is not important. +The main effect of misalignment is a broadening of the spectral +features due to the difference in wavelength on opposite sides of the +extraction aperture. If the misalignment is significant, however, the +image may be rotated with the task \fBrotate\fR in the \fBimages\fR +package or remapped with the \fBlongslit\fR package tasks for +coordinate rectification. +.PP +It does not matter which image axis is the dispersion axis since the +tasks work equally well in either orientation. However, the dispersion +axis must be defined, with the \fBtwodspec\fR task \fBsetdisp\fR, +before these tasks may be used. This task is a simple script which +adds the parameter DISPAXIS to the image headers. The \fBapextract\fR +tasks, like the \fBlongslit\fR tasks, look in the header to determine +the dispersion axis. +.NH +Apertures +.PP +Apertures are the basic data structures used in the package; hence the +package name. An aperture defines a region of the two dimensional image +to be extracted. The aperture definitions are stored in a database. +An aperture consists of the following components. + +.IP ID +.br +An integer identification number. The identification number must be +unique. It is used as the default extension during extraction of +the spectra. Typically the IDs are consecutive positive integers +ordered by increasing or decreasing slit position. +.IP BEAM +.br +An integer beam number. The beam number need not be +unique; i.e. several apertures may have the same beam number. +The beam number will be recorded in the image header of the +the extracted spectrum. By default the beam number is the same as +the ID. +.IP APAXIS +.IP CENTER[2] +.br +The center of the aperture along the slit and dispersion axes in the two +dimensional image. +.IP LOWER[2] +.br +The lower limits of the aperture, relative to the aperture center, +along the slit and dispersion axes. The lower limits need not be less +than the center. +.IP UPPER[2] +.br +The upper limits of the aperture, relative to the aperture center, +along the slit and dispersion axes. The upper limits need not be greater +than the center. +.IP CURVE +.br +An offset to be added to the center position for the \fIslit\fR axis which is +a function of the wavelength. The function is one of the standard IRAF +types; a legendre polynomial, a chebyshev polynomial, a linear spline, +or a cubic spline. +.IP background +.br +Parameters for background subtraction based on the interactive +curve fitting (\fBicfit\fR) tools. + +.PP +The aperture center is the only absolute coordinate (relative to the +image or image section). The other aperture parameters and the +background fitting regions are defined relative to the center. Thus, +an aperture may be repositioned easily by changing the center +coordinates. Also a constant aperture size, shape (curve), and +background regions may be maintained for many apertures. The center +and aperture limits, in image coordinates, along the slit axis are +given by: + +.EQ I + ~roman center ( lambda )~mark = roman cslit + roman curve ( lambda ) +.EN +.EQ I +roman lower ( lambda )~lineup = roman center ( lambda ) + roman lslit +.EN +.EQ I +roman upper ( lambda )~lineup = roman center ( lambda ) + roman uslit +.EN + +where $lambda$ is the wavelength coordinate. Note that both the lower and +upper constants are added to the center defined by the aperture center and +the offset curve. The aperture limits along the dispersion axis are +constant since there is no offset curve: + +.EQ I +roman center (s)~lineup = roman cdisp +.EN +.EQ I +roman lower (s)~lineup = roman center (s) + roman ldisp +.EN +.EQ I +roman upper (s)~lineup = roman center (s) + roman udisp +.EN + +.PP +Apertures for a particular image may be defined in several ways. +These methods are arranged in a hierarchy. + +.IP (1) +The database is first searched for previously defined apertures. +.IP (2) +If no apertures are found and a reference image is specified then the +database is searched for apertures defined for the reference image. +.IP (3) +The user may then edit the apertures interactively with graphics +commands if the \fIapedit\fR parameter is set. This includes creating +new apertures and deleting or modifying existing apertures. This +interactive editing procedure may be entered from any of the \fBapextract\fR +tasks. +.IP (4) +For the tasks \fBtrace\fR, \fBsumextract\fR, and \fBstripextract\fR +if no apertures are defined at this point a default aperture +is created consisting of the entire image with center at the center of +the image. Note that if an image section is used then the aperture +spans the image section only. +.IP (5) +Any apertures created, modified, or adopted from a reference image are recorded +in the database for the image. + +.PP +There are several important points to appreciate in the above logic. +First, any of the tasks may be used without prior use of the others. +For example one may use extract with the \fIapedit\fR switch set and +define the apertures to be extracted (except for tracing). +Alternatively the apertures may be defined with \fBapedit\fR +interactively and then traced and extracted noninteractively. Second, +image sections may be used to define apertures (step 4). For example +a list of image sections (such as are used in multislit spectra) may be +extracted directly and noninteractively. Third, multiple images may +use a reference image to define the same apertures. There are several +more options which are illustrated in the examples section. +.PP +Another subtlety is the way in which reference images may be +specified. The tasks in the package all accept list of images +(including image sections). Reference images may also be given as a +list of images. The lists, however, need not be of the same length. +The reference images in the reference image list are paired in order +with the input images. If the reference list ends before the image +list then the last reference image is used for the remaining images. +The most common situations are when there is no reference image, when +only one reference image is given for a set of input images, and when a +matching list of reference images is given. In the second case the +reference image refers to all the input images while in the last case +each input image has a reference image. +.PP +There is a trick which may be played with the reference images. If a list +of input images is given and the reference image is the same as the first +input image then only the first image need be done interactively. +This is because after the apertures for the first image have been defined +they are recorded in the database. Then when the database is searched +for apertures for the second image, the apertures of the reference image +will be available. +.NH +.PP +\fBApedit\fR is a generally interactive task which graphs a line of +an image along the slit axis and allows the user to define and edit +apertures with the graphics cursor. The defined apertures are recorded +in a database. The task \fBtrace\fR traces the positions of the +spectrum profiles from one wavelength to other wavelengths in the image +and fits a smooth curve to the positions. This allows apertures +to follow shifts in the spectrum along the slit axis. The tasks +\fBsumextract\fR and \fBstripextract\fR perform the actual aperture +extraction to one and two dimensional spectra. They have options for +performing background subtraction, detecting and replacing bad pixels, +modeling the spectrum profile, and weighting the pixels in the aperture +when summing across the dispersion. +.NH +Tracing +.PP +The spectra to be extracted are not always aligned exactly with the +image columns or lines (the extraction axes). +For consistent extraction it is important that the same +part of the spectrum profile be extracted at each wavelength point. +Thus, the extraction apertures allow for shifts along the spatial axis +at each dispersion point. The shifts are defined by a curve which is a +function of the wavelength. The curve is determined by tracing the +positions of the spectrum profile at a number of wavelengths and +fitting a function to these positions. +.PP +The task \fBtrace\fR performs the tracing and curve fitting and records +the curve in the database. The starting point along the +dispersion axis (a line or column) for the tracing is specified by the +user. The position of the profile is then determined using the +\fBcenter1d\fR algorithm described elsewhere (see the help page for +\fBcenter1d\fR or the paper \fIThe Long Slit Reduction Package\fR). +The user specifies a step along the dispersion axis. At each step the +positions of the profiles are redetermined using the preceding +position as the initial guess. In order to enhance and trace weak +spectra the user may specify a number of neighboring profiles to be +summed before determining the profile positions. +.PP +Once the +positions have been traced from the starting point to the ends of the +aperture, or until the positions become indeterminate, a curve of a +specified type and order is fit to the positions as a function of +wavelength. The function fitting is performed with the \fBicfit\fR +tools (see the IRAF help page). The curve fitting may be performed +interactively or noninteractively. Note that when the curve is fit +interactively the actually positions measured are graphed. However, the +curve is stored in the aperture definition as an offset relative to the +aperture center. +.PP +The tracing requires that the spectrum profile have a shape from which +\fBcenter1d\fR can determine a position. This pretty much means +gaussian type profiles. To extract a part of a long slit spectrum +which does not have such a profile the user must trace a profile from +another part of the image or a different image and then shift the +center of the aperture without changing the shape. For example the +center of a extended galaxy spectrum can be traced and the aperture +shifted to other parts of the galaxy. +.NH +Extraction +.PP +There are two types of extraction; strip extraction and sum +extraction. Strip extraction produces two dimensional images with +pixels corresponding to the center of an aperture aligned along the +lines or columns. Sum extraction consists of the weighted sum of the +pixels within an aperture along the image axis nearest the spatial axis +at each point along the dispersion direction. It is important to +understand that the extraction is along image lines or columns while +the actual dispersion/spatial coordinates may not be aligned exactly +with the image axes. If this misalignment is important then for simple +rotations the task \fBrotate\fR in the \fBimages\fR package may be used +while for more complex coordinate rectifications the tasks in the +\fBlongslit\fR package may be used. +.NH 2 +Sum Extraction +.PP +Denote the image axis nearest the spatial axis by the index $s$ and +the other image axis corresponding to the dispersion axis by $lambda$. +The extraction is defined by the equation + +.EQ I (1) +f sub lambda~=~sum from s (W sub sl (I sub sl - B sub sl ) / P sub sl ) / +sum from s W sub sl +.EN + +where the sums are over all pixels along the spatial axis within some +aperture. The $W$ are weights, the $I$ are pixel intensities, +the $B$ are background intensities, and the $P$ are a normalized +profile model. +.PP +There are many possible choices for the extraction weights. The extraction +task currently provides two: + +.EQ I (2a) +W sub sl~mark =~P sub sl +.EN +.EQ I (2b) +W sub sl~lineup =~P sub sl sup 2 / V sub sl +.EN + +where $V sub sl$ is the variance of the pixel intensities given by the +model + +.EQ I + V sub sl~=~v sub 0 + v sub 1~max (0,~I sub sl )~~~~if v sub 0~>~0 +.EN +.EQ I + V sub sl~=~v sub 1~max (1,~I sub sl )~~~~~~~~~if v sub 0~=~0 +.EN + +Substituting these weights in equation (1) yields the extraction equations + +.EQ I (3a) +f sub lambda~mark =~sum from s (I sub sl - B sub sl ) +.EN +.EQ I (3b) +f sub lambda~lineup =~sum from s (P sub sl (I sub sl - B sub sl ) / V sub sl ) / +sum from s (P sub sl sup 2 / V sub sl ) +.EN + +.PP +The first type of weighting (2a), called \fIprofile\fR weighting, weights +by the profile. Since the weights cancel this gives the simple extraction (3a) +consisting of the direct summation of the pixels within the aperture. +It has the virtue of being simple and computationally fast (since the +profile model does not have to be determined). +.PP +The second type of weighting (2b), called \fIvariance\fR weighting, +uses a model for the variance of the pixel intensities. +The model is based on Poisson statistics for a linear quantum detector. +The first term is commanly call the \fIreadout\fR noise and the second term +is the Poisson noise. The actual value of $v sub 1$ is the reciprical of +the number of photons per digital intensity unit (ADU). A simple variant of +this type of weighting is to let $v sub 1$ equal zero. Since the actual +scale of the variance cancels we can then set $v sub 0$ to unity to obtain + +.EQ I (4) +f sub lambda~=~sum from s (P sub sl (I sub sl - B sub sl )) / +sum from s P sub sl sup 2 . +.EN + +The interpretation of this extraction is that the variance of the intensities +is constant. It gives greater weight to the stronger parts of the spectrum +profile than does the profile weighting (3a) since the weights are +$P sub sl sup 2$. Equation (4) has the virtue that one need not know the +readout noise or the ADU to photon number conversion. +.NH 3 +Optimal Extraction +.PP +Variance weighted extraction is sometimes called optimal extraction because +it is optimal in a statistical sense. Specifically, +the relative contribution of a pixel to the sum is related to the uncertainty +of its intensity. The uncertainty is measured by the expected variance of +a pixel with that intensity. The degree of optimality depends on how well +the relative variances of the pixels are known. +.PP +A discussion of the concepts behind optimal extraction is given in the paper +\fIAn Optimal Extraction Algorithm for CCD Spectroscopy\fR by Keith Horne +(\fBPASP\fR, June 1986). The weighting described in Horne's paper is the +same as the variance weighting described in this paper. The differences +in the algorithms are primarily in how the model profiles $P sub sl$ are +determined. +.NH 3 +Profile Determination +.PP +The profiles of the spectra along the spatial axis are determined when +either the detection and replacement of bad pixels or variance +weighting are specified. The requirements on the profiles are that +they have the same shape as the image profiles at a each dispersion +point and that they be as noise free and uncontaminated as possible. +The algorithm used to create these profiles is to average a specified +number of consecutive background subtracted image profiles immediately +preceding the wavelength to which a profile refers. When there are an +insufficient number of image profiles preceding the wavelength being +extracted then the following image profiles are also used to make up +the desired number. The image profiles are interpolated to a common +center before averaging using the curve given in the aperture +definition. The averaging reduces the noise in the image data while +the centering eliminates shifts in the spectrum as a function of +wavelength which would broaden the profile relative to the profile of a +single image line or column. It is assumed that the spectrum profile +changes slowly with wavelength so that by using profiles near a given +wavelength the average profile shape will correctly reflect the profile +of the spectrum at that wavelength. +.PP +The average profiles are determined in parallel with the extraction, +which proceeds sequentially through the image. Initially the first set +of spectrum profiles is read from the image and interpolated to a common +center. The profiles are averaged excluding the first profile to be +extracted; the image profiles in the average never include the image +profile to be extracted. Subsequently the average profile is updated +by adding the last extracted image profile and subtracting the image +profile which no longer belongs in the average. This allows each image +profile to be accessed and interpolated only once and makes the +averaging computationally efficient. This scheme also allows excluding +bad pixels from the average profile. The average profile is used to +locate and replace bad pixels in the image profile being extracted as +discussed in the following sections. Then when this profile is added +into the average for the next image profile the detected bad pixels are +no longer in the profile. +.PP +In summary this algorithm for determining the spectrum profile +has the following advantages: + +.IP (1) +No model dependent smoothing is done. +.IP (2) +There is no assumption required about the shape of the profile. +The only requirement is that the profile shape change slowly. +.IP (3) +Only one pass through the image is required and each image profile +is accessed only once. +.IP (4) +The buffered moving average is very efficient computationally. +.IP (5) +Bad pixels are detected and removed from the profile average as the +extraction proceeds. + +.NH 3 +Detection and Elimination of Bad Pixels +.PP +One of the important features of the aperture extraction package is the +detection and elimination of bad pixels. The average profile described +in the previous section is used to find pixels which deviate from this +profile. The algorithm is straightforward. A model spectrum of the +image profile is obtained by scaling the normalized profile to the +image profile. The scale factor is determined using chi squared fitting: + +.EQ I (6) +M sub sl~=~P sub sl~left { sum from s ((I sub sl - B sub sl ) P sub sl / +V sub sl)~/~ sum from s (P sub sl sup 2 / V sub sl ) right } . +.EN + +The RMS of this fit is determined and pixels deviating by more than a +user specified factor times this RMS are rejected. The fit is then +repeated excluding the rejected points. These steps are repeated until +the user specified number of points have been rejected or no further deviant +points are detected. The rejected points in the image profile are then +replaced by their model values. +.PP +This algorithm is based only on the assumption that the spatial profile +of the spectrum (no matter what it is) changes slowly with wavelength. +It is very sensitive at detecting departures from the expected profile. +Its main defect is that in the first pass at the fit all of the image profile +is used. If there is a very badly deviant point and the rest of the profile +is weak then the scale factor may favor the bad pixel more than the +rest of the profile resulting in rejecting good profile points and not +the bad pixel. +.NH 3 +Relation of Optimal Extraction to Model Extraction +.PP +Equation (1) defines the extraction process in terms of a weighted sum +of the pixel intensities. However, the actual extraction operations +performed by the task \fBsumextract\fR are + +.EQ I (7a) +f sub lambda~mark =~sum from s (I sub sl - B sub sl ) +.EN +.EQ I (7b) +f sub lambda~lineup =~sum from s M sub sl +.EN + +where $M sub sl$ is the model spectrum fit to the background subtracted +image spectrum $(I sub sl - B sub sl )$ +defined in the previous section (equation 6). It is not obvious at first that +(7b) is equivalent to (3b). However, if one sums (6) and uses the fact +that the sum of the normalized profile is unity one is left with equation (3b). +.PP +Equations (6) and (7b) provide an alternate way to think about the +extracted one dimensional spectra. Sum extraction of the model spectrum +is used instead of the weighted sum for variance weighted extraction +because the model spectrum is a product of the profile determination +and the bad pixel cleaning process. It is then more convenient +and efficient to use the simple equations (7). +.NH 2 +Strip Extraction +.PP +The task \fBstripextract\fR uses one dimensional image interpolation +to shift the pixels along the spatial axes so that in the resultant +output image the center of the aperture is exactly aligned with the +image lines or columns. The cleaning of bad pixels is an option +in this extraction using the methods described above. In addition +the model spectrum described above may be extracted as a two +dimensional image. In fact, the only difference between strip extraction +and sum extraction is whether the final step of summing the pixels +in the aperture along the spatial axis is performed. +.PP +The primary use of \fBstripextract\fR is as a diagnostic tool. It +allows the user to see the background subtracted, cleaned and/or model +spectrum as an image before it is summed to a one dimensional spectrum. +In addition the two dimensional format allows use of other IRAF tools such as +smoothing operators. When appropriate +it is a much simpler method of removing detector distortions and alignment +errors than the full two dimensional mapping and image transformation +available with the \fBlongslit\fR package. +.NH +Examples +.de CS +.nf +.ft L +.. +.de CE +.fi +.ft R +.. +.PP +This section is included because the flexibility and many options of +the tasks allows a wide range of applications. The examples illustrate +the use of the task parameters for manipulating input images, output +images, and reference images, and setting apertures interactively and +noninteractively. They do not illustrate the different possibilities +in extraction or the interactive aperture definition and editing +features. These examples are meant to be relevant to actual data +reduction and analysis problems. For the purpose of these examples we +will assume the dispersion axis is along the second image axis; i.e. +DISPAXIS = 2. +.PP +The simplest problem is the extraction of an object spectrum which +is centered on column 200. To extract the spectrum with an aperture +width of 20 pixels an image section can be used. + +.CS +cl> sumextract image[190:209,*] obj1d +cl> stripextract image[190:209,*] obj2d +.CE + +To set the aperture center and limits interactively the edit option can be +used with or without the image section. This also allows fractional pixel +centering and limits. +.PP +If the object slit position changes the spectrum profile can be traced first +and then extracted. + +.CS +cl> trace image[190:209,*] +cl> sumextract image[190:209,*] obj1d +cl> stripextract image[190:209,*] obj2d +.CE + +By default the apertures are defined and/or edited interactively in +\fBtrace\fR and editing is not the default in \fBsumextract\fR or +\fBstripextract\fR. +.PP +A more typical example involves many images. In this case a list of images +is used though, of course, each image could be done separately as +in the previous examples. There are three common forms of lists, a +pattern matching template, a comma separated list, and an "@" file. +In addition the template editing metacharacter, "%", may be used +to create new output image names based on input image names. +If the object positions are different in each image then we can select +apertures with image sections or using the editing option. Some examples +are + +.CS +cl> sumextract image1[10:29,*],image2[32:51] obj1,obj2 +cl> sumextract image* e//image* edit+ +cl> sumextract image* image%%ex%* edit+ +cl> sumextract @images @images edit+ +.CE + +The "@" files can be created from the other two types of lists using the +\fBsections\fR task in the \fBimages\fR package. An important feature +of the image templates is the use of the concatenation operator. Note, +however, this a feature of image templates and not file templates. +Also the output root name may be the same as the input +name because an extension is added provided there are no image +sections in the input images. +.PP +If the object positions are the same then the apertures can be defined once +and the remaining objects can be extracted using a reference image. + +.CS +cl> apedit image1 +cl> sumextract image* image* ref=image1 +.CE + +Rather than using \fBapedit\fR one can use \fBsumextract\fR alone with +the edit switch set. The command is + +.CS +cl> sumextract image* image* ref=image1 edit+ +.CE + +The task queries whether to edit the apertures for each image. +For the first image respond with "yes" and set the apertures interactively. +For the second task respond with "NO". Since the aperture for "image1" +was recorded when the first image was extracted it then acts as the reference +for the remaining images. The emphatic response "NO" turns off the edit switch +for all the other images. One difference between this example and the +previous one is that the task cannot be run as a background batch task. +.PP +The extension to using traced apertures in the preceding examples is +very similar. + +.CS +cl> apedit image1 +cl> trace image* ref=image1 edit- +cl> sumextract image* image* +cl> stripextract image* image* +.CE + +.PP +Another common type of data has multiple spectra on each image. Some examples +are echelle and multislit spectra. Echelle extractions usually are done +interactively with tracing. Thus, the commands are + +.CS +cl> trace ech* +cl> sumextract ech* ech* +.CE + +For multislit spectra the slitlets are usually referenced by creating +an "@" file containing the image sections. The usage for extraction +is then + +.CS +cl> sumextract @slits @slitsout +.CE + +.PP +The aperture definitions can be transfered from a reference image to +other images using \fBapedit\fR. There is no particular reason to +do this except that reference images would not be needed in +\fBtrace\fR, \fBsumextract\fR or \fBstripextract\fR. The transfer +is accomplished with the following command + +.CS +cl> apedit image1 +cl> apedit image* ref=image1 edit- +.CE + +The above can also be combined into one step by editing the first image +and then responding with "NO" to the second image query. +.NH +Future Developments +.PP +The IRAF extraction package \fBapextract\fR is going to continue to +evolve because 1) the extraction of one and two dimensional spectra +from two dimensional images is an important part of reducing echelle, +longslit, multislit, and multiaperture spectra, 2) the final strategy +for handling multislit and multiaperture spectra produced by aperture +masks or fiber optic mapping has not yet been determined, and 3) the +extraction package and the algorithms have not received sufficient user +testing and evaluation. Changes may include some of the following. + +.IP (1) +Determine the actual variance from the data rather than using the Poisson +CCD model. +.IP (2) +Another task, possibly called \fBapfind\fR, is needed to automatically find +profile positions in multiaperture, multislit, and echelle spectra. +.IP (3) +The bad pixel detection and removal algorithm does not handle well the case +of a very strong cosmic ray event on top of a very weak spectrum profile. +A heuristic method to make the first fitting pass of the average +profile to the image data less prone to errors due to strong cosmic rays +is needed. +.IP (4) +The aperture definition structure is general enough to allow the aperture +limits along the dispersion dimension to be variable. Eventually aperture +definition and editing will be available using an image display. Then +both graphics and image display editing switches will be available. +An image display interface will make extraction of objective prism +spectra more convenient than it is now. +.IP (5) +Other types of extraction weighting may be added. +.IP (6) +Allow the extraction to be locally perpendicular to the traced curve. diff --git a/noao/twodspec/apextract/doc/old/apextract1.ms b/noao/twodspec/apextract/doc/old/apextract1.ms new file mode 100644 index 00000000..b586daad --- /dev/null +++ b/noao/twodspec/apextract/doc/old/apextract1.ms @@ -0,0 +1,811 @@ +.EQ +delim $$ +define sl '{s lambda}' +.EN +.RP +.TL +The IRAF APEXTRACT Package +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +.AB +The IRAF \fBapextract\fR package provides tools for the extraction of +one and two dimensional spectra from two dimensional images +such as echelle, long slit, multifiber, and multislit spectra. +Apertures of fixed spatial width define the regions of +the two dimensional images to be extracted at each point along the +dispersion axis. Apertures may follow changes in the positions of +the spectra as a function of position along the dispersion axis. +The spatial and dispersion axes may be oriented along either image axis. +Extraction to one dimensional spectra consists of a weighted sum of the pixels +within the apertures at each point along the dispersion axis. The +weighting options provide the simple sum of the pixel values and a +weighting by the expected uncertainty of each pixel. Two dimensional +extractions interpolate the spectra in the spatial axis to produce +image strips with the position of the spectra exactly aligned with one +of the image dimensions. The extractions also include optional +background subtraction, modeling, and bad pixel detection and replacement. +The tasks are flexible in their ability to define and edit apertures, +operate on lists of images, use apertures defined for reference +images, and operate both very interactively or noninteractively. +The extraction tasks are efficient and require only one pass through +the data. This paper describes the package organization, the tasks, +the algorithms, and the data structures. +.AE +.NH +Introduction +.PP +The IRAF \fBapextract\fR package provides tools for the extraction of +one and two dimensional aperture spectra from two dimensional format +images such as those produced by echelle, long slit, multifiber, and +multislit spectrographs. This type of data is becoming increasingly +common because of the efficiency of data collection and technological +improvements in spectrographs and detectors. The trend is to greater +and greater numbers of spectra per image. Extraction is one of the +fundamental operations performed on these types of two dimensional +spectral images, so a great deal of effort has gone into the design and +development of this package and to making it easy to use. +.PP +The tasks are flexible and have many options. To make the best use of +them it is important to understand how they work. This paper provides +a general description of the package organization, the tasks, the algorithms, +and the data structures. Specific descriptions of parameters +and usage may be found in the IRAF help pages for the tasks which +are included as appendices to this paper. The image reduction "cookbooks" +also provide examples of usage for specific instruments or types +of instruments. +.PP +Extraction of spectra consists of three logical steps. First, locating +the spectra in the two dimensional image. This includes defining the +dispersion direction, the positions of the spectra at some point +along the dispersion direction, the spatial extent or aperture to be +used for extraction, and possible information about where the background +for each spectrum is to be determined. This information is maintained +in the package as structures called \fIapertures\fR. The second step is +to measure the positions of the spectra at other points along the dispersion. +This process is called tracing. Tracing is optional if the spectra +are exactly aligned with the dispersion direction. The final step is +to extract the spectra into one or two dimensional images. +.PP +The \fBapextract\fR package identifies the image axes with the spatial +and dispersion axes. Thus, during extraction, pixels of constant +wavelength are assumed to be along a line or column. In this paper the +terms \fIslit\fR or \fIspatial\fR axis and \fIdispersion\fR or +\fIwavelength\fR axis are used to refer to the image axes corresponding +to the spatial and dispersion axes. To simplify the presentation a +cut across the dispersion axis will be called a line even though it +could also be a column. +.PP +Often a small degree of +misalignment between the image axes and the true dispersion and spatial +axes is not important. The main effect of misalignment is a broadening +of the spectral features due to the difference in wavelength on +opposite sides of the extraction aperture. If the misalignment is +significant, however, the image may be rotated with the task +\fBrotate\fR in the \fBimages\fR package or remapped with the +\fBlongslit\fR package tasks for coordinate rectification. +.PP +It does not matter which image axis is the dispersion axis since the +tasks work equally well in either orientation. However, the dispersion +axis must be defined, with the \fBtwodspec\fR task \fBsetdisp\fR, +before these tasks may be used. This task is a simple script which +adds the parameter DISPAXIS to the image headers. The \fBapextract\fR +tasks, like the \fBlongslit\fR tasks, look in the header to determine +the dispersion axis. +.NH +The APEXTRACT Package +.PP +In this section the organization of the \fBapextract\fR package and the +functions and parameters of the tasks are briefly described. More detailed +descriptions are given in the help pages for the tasks. The tasks in the +package are: + +.ce +.ft B +The APEXTRACT Tasks + +.ft L +.nf + apdefault - Set the default aperture parameters + apedit - Edit apertures interactively + apfind - Automatically find spectra and define apertures + apio - Set the I/O parameters for the APEXTRACT tasks + apnormalize - Normalize 2D apertures by 1D functions + apstrip - Extract two dimensional aperture strips + apsum - Extract one dimensional aperture sums + aptrace - Trace positions of spectra +.fi +.ft R + +.PP +The tasks are highly integrated so that each task includes some or all of +the functions and parameters of the other tasks. Thus, these tasks +reflect the logical organization of the extraction process rather than +a set of disparate tools. One reason for this organization is to group +the parameters by function into easy to manage \fIparameter sets +(psets)\fR. The tasks \fBapdefault\fR and \fBapio\fR are just psets +for specifying the default aperture parameters and the I/O parameters +of the package; in other words, they do nothing but provide a grouping +of parameters. Executing these tasks is a shorthand for the command +"eparam apdefault" or "eparam apio". +.PP +The input/output parameters in \fBapio\fR specify the aperture database, +an optional log file for brief, time stamped log information, an optional +metacode plot file for saving plots of the apertures, the traces, and the +quick look extracted spectra, and the graphics input and output devices +(almost always the user's terminal). One point about the plot file is +that the plots are recorded even if the user chooses not to view these +graphs as the task is run interactively or noninteractively. This allows +reviewing the traces and spectra with a tool like \fBgkimosaic\fR. +.PP +The default aperture parameters specify the aperture limits (basically +the width of the aperture and position relative to the center of the +spectrum) and the background fitting parameters. The background +parameters are the standard parameters used by the \fBicfit\fR package +with which the user is assumed to be familiar. For more on this see +the help information for \fBicfit\fR. +.PP +The other tasks are both psets and executable tasks. There are a +number features which are common to all these tasks. First, they +follow the same steps in defining apertures for the input images. +These steps are: +.IP (1) +If a reference image is specified then the database is searched for +apertures previously defined for this image. +.IP (2) +If apertures are found for the reference image they may be recentered +on the spectra in the input image at a specified line. This does not +change the shape of the apertures but only adds a shift in the center +coordinate of the apertures along the spatial axis. +.IP (3) +If a reference image is not specified or if no reference apertures are found +then the database is searched for previous apertures for the input image. +.IP (4) +If there are no apertures defined either from a reference image or previous +apertures for the input image then an automatic algorithm may be used to find +a specified number of spectra (based on peak values) and assign them default +apertures. +.IP (5) +Finally, a sophisticated graphical aperture editor may be used to examine, +define, and modify apertures. +.IP (6) +When tracing, extracting, or normalizing flat field spectra, +if no apertures have been defined by the steps above then a single default +aperture, centered in the image, is defined. + +Any apertures created, modified, or adopted from a reference image +may be recorded in the database for the input image. +.PP +The operations listed above are selected by parameters common to each of the +tasks. For example the parameter \fIedit\fR selects whether to enter +the aperture editor and is present in each of the executable tasks. +On the other hand the parameters specific to the aperture editor, +while accessed by any of the tasks, reside only in the parameter set of +the task \fBapedit\fR. In this way parameters are distributed +by logical function rather than including them in each task. +.PP +In addition to the aperture editing and finding functions available in +every task, some of the tasks include functions for tracing, extracting, +or normalizing the spectra. The tasks \fBapsum\fR and \fBapstrip\fR, +which extract one and two dimensional spectra, are at the top of the +hierarchy and include all the logical functions provided by the package. +Thus, in most cases the user need only use the task \fBapsum\fR to define +apertures, trace the spectra, and extract them. +.PP +Another feature common to the tasks is their interactive and noninteractive +modes. When the parameter \fIinteractive\fR is set to \fIno\fR then the +aperture editing, interactive trace fitting, and review of the extracted +one dimensional spectra functions of the package are bypassed. Note that +this means you do not have to explicitly set the parameter \fIedit\fR, +or those for other purely interactive functions, +to \fIno\fR when extracting spectra noninteractively. In the noninteractive +mode there are also no queries. +.PP +The interactive mode includes the interactive graphical functions of +aperture editing, trace fitting, and extraction review. In addition +the user is queried at each step. For example the user will be queried +whether to edit the apertures for a particular image if the task +parameter for editing is set. The queries have four responses: \fIyes, +no, YES,\fR and \fINO\fR. The lower case responses apply only to the +particular query. The upper case responses apply to any further +queries of the same type and suppress the query from appearing again. +This is particularly useful when dealing with many images or many +apertures. For example, when fitting the traced points interactively +the user may examine the first few and then say \fINO\fR to skip the +remaining apertures using the last defined fitting parameters. Note +that if a plot file is specified the graphs showing the traced points +and the fits are recorded even if they are not viewed interactively. +.NH +Algorithms +.PP +The \fBapextract\fR package consists of a number of logical functions or, +in computerese, algorithms. These algorithms manipulate the aperture +structure data and create output data in the form of images. In +this section the various algorithms are described. In addition to the +algorithms specific to the package, there are some general algorithms +and tools used which appear in other IRAF tasks. Specifically there are the +interactive curve fitting tools called \fBicfit\fR and the one +dimensional centering algorithm called \fBcenter1d\fR. These are +mentioned below and described in detail elsewhere in the help documentation. +.NH 2 +Finding Spectra +.PP +When dealing with images containing large numbers of spectra it may be +desirable to locate the spectra and define apertures automatically. The +\fBapfind\fR algorithm provides this ability from any of the executable +tasks and from the aperture editor using the 'f' key. It takes a cut +across the dispersion axis by summing one or more image lines. +All the local maxima are identified and ranked by intensity. Starting +with the highest maxima any other peaks within a specified minimum +separation are eliminated. The weakest remaining peaks exceeding the +specified number are eliminated next. The positions of the +spectra based on peak positions are refined by centering using the +\fBcenter1d\fR algorithm. Finally identical apertures are assigned +for each spectrum found. +.PP +When the algorithm is invoked by a task, with the parameter \fIfind\fR, +there must be no previous or reference apertures in the database. +The apertures assigned to the spectra have the parameters +specified in the \fBapdefault\fR pset. When the algorithm is invoked +from the aperture editor with the 'f' key then new apertures are +added to any existing apertures up to the total number of apertures, +existing plus new, given by the \fInfind\fR parameter. If there +is a current aperture then copies of it are used to define the +apertures for the new spectra. Thus, one method for defining many +apertures is to use the editor to define one aperture, set its +limits and background parameters, and then find the remaining apertures +automatically. +.NH 2 +Centering and Recentering +.PP +When new apertures are defined (except for a special key to mark apertures +without centering) or when apertures are recentered, either with the +centering key in the editor or with the task parameter \fIrecenter\fR, +the center is determined using the \fBcenter1d\fR algorithm. +This is described in the help documentation under the name \fBcenter1d\fR. +Briefly, the data line is convolved with an asymmetric function of specified +width. The convolution integral is evaluated using image interpolation. +The sign of the convolution acts as a gradient to move from the starting +position to the final position where the convolution is zero. This algorithm +is good to about 5% of a pixel. It has two important parameters; the +width of the convolution and the error distance between the starting +and final positions. The width of the convolution determines the scale +of features to which the centering is sensitive. The error distance is +the greatest change allowed in the initial positions. If this error +distance is exceeded then the centering fails and either a new aperture +is not defined or the position of an existing aperture is not changed. +.NH 2 +The Aperture Editor +.PP +The aperture editor is a sophisticated tool for defining and modifying +apertures. It may also be used to selectively trace and extract +spectra. Thus, the aperture editor may be used alone to perform all +the functions for extracting spectra. The aperture editor uses a +graphical presentation. A line or sum of lines is displayed. The +apertures are marked above the line and identified with the aperture +number. Information about the current aperture is shown on the status +line. The cursor is used to mark new apertures, shift the center or +aperture limits, and perform a variety of functions. Because there may +be many apertures which the user wants to modify in the same way there +is a mode switch to apply commands to all the apertures. The switch is +toggled with the 'a' key and the mode is indicated on the status line. +.PP +There are also a number of colon commands. These allow resetting parameters +explicitly rather than by cursor and interacting with the aperture +database and the image data. The background fitting parameters such as +the background regions and function order are set by switching to the +interactive curve fitting package \fBicfit\fR. The line being edited is +used to set the parameters. No background is actually extracted at this +stage. The ALL mode applies to the background parameters as well. +.PP +The aperture editor has many commands. For a description of the +commands see the help information for the task \fBapedit\fR. In +summary the aperture editor is used to interactively define apertures, +both centered on spectra and at arbitrary positions, adjust the limits +and background parameters, and possibly select apertures to be traced +and extracted. These functions may be applied independently on each +aperture for maximum flexibility or applied to all apertures for ease +of use with many apertures. +.NH 2 +Tracing +.PP +The spectra to be extracted are not always aligned exactly with the +image columns or lines. For consistent +extraction it is important that the same part of the spectrum profile +be extracted at each wavelength point. Thus, the extraction apertures +allow for shifts along the spatial axis at each wavelength. The +shifts are defined by a curve which is a function of the wavelength. +The curve is determined by tracing the positions of the spectrum +profile at a number of wavelengths and fitting a function to these +positions. +.PP +The \fIaptrace\fR algorithm performs the tracing and curve fitting. +The starting point along the dispersion axis (a line or column) for +the tracing is specified by the user. The positions of the spectrum +profiles are determined using the \fBcenter1d\fR algorithm +(see the previous section on centering and the help page for \fBcenter1d\fR). +The user specifies a step along the dispersion axis. At each step the +positions of the profiles are redetermined using the preceding +positions as the initial guesses. If the positions are lost at one step +an attempt is made to recover the spectrum in the next step. If this +also fails then tracing of that spectrum in that direction is finished. +In order to enhance and trace weak spectra the user may specify a number +of neighboring profiles to be summed before determining the profile positions. +In addition to the other centering parameters, there is also a +\fIthreshold\fR parameter to define a minimum contrast between the spectrum +and the background. +.PP +Once the positions have been traced from the starting point to the ends of the +aperture, or until the positions become indeterminate, a curve of a +specified type and order is fit to the positions as a function of +wavelength. The function fitting is performed with the \fBicfit\fR +tools (see the help documentation for \fBicfit\fR). The curve fitting +may be performed interactively or noninteractively. Note that when the +curve is fit interactively the actual positions measured are graphed. +However, the curve is stored in the aperture definition as an offset +relative to the aperture center. +.PP +The tracing requires that the spectrum profile be continuous and have +some kind of maxima. This means that arc calibration spectra or +arbitrary regions of an extended object in a long slit spectrum cannot +be traced. Flat topped spectra such as quartz lamp images taken through +slits can be measured provided the width of the centering function is +somewhat wider than the profile (to avoid centering on little peaks +within the slit). For images which cannot be traced, reference apertures +from images that can be traced are used. This is how apertures for +arc spectra are defined and extracted. For sky apertures or the +wings of extended objects the reference apertures can be shifted +by the aperture editor without altering the shape of the aperture. +.NH 2 +Sum Extraction +.PP +Sum extraction consists of the weighted sum of the pixels along the spatial axis +within the aperture limits at each point along the dispersion axis. +A background at each point along the dispersion may be determined by fitting a +function to data in the vicinity of the spectrum and subtracting the +function values estimated at each point within the aperture. The estimated +background may be output as a one dimensional spectrum. Other options +include the detection and replacement of deviant points such as due to +cosmic rays. +.PP +Denote the image axis nearest the spatial axis by the index $s$ and +the other image axis corresponding to the dispersion axis by $lambda$. +The weighted extraction is defined by the equation + +.EQ I (1) +f sub lambda~=~sum from s (W sub sl (I sub sl - B sub sl ) / P sub sl ) / +sum from s W sub sl +.EN + +where the sums are over all pixels along the spatial axis within some +aperture. The $W$ are weights, the $I$ are pixel intensities, +the $B$ are background intensities, and the $P$ are a normalized +profile model. +.PP +There are many possible choices for the extraction weights. The extraction +task \fBapsum\fR currently provides two: + +.EQ I (2a) +W sub sl~mark =~P sub sl +.EN +.EQ I (2b) +W sub sl~lineup =~P sub sl sup 2 / V sub sl +.EN + +where $V sub sl$ is the variance of the pixel intensities given by the +model + +.EQ I + V sub sl~=~v sub 0 + v sub 1~max (0,~I sub sl )~~~~if v sub 0~>~0 +.EN +.EQ I + V sub sl~=~v sub 1~max (1,~I sub sl )~~~~~~~~~if v sub 0~=~0 +.EN + +Substituting these weights in equation (1) yields the extraction equations + +.EQ I (3a) +f sub lambda~mark =~sum from s (I sub sl - B sub sl ) +.EN +.EQ I (3b) +f sub lambda~lineup =~sum from s (P sub sl (I sub sl - B sub sl ) / V sub sl ) / +sum from s (P sub sl sup 2 / V sub sl ) +.EN + +.PP +The first type of weighting (2a), called \fIprofile\fR weighting, weights +by the profile. Since the weights cancel this gives the simple extraction (3a) +consisting of the direct summation of the pixels within the aperture. +It has the virtue of being simple and computationally fast (since the +profile model does not have to be determined). +.PP +The second type of weighting (2b), called \fIvariance\fR weighting, +uses a model for the variance of the pixel intensities. +The model is based on Poisson statistics for a linear quantum detector. +The first term is commonly call the \fIreadout\fR noise and the second term +is the Poisson noise. The actual value of $v sub 1$ is the reciprocal of +the number of photons per digital intensity unit (ADU). A simple variant of +this type of weighting is to let $v sub 1$ equal zero. Since the actual +scale of the variance cancels we can then set $v sub 0$ to unity to obtain + +.EQ I (4) +f sub lambda~=~sum from s (P sub sl (I sub sl - B sub sl )) / +sum from s P sub sl sup 2 . +.EN + +The interpretation of this extraction is that the variance of the intensities +is constant. It gives greater weight to the stronger parts of the spectrum +profile than does the profile weighting (3a) since the weights are +$P sub sl sup 2$. Equation (4) has the virtue that one need not know the +readout noise or the ADU to photon number conversion. +.NH 3 +Optimal Extraction +.PP +Variance weighted extraction is sometimes called optimal extraction because +it is optimal in a statistical sense. Specifically, +the relative contribution of a pixel to the sum is related to the uncertainty +of its intensity. The uncertainty is measured by the expected variance of +a pixel with that intensity. The degree of optimality depends on how well +the relative variances of the pixels are known. +.PP +A discussion of the concepts behind optimal extraction is given in the paper +\fIAn Optimal Extraction Algorithm for CCD Spectroscopy\fR by Keith Horne +(\fBPASP\fR, June 1986). The weighting described in Horne's paper is the +same as the variance weighting described in this paper. The differences +in the algorithms are primarily in how the model profiles $P sub sl$ are +determined. +.NH 3 +Profile Determination +.PP +The profiles of the spectra along the spatial axis are determined when +either the detection and replacement of bad pixels or variance +weighting are specified. The requirements on the profiles are that +they have the same shape as the image profiles at a each dispersion +point and that they be as noise free and uncontaminated as possible. +The algorithm used to create these profiles is to average a specified +number of consecutive background subtracted image profiles immediately +preceding the wavelength to which a profile refers. When there are an +insufficient number of image profiles preceding the wavelength being +extracted then the following image profiles are also used to make up +the desired number. The image profiles are interpolated to a common +center before averaging using the curve given in the aperture +definition. The averaging reduces the noise in the image data while +the centering eliminates shifts in the spectrum as a function of +wavelength which would broaden the profile relative to the profile of a +single image line or column. It is assumed that the spectrum profile +changes slowly with wavelength so that by using profiles near a given +wavelength the average profile shape will correctly reflect the profile +of the spectrum at that wavelength. +.PP +The average profiles are determined in parallel with the extraction, +which proceeds sequentially through the image. Initially the first set +of spectrum profiles is read from the image and interpolated to a common +center. The profiles are averaged excluding the first profile to be +extracted; the image profiles in the average never include the image +profile to be extracted. Subsequently the average profile is updated +by adding the last extracted image profile and subtracting the image +profile which no longer belongs in the average. This allows each image +profile to be accessed and interpolated only once and makes the +averaging computationally efficient. This scheme also allows excluding +bad pixels from the average profile. The average profile is used to +locate and replace bad pixels in the image profile being extracted as +discussed in the following sections. Then when this profile is added +into the average for the next image profile the detected bad pixels are +no longer in the profile. +.PP +In summary this algorithm for determining the spectrum profile +has the following advantages: + +.IP (1) +No model dependent smoothing is done. +.IP (2) +There is no assumption required about the shape of the profile. +The only requirement is that the profile shape change slowly. +.IP (3) +Only one pass through the image is required and each image profile +is accessed only once. +.IP (4) +The buffered moving average is very efficient computationally. +.IP (5) +Bad pixels are detected and removed from the profile average as the +extraction proceeds. + +.NH 3 +Detection and Elimination of Bad Pixels +.PP +One of the important features of the aperture extraction package is the +detection and elimination of bad pixels. The average profile described +in the previous section is used to find pixels which deviate from this +profile. The algorithm is straightforward. A model spectrum of the +image profile is obtained by scaling the normalized profile to the +image profile. The scale factor is determined using chi-squared fitting: + +.EQ I (6) +M sub sl~=~P sub sl~left { sum from s ((I sub sl - B sub sl ) P sub sl / +V sub sl )~/~ sum from s (P sub sl sup 2 / V sub sl ) right } . +.EN + +The RMS of this fit is determined and pixels deviating by more than a +user specified factor times this RMS are rejected. The fit is then +repeated excluding the rejected points. These steps are repeated until +the user specified number of points have been rejected or no further deviant +points are detected. The rejected points in the image profile are then +replaced by their model values. +.PP +This algorithm is based only on the assumption that the spatial profile +of the spectrum (no matter what it is) changes slowly with wavelength. +It is very sensitive at detecting departures from the expected +profile. It has two problems currently. Because the input line is +first interpolated to the same center as the profile, single bad pixels +are generally broadened to two bad pixels, making it harder to find the +bad data. Also, in the first pass at the fit all of the image profile +is used so if there is a very badly deviant point and the rest of the +profile is weak then the scale factor may favor the bad pixel more than +the rest of the profile. This may result in rejecting good profile +points and not the bad pixel. +.NH 3 +Relation of Optimal Extraction to Model Extraction +.PP +Equation (1) defines the extraction process in terms of a weighted sum +of the pixel intensities. However, the actual extraction operations +performed by the task \fBapsum\fR are + +.EQ I (7a) +f sub lambda~mark =~sum from s (I sub sl - B sub sl ) +.EN +.EQ I (7b) +f sub lambda~lineup =~sum from s M sub sl +.EN + +where $M sub sl$ is the model spectrum fit to the background subtracted +image spectrum $(I sub sl - B sub sl )$ +defined in the previous section (equation 6). It is not obvious at first that +(7b) is equivalent to (3b). However, if one sums (6) and uses the fact +that the sum of the normalized profile is unity one is left with equation (3b). +.PP +Equations (6) and (7b) provide an alternate way to think about the +extracted one dimensional spectra. Sum extraction of the model spectrum +is used instead of the weighted sum for variance weighted extraction +because the model spectrum is a product of the profile determination +and the bad pixel cleaning process. It is then more convenient +and efficient to use the simple equations (7). +.NH 2 +Strip Extraction +.PP +The task \fBapstrip\fR uses one dimensional image interpolation +to shift the pixels along the spatial axis so that in the resultant +output image the center of the aperture is exactly aligned with the +image lines or columns. The cleaning of bad pixels is an option +in this extraction using the methods described above. In addition +the model spectrum, described above, may be extracted as a two +dimensional image. In fact, the only difference between strip extraction +and sum extraction is whether the final step of summing the pixels +in the aperture along the spatial axis is performed. +.PP +The primary use of \fBapstrip\fR is as a diagnostic tool. It +allows the user to see the background subtracted, cleaned, and/or model +spectrum as an image before it is summed to a one dimensional spectrum. +In addition the two dimensional format allows use of other IRAF tools such as +smoothing operators. When appropriate +it is a much simpler method of removing detector distortions and alignment +errors than the full two dimensional mapping and image transformation +available with the \fBlongslit\fR package. +.NH 2 +Aperture Normalization +.PP +The special algorithm/task \fBapnormalize\fR normalizes the two dimensional +image data within an aperture by a smooth function of the dispersion +coordinate. Unlike the extraction tasks the output of this algorithm is +a two dimensional image of the same format as the input image. This function +is used primarily for creating flat field images in which the large +scale shape of the quartz spectra and the variations in level between the +spectra are removed and the regions between the spectra, where there is no +signal, are set to unity. It may also be used to normalize two dimensional +spectra to a unit continuum at some point in the spectrum, such as the center. +.PP +The algorithm is to extract a one dimensional spectrum for each aperture, +fit a smooth function to the spectrum, and then divide this spectrum +back into the two dimensional image. Points outside the apertures are +set to 1. This is the same algorithm used in the \fBlongslit\fR package +by the task \fBresponse\fR except that it applies to arbitrary apertures +rather than to image sections. +.PP +Apertures are defined in the same way as for extraction. The normalization +spectrum may be obtained from a different aperture than the aperture to be +normalized. Generally the normalization apertures are either the same or +narrower than the apertures to be normalized. The continuum fitting also +uses the \fBicfit\fR package. Sample regions and iterative sigma clipping +are used to remove spectral lines from the continuum fits. +.PP +There are two commonly used approaches to fitting the extracted spectra +in flat field images. First, a constant function is fit. This has the +effect of simply normalizing the apertures to near unity without affecting +the shape of spectra in any way. This removes response effects at all scales, +from spectra flatten with this flat field. However, it does not +preserve total counts, it introduces the shape of the quartz spectrum, +and it removes the blaze function. The second approach is to fit the +large scale shape of the quartz spectra. This removes smaller scale +response effects such a fringing and individual pixel responses while +preserving the total counts by leaving the blaze function alone. There are +cases where each of these approaches is applicable. +.NH +Apertures +.PP +Apertures are the basic data structures used in the package; hence the +package name. An aperture defines a region of the two dimensional image +to be extracted. The aperture definitions are stored in a database. +An aperture consists of the following components: + +.IP ID +.br +An integer identification number. The identification number must be +unique. It is used as the default extension during extraction of +the spectra. Typically the IDs are consecutive positive integers +ordered by increasing or decreasing slit position. +.IP BEAM +.br +An integer beam number. The beam number need not be +unique; i.e. several apertures may have the same beam number. +The beam number will be recorded in the image header of the +the extracted spectrum. By default the beam number is the same as +the ID. +.IP CENTER[2] +.br +The center of the aperture along the slit and dispersion axes in the two +dimensional image. +.IP LOWER[2] +.br +The lower limits of the aperture, relative to the aperture center, +along the slit and dispersion axes. The lower limits need not be less +than the center. +.IP UPPER[2] +.br +The upper limits of the aperture, relative to the aperture center, +along the slit and dispersion axes. The upper limits need not be greater +than the center. +.IP APAXIS +.br +The aperture or spatial axis. +.IP CURVE +.br +An offset to be added to the center position for the aperture axis as +a function of the wavelength. The function is one of the standard IRAF +types; a legendre polynomial, a chebyshev polynomial, a linear spline, +or a cubic spline. +.IP BACKGROUND +.br +Parameters for background subtraction along the aperture axis based on +the interactive curve fitting (\fBicfit\fR) tools. + +.PP +The aperture center is the only absolute coordinate (relative to the +image or image section). The other aperture parameters and the +background fitting regions are defined relative to the center. Thus, +an aperture may be repositioned easily by changing the center +coordinates. Also constant aperture size, shape (curve), and +background regions may be maintained for many apertures. +.PP +The edges of the aperture along the spatial axis at each point along the +dispersion axis are given by evaluating the offset curve at that dispersion +coordinate and adding the aperture center and the lower or upper limits +for the aperture axis. The edges of the aperture along the dispersion axis +do not have an offset curve and are currently fixed to define the entire +length of the image. In the future this may not be the case such as +in applications with objective prism spectra. +.PP +Apertures for a particular image may be defined in several ways. They +may be defined and modified graphically with an aperture editor. Default +apertures may be defined automatically with parameters from the +\fBapdefault\fR pset using an aperture finding algorithm. Another +method is to specify that the apertures for one image use the aperture +definitions from another "reference" image. In the rare cases where +apertures are not defined at the stage of tracing or extracting then +a single default aperture centered in the image is created. +.NH 2 +The Database +.PP +The aperture information is stored in a database. The structure and type of +database is expected to change in the future and as far as the package and +user need be concerned it is just a black box with some name specified in +the database name parameter. However, accepting that the database structure may +change it may be of use to the user to understand the nature of the current +text file / directory format database. The database is a directory containing +text files. It is automatically created if necessary. The aperture data +for all the apertures from a single image are stored in a text file +with the name given by the image name (with special characters replaced +with '_') prefixed with "ap". Updates of the aperture data are performed +by overwriting the database file. +.PP +The content of a file consists of a comment (beginning with a #) giving +the date created/updated, a record identification (there is one record +per aperture) with the image name, aperture number and aperture +coordinate in the aperture and dispersion axes. The following lines +give information about the aperture. The position and shape of an +aperture is given by a center coordinate along the aperture axis (given +by the axis keyword) and the dispersion axis. There are lower and +upper limits for the aperture relative to this center, again along both +axis. Currently the limits along the dispersion axis are the image +boundaries. The background keyword introduces the background +subtraction parameters. Finally there is an offset or trace function +which is added to the center at each point along the dispersion axis. +function. The offset is generally zero at the dispersion point +corresponding to the aperture center. +.PP +This offset or trace function is described by a \fBcurfit\fR array under +the keyword curve. The first value is the number of elements in this +array. The first element is a magic number specifying the function +type. The next number is the order or number of spline pieces. The +next two elements give the range over which the curve is defined. In +the \fBapextract\fR case it is the edges of the image along the dispersion. +The remaining elements are the function coefficients. The form of the +the function is specific to the IRAF \fBcurfit\fR math routines. Note that +the coefficients apply to an independent variable which is -1 at the +beginning of the defined range (element 3) and 1 at the end of the range +(element 4). For further details consult the IRAF group. +.PP +An example database file for one aperture from an image "ech001" is given +below. + +.ft L +.nf + # Fri 14:33:35 08-May-87 + begin aperture ech001 1 22.75604 100. + image ech001 + aperture 1 + beam 1 + center 22.75604 100. + low -2.680193 -99. + high 3.910698 100. + background + xmin -262. + xmax 262. + function chebyshev + order 1 + sample -10:-6,6:10 + naverage -3 + niterate 0 + low_reject 3. + high_reject 3. + grow 0. + axis 1 + curve 6 + 2. + 2. + 1. + 200. + -0.009295368 + -0.3061974 +.fi +.ft R +.NH +Future Developments +.PP +The IRAF extraction package \fBapextract\fR is going to continue to +evolve because the extraction of one and two dimensional spectra +from two dimensional images is an important part of reducing echelle, +longslit, multislit, and multiaperture spectra. Changes may include +some of the following: + +.IP (1) +Determine the actual variance from the data rather than using the Poisson +CCD model. Also output the variance vector if desired. +.IP (2) +The bad pixel detection and removal algorithm does not handle well the case +of a very strong cosmic ray event on top of a very weak spectrum profile. +A heuristic method to make the first fitting pass of the average +profile to the image data less prone to errors due to strong cosmic rays +is needed. Also the detection should be done by interpolating the profile +to the original image data rather than the other way around, in order to +avoid broadening cosmic rays by interpolation. +.IP (3) +The aperture definition structure is general enough to allow the aperture +limits along the dispersion dimension to be variable. Eventually aperture +definition and editing will be available using an image display. Then +both graphics and image display editing switches will be available. +An image display interface will make extraction of objective prism +spectra more convenient than it is now. +.IP (4) +Other types of extraction weighting may be added. diff --git a/noao/twodspec/apextract/doc/old/apextract2.ms b/noao/twodspec/apextract/doc/old/apextract2.ms new file mode 100644 index 00000000..35b42390 --- /dev/null +++ b/noao/twodspec/apextract/doc/old/apextract2.ms @@ -0,0 +1,14 @@ +.RP +.TL +Cleaning and Optimal Extraction with the IRAF APEXTACT Package +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +.AB +.AE +.NH +Introduction +.PP diff --git a/noao/twodspec/apextract/doc/revisions.v3.ms b/noao/twodspec/apextract/doc/revisions.v3.ms new file mode 100644 index 00000000..f78362a5 --- /dev/null +++ b/noao/twodspec/apextract/doc/revisions.v3.ms @@ -0,0 +1,522 @@ +.nr PS 9 +.nr VS 11 +.RP +.ND +.TL +APEXTRACT Package Revisions Summary: IRAF Version 2.10 +.AU +Francisco Valdes +.AI +IRAF Group - Central Computer Services +.K2 +P.O. Box 26732, Tucson, Arizona 85726 +September 1990 +.AB +This paper summarizes the changes in Version 3 of the IRAF \fBapextract\fR +package which is part of IRAF Version 2.10. The major new features and +changes are: + +.IP \(bu +New techniques for cleaning and variance weighting extracted spectra +.IP \(bu +A new task, \fBapall\fR, which integrates all the parameters used for +one dimensional extraction of spectra +.IP \(bu +A new extended output format for recording both weighted and unweighted +extractions, subtracted background, and variance information. +.IP \(bu +Special featurers for automatically numbering and identifying large +numbers of apertures. +.IP \(bu +New tasks and algorithms, \fBaprecenter\fR and \fBapresize\fR, +for automatically recentering and resizing aperture definitions +.IP \(bu +A new task, \fBapflatten\fR, for creating flat fields from +fiber and slitlet spectra +.IP \(bu +A new task, \fBapfit\fR, providing various types of fitting for +two dimensional multiobject spectra. +.IP \(bu +A new task, \fBapmask\fR, for creating mask images from aperture definitions. +.AE +.NH +Introduction +.PP +A new version of the IRAF \fBapextract\fR package has been completed. +It is Version 3 and is part of IRAF Version 2.10. The package will +be made available as an external package prior to the release of V2.10. +This paper describes the changes and new features of the package. It +does not describe them in detail. Full details of the algorithms, +functions, and parameters are found in the task descriptions. +Reference is made to the previous version so familiarity with that +version is useful though not necessary. There were three goals for the +new package: new and improved cleaning and variance weighting (optimal +extraction) algorithms, the addition of recommended or desirable new +tasks and algorithms (particularly to support large numbers of spectra +from fiber and aperture mask instruments), and special support for the +new image reduction scripts. Features relating to the last point are +not discussed here. +.PP +Table 1 summarizes the major new features and changes in the package. + +.ce +Table 1: Summary of Major New Features and Changes + +.IP \(bu +New techniques for cleaning and variance weighting extracted spectra +.IP \(bu +A new task, \fBapall\fR, which integrates all the parameters used for +one dimensional extraction of spectra +.IP \(bu +A new extended output format for recording both weighted and unweighted +extractions, subtracted background, and variance information. +.IP \(bu +Special featurers for automatically numbering and identifying large +numbers of apertures. +.IP \(bu +New tasks and algorithms, \fBaprecenter\fR and \fBapresize\fR, for +automatically recentering and resizing aperture definitions +.IP \(bu +A new task, \fBapflatten\fR, for creating flat fields from fiber and slitlet +spectra +.IP \(bu +A new task, \fBapfit\fR, providing various types of fitting for two dimensional +multiobject spectra. +.IP \(bu +A new task, \fBapmask\fR, for creating mask images from aperture definitions. +.NH +Cleaned and Variance Weighted Extractions: apsum and apall +.PP +There are two types of aperture extraction (estimating the background +subtracted flux across a fixed width aperture at each image line or +column) just as in the previous version. One is a simple sum of pixel +values across an aperture. In the previous version this was called +"profile" weighting while in this version it is simply called +unweighted or "none". The second type weights each pixel in the sum by +its estimated variance based on a spectrum model and detector noise +parameters. As before this type of extraction is selected by +specifying "variance" for the weighting parameter. +.PP +Variance weighting is often called "optimal" extraction since it +produces the best unbiased signal-to-noise estimate of the flux in the +two dimensional profile. It also has the advantage that wider +apertures may be used without penalty of added noise. The theory and +application of this type of weighting has been described in several +papers. The ones which were closely examined and used as a model for +the algorithms in this software are \fIAn Optimal Extraction Algorithm +for CCD Spectroscopy\fR, \fBPASP 98\fR, 609, 1986, by Keith Horne and +\fIThe Extraction of Highly Distorted Spectra\fR, \fBPASP 100\fR, 1032, +1989, by Tom Marsh. +.PP +The noise model for the image data used in the variance weighting, +cleaning, and profile fitting consists of a constant gaussian noise and +a photon count dependent poisson noise. The signal is related to the +number of photons detected in a pixel by a gain parameter given +as the number of photons per data number. The gaussian noise is given +by a readout noise parameter which is a defined as a sigma in +photons. The poisson noise is approximated as gaussian with sigma +given by the number of photons. The method of specifying this noise +model differs from the previous version in that the more common CCD +detector parameters of readout noise and gain are used rather than the +linear variance parameters "v0" and "v1". +.PP +Some additional effects which should be considered in principle, and +which are possibly important in practice, are that the variance +estimate should be based on the actual number of photons detected before +correction for pixel sensitivity; i.e. before flat field correction. +Furthermore the uncertainty in the flat field should also be included +in the weighting. However, the profile must be determined free of +sensitivity effects including rapid larger scale variations such as +fringing. Thus, ideally one should input the unflat-fielded +observation and the flat field data and carry out the extractions with +the above points in mind. However, due to the complexity often +involved in basic CCD reductions and special steps required for +producing spectroscopic flat fields this level of sophistication is not +provided by the current package. +.PP +The package does provide, however, for propagation of an approximate +uncertainty in the background estimate when using background subtraction. +If background subtraction is done, a background variance is computed +using the poisson noise model based on the estimated background counts. +Because the background estimate is based on a finite number of +pixels, the poisson variance estimate is divided by the number (minus +one) of pixels used in determining the background. The number of +pixels used includes any box car smoothing. Thus, the larger the +number of background pixels the smaller the background noise +contribution to the variance weighting. This method is only +approximate since no correction is made for the number of degrees of +freedom and correlations when using the fitting method of background +estimation. +.PP +If removal of cosmic rays and other deviant pixels is desired (called +cleaning) they are iteratively rejected based on the estimated variance +and excluded from the weighted sum. Unlike the previous version, a +cleaned extraction is always variance weighted. This makes sense since +the detector noise parameters must be specified and the spectrum +profile computed, so all of the computational effort must be done +anyway, and the variance weighting is as good or superior to a simple +unweighted extraction. +.PP +The detection and removal of deviant pixels is straightforward. Based +on the noise model, pixels deviating by more than a +specified number of sigma (square root of the variance) above or below +the model are removed from the weighted sum. A new spectrum estimate +is made and the rejection is repeated. The rejections are made one at +a time starting with the most deviant and up to half the pixels in the +aperture may be rejected. +.NH +Spectrum Profile Determination: apsum, apall, apflatten, apfit +.PP +The foundation of variance weighted or optimal extraction, cosmic ray +detection and removal, two dimensional flat field normalization, and +spectrum fitting and modeling is the accurate determination of the +spectrum profile across the dispersion as a function of wavelength. +The previous version of the \fBapextract\fR package accomplished this by +averaging a specified number of profiles in the vicinity of each +wavelength after correcting for shifts in the center of the profile. +This technique was sensitive to perturbations from cosmic rays +and the exact choice of averaging parameters. The current version of +the package uses a different algorithm, actually a combination of +two algorithms, which is much more stable. +.PP +The basic idea is to normalize each profile along the dispersion to +unit flux and then fit a low order function to sets of unsaturated +points at nearly the same point in the profile parallel to the +dispersion. The important point here is that points at the same +distance from the profile center should have the nearly the same values +once the continuum shape and spectral features have been divided out. +Any variations are due to slow changes in the shape of the profile with +wavelength, differences in the exact point on the profile, pixel +binning or sampling, and noise. Except for the noise, the variations +should be slow and a low order function smoothing over many points will +minimize the noise and be relatively insensitive to bad pixels such as +cosmic rays. Effects from bad pixels may be further eliminated by +chi-squared iteration and clipping. Since there will be many points +per degree of freedom in the fitting function the clipping may even be +quite aggressive without significantly affecting the profile +estimates. Effects from saturated pixels are minimized by excluding +from the profile determination any profiles containing one or more +saturated pixels. +.PP +The normalization is, in fact, the one dimensional spectrum. Initially +this is the simple sum across the aperture which is then updated +by the variance weighted sum with deviant pixels possibly removed. +This updated one dimensional spectrum is what is meant by the +profile normalization factor in the discussion below. The two dimensional +spectrum model or estimate is the product of the normalization factor +and the profile. This model is used for estimating +the pixel intensities and, thence, the variances. +.PP +There are two important requirements that must be met by the profile fitting +algorithm. First it is essential that the image data not be +interpolated. Any interpolation introduces correlated errors and +broadens cosmic rays to an extent that they may be confused with the +spectrum profile, particularly when the profile is narrow. This was +one of the problems limiting the shift and average method used +previously. The second requirement is that data fit by the smoothing +function vary slowly with wavelength. This is what precludes, for +instance, fitting profile functions across the dispersion since narrow, +marginally sampled profiles require a high order function using only a +very few points. One exception to this, which is sometimes useful but +of less generality, is methods which assume a model for the profile +shape such as a gaussian. In the methods used here there is no +assumption made about the underlying profile other than it vary +smoothly with wavelength. +.PP +These requirements lead to two fitting algorithms based on how well the +dispersion axis is aligned with the image columns or lines. When the +spectra are well aligned with the image axes one dimensional functions +are fit to the image columns or lines. Small excursions of a few +pixels over the length of the spectrum can be adequately fit in this +way. When the spectra become strongly tilted then single lines or +columns may cross the actual profile relatively quickly causing the +requirement of a slow variation to be violated. One thought is to use +interpolation to fit points always at the same distance from the +profile. This is ruled out by the problems introduced by image +interpolation. However, there is a clever method which, in effect, +fits low order polynomials parallel to the direction defined by tracing +the spectrum but which does not interpolate the image data. Instead it +weights and couples polynomial coefficients. This method was developed +by Tom Marsh and is described in detail in the paper, \fIThe Extraction +of Highly Distorted Spectra\fR, \fBPASP 101\fR, 1032, Nov. 1989. Here +we refer to this method as the Marsh algorithm and do not attempt to +explain it further. +.PP +Both fitting algorithms weight the pixels by their variance as computed +from the background and background variance if background subtraction +is specified, the spectrum estimate from the profile and the spectrum +normalization, and the detector noise parameters. The noise model is +that described earlier. +.PP +The profile fitting can be iterated to remove deviant pixels. This is +done by rejecting pixels greater than a specified number of sigmas +above or below the expected value based on the profile, the +normalization factor, the background, the detector noise parameters, +and the overall chi square of the residuals. Rejected points are +removed from the profile normalization and from the fits. +.NH +New Extraction Task: apall +.PP +All of the functions of the \fBapextract\fR package are actually part +of one master program. The organization of the package into tasks by +function with parameters to allow selection of some of the other +functions, for example the aperture editor may be entered from +virtually every task, was done to highlight the logic and organize the +parameters into small sets. However, there was often confusion about +which parameters were being used and the need to set parameters in one +task, say \fBaptrace\fR, in order to use the trace option in another +task, say \fBapsum\fR. In practice, for the most common function of +extraction of two dimensional spectra to one dimension most users end up +using \fBapsum\fR for all the functions. +.PP +In the new version, the old organization is retained (with the addition +of new functions and some changes in parameters) but a new task, +\fBapall\fR, is also available. This task contains all of the +parameters needed for extraction with a parameter organization which is +nicely formated for use with \fBeparam\fR. The parameters in +\fBapall\fR are independent of the those in the other tasks. It is +expected that many, if not most users will opt to use this task for +spectrum extraction in preference to the individual functions. +.PP +The organization by function is still used in the documentation. This +is still the best way to organize the descriptions of the various +algorithms and parameters. As an example, the profile tracing algorithm +is described in most detail under the topic \fBaptrace\fR. +.NH +Extraction Output Formats: apsum and apall +.PP +The extracted spectra are recorded in one, two, or three dimensional +images depending on the \fIformat\fR and \fIextras\fR parameters. If +the \fIextras\fR parameter is selected the formats are three +dimensional with each plane in the third dimension containing +associated information for the spectra in the first plane. This +information includes the unweighted spectrum and a sigma spectrum +(estimated from the variances and weights of the pixels extracted) when +using variance weighting, and the background spectrum when background +subtraction is used. When \fIextras\fR is not selected only the +extracted spectra are output. +.PP +The formats are basically the same as in the previous version; +onedspec, multispec, and echelle. In addition, the function of the +task \fBapstrip\fR in the previous version has been transferred to the +extraction tasks by simply specifying "strip" for the format. +.PP +There are some additions to the header parameters in multispec and +echelle format. Two additional fields have been added to the +aperture number parameter giving the aperture limits (at the reference +dispersion point). Besides being informative it may be used for +interpolating dispersion solutions spatially. A second, optional keyword per +spectrum has been added to contain a title. This is useful for +multiobject spectra. +.NH +Easier and Extended Aperture Identifications: apfind and apedit +.PP +When dealing with a large number of apertures, such as occur with +multifiber and multiaperture data, the burden of making and maintaining +useful aperture identifications becomes large. Several very useful +improvements were made in this area. These improvements generally +apply equally to aperture identifications made by the automated +\fBapfind\fR algorithm and those made interactively using +\fBapedit\fR. In the simplest usage of defining apertures +interactively or with the aperture finding algorithm, aperture numbers +are assigned sequentially beginning with 1. In the new version the +parameter "order" allows the direction of increasing aperture numbers +with respect to the direction of increasing pixel coordinate (either +column or line) to be set. An "increasing" order parameter value +numbers the apertures from left to right (the direction naturally +plotted) in the same sense as the pixel coordinates. A "decreasing" +order reverses this sense. +.PP +Some instruments, particularly multifiber instruments, produce nearly +equally spaced spectra for which one wants to maintain a consistent +numbering sequence. However, at times some spectra may be missing +due to broken or unassigned fibers and one would like to skip an +aperture identification number to maintain the same fiber assignments. +To do this automatically, a new parameter called \fImaxsep\fR has been +added. This parameter defines the maximum separation between two +apertures beyond which a jump in the aperture sequence is made. In +other words the sequence increment is given by rounding down the +separation divided by this parameter. How accurately this value has +to be specified depends on how large the gaps may be and the natural +variability in the aperture positions. In conjunction with the +minimum separation parameter this algorithm works quite well in +accounting for missing spectra. +.PP +One flaw in this scheme is when the first spectrum is missing causing +the identifications will be off. In this case the modified interactive +aperture editing command 'o' asks for the aperture identification +number of the aperture pointed at by the cursor and then automatically +renumbers the other apertures relative to that aperture. The other +possible flaw is identification of noise as a spetrum but this is +controlled by the \fIthreshold\fR parameter and, provided the actual +number of spectra is known, say by counting off a graph, then the +\fInfind\fR parameter generally limits this type of problem. +.PP +A new attribute of an aperture is a title. If present this title +is propagated through the extraction into the image headers. The title +may be set interactively but normally the titles are supplied in +another new feature, an "aperture identification" file specified by +the parameter \fIapidtable\fR. This file provides +the most flexibility in making aperture identification assignments. +The file consists of lines with three fields, a unique aperture number, +a beam or aperture type number which may be repeated, and the +aperture title. The aperture identification lines from the file are +assigned sequentially in the same order as would be done if using +the default indexing including skipping of missing spectra based on +the maximum separation. +.PP +By default the beam number is the same as the aperture number. When +using an aperture identification file the beam number can be used +to assign spectrum types which other software may use. For example, +some of the specialized fiber reduction packages use the beam number +to identify sky fibers and embedded arc fibers. +.NH +New Aperture Recentering Task: aprecenter +.PP +An automated recentering algorithm has been added. It may be called +through the new \fBaprecenter\fR command, from any of the tasks containing +the \fIrecenter\fR parameter, or from the aperture editor. The purpose of +this new feature is to allow automatically adjusting the aperture +centers to follow small changes in the positions of spectra expected to be at +essentially the same position, such as with fiber fed spectrographs. +This does not change the shape of the trace but simply adds a shift +across the dispersion axis. +.PP +Typically, one uses a strong image to define reference apertures and +then for subsequent objects uses the reference positions with a +recentering to correct for flexure effects. However, it may be +inappropriate to base a new center on weak spectra or to have multiple +spectra recentered independently. The recentering options provide for +selecting specific apertures to be recentered, selecting only a +fraction of the strongest (highest peak data level) spectra and +averaging the shifts determined (possible from only a subset of the +spectra) and applying the average shift to all the apertures. +Note that one may still specify the dispersion line and number of +dispersion lines to sum in order to improve the signal for centering. +.NH +New Aperture Resizing Task: apresize +.PP +An automated resizing algorithm has been added. It may be called +through the new \fBapresize\fR command, from any of the tasks +containing the \fIresize\fR parameter, or from the aperture editor with +the new key 'z' (the y cursor level command is still available with the +'y' key). The purpose of this new feature is to allow automatically +adjusting the aperture widths to follow changes in seeing and to +provide a greater variety of global aperture sizing methods. +.PP +In all the methods the aperture limits are set at the pixel positions +relative to the center which intersect the linearly interpolated data +at some data value. The methods differ in how the data level is +determined. The methods are: + +.IP \(bu +Set size at a specified absolute data level +.IP \(bu +Set size at a specified data level above a background +.IP \(bu +Set size at a specified fraction of the peak pixel value +.IP \(bu +Set size at a specified fraction of the peak pixel value above a background +.LP +The automatic background is quite simple; a line connecting the first +local minima from the aperture center. +.PP +The limits determined by one of the above methods may be further +adjusted. The limits may be increased or decreased by a specified +fraction. This allows setting wider limits based on more accurately +determined limits from the stronger part of the profile; for example +doubling the limits obtained from the half intensity point. A maximum +extent may be imposed. Finally, if there is more than one aperture and one +wants to maintain the same aperture size, the apertures sizes +determined individually may be averaged and substituted for all the +apertures. +.NH +New Aperture Mask Output: apmask +.PP +A new task, \fBapmask\fR, has been added to produce a mask file/image +of 1's and 0's defined by the aperture definitions. This is based on +the new IRAF mask facilities. The output is a compact binary file +which may be used directly as an image in most applications. In +particular the mask file can be used with tasks such as \fBimarith\fR, +\fBimreplace\fR, and \fBdisplay\fR. Because the mask facility is new, +there is little that can be done with masks other than using it as an +image. However, eventually many tasks will be able to use mask +images. The aperture mask will be particularly well suited to work +with \fBimsurfit\fR for fitting a surface to the data outside the apertures. +This would be an alternative for scattered light modeling to the +\fBapscatter\fR tasks. +.NH +Aperture Flat Fields and Normalization: apflatten and apnormalize +.PP +Slitlet, echelle, and fiber spectra have the characteristic that the +signal falls off to near zero values outside regions of the image +containing spectra. Also fiber profiles are usually undersampled +causing problems with gradients across the pixels. Directly dividing +by a flat field produces high noise (if not division by zero) where the +signal is low, introduces the spectrum of the flat field light, and +changes the profile shape. +.PP +One method for modifying the flat field to avoid these problems is +provided by the task \fBimred.generic.flat1d\fR. However, this task +does not use any knowledge of where the spectra are. There are two +tasks in the \fBapextract\fR package which can be used to modify flat +field images. \fBapnormalize\fR is not new. It divides the spectra +within specified apertures by a one dimensional spectrum, either a +constant for simple throughput normalization or some smoothed version +of the spectrum in the aperture to remove the spectral shape. Pixels +outside specified apertures are set to unity to avoid division +effects. This task has the effect of preserving the profile shape in +the flat field which may be desired for attempts to remove slit +profiles. +.PP +Retaining the profile shape of the flat field can give very bad edge +effects, however, if there is image flexure. A new task similar to +\fBflat1d\fR but which uses aperture information is \fBapflatten\fR. +It uses the spectrum profile model described earlier. For nearly image +axes aligned spectra this amounts very nearly to the line or column +fitting of \fBflat1d\fR. As with \fBapnormalize\fR there is an option +to fit the one dimensional spectrum to remove the large scale shape of +the spectrum while preserving small scale sensitivity variations. The +smoothed spectrum is multiplied by the normalized profile and divided +into the data in each aperture. Pixels outside the aperture are set to +1. Pixels with model values below a threshold are also set to 1. This +produces output images which have the small scale sensitivity +variations, a normalized mean, and the spectrum profile removed. +.NH +Two Dimensional Spectrum Fitting: apfit +.PP +The profile and spectrum fitting used for cleaning and variance +weighted extraction may be used and output in the new task +\fBapfit\fR. The task \fBapfit\fR is similar in structure to +\fBfit1d\fR. One may output the fit, difference, or ratio. The fit +may be used to examine the spectrum model used for the cleaning and +variance weighted extraction. The difference and ratio may used to +display small variations and any deviant pixels. While specific uses +are not given this task will probably be used in interesting ways not +anticipated by the author. +.NH +I/O and Dispersion Axis Parameters: apextract and apio +.PP +The general parameters, primarily concerning input and output devices +and files, were previously in the parameter set \fBapio\fR. This "pset" +task has been removed and those parameters are now found as part of +the package parameters, i.e. \fBapextract\fR. There is one new parameter +in the \fBapextract\fR package parameters, dispaxis. In the previous +version of the package one needed to run the task \fBsetdisp\fR to insert +information in the image header identifying the dispersion direction +of the spectra in the image. Often people would forget this step +and receive an error message to that effect. The new parameter +allows skipping this step. If the DISPAXIS image header parameter +is missing the package parameter value is inserted into the image +header as part of the processing. Note that if the parameter is +present in the image header either because \fBsetdisp\fR was used or the +image creation process inserted it (a future ideal case) then that +value is used in preference to the package parameter. +.NH +Strip Extraction: apstrip +.PP +The task \fBapstrip\fR from the previous version has been removed. +However, it is possible to obtain two dimensional strips aligned with +the image axes by specifying a format of "strip" when using \fBapsum\fR +or \fBapall\fR. While the author doesn't anticipate a good scientific +use for this feature others may find it useful. |