.RP
.TL
An Overview of the IRAF DTOI Package
.AU
Suzanne Hammond Jacoby
.AI
IRAF Group - Central Computer Services
.K2 "" "" "*"
February 1987
.br
Revised July 1988

.AB
This document describes the DTOI package, which contains tasks
for determining and applying a density to intensity transformation to
photographic data.  The transformation is determined from a set
of calibration spots with known relative intensities.  A curve is
interactively fit to the densities and intensities of the calibration
spots.  The transformation is then applied and a new output image written.
.AE

.NH
Introduction
.PP
The DTOI package contains tasks for computing and applying a density
to intensity transformation to photographic data.  These tasks perform the
standard steps in linearizing data: calculating HD data points from 
calibration spots, fitting a curve to these points and applying the HD
curve to the data.  It is also possible to combine related HD curves.
Communication between the tasks is via text files which the user can 
inspect or modify.  It is intended
to be easy for users to introduce data from outside the DTOI package
into the processing.
.PP
There are currently six tasks in the package.  They are:

.ce
The \fBDTOI\fR Package
.TS
center;
n.
spotlist \&- Calculate densities and weights of calibration spots.
dematch \&- Match densities to log exposure values.
hdfit \&- Fit characteristic curve to density, exposure data.
hdtoi \&- Apply HD transformation to image data.
hdshift \&- Align related characteristic curves.
selftest \&- Test transformation algorithm.
.TE
.PP
The DTOI package does not currently support the self calibration of images, 
but the addition of this capability is planned.  This would involve
determining the HD curve from the data itself, by assuming the point spread
function scales linearly with intensity.
.PP
Upon entering the package, your calibration spots and images to be
transformed should be on the disk in IRAF image format.

.NH 
Determining the HD Curve Data 
.PP
To determine the HD curve, you need two sets of data:  the
measured photographic densities of a set of calibration spots and
the log exposure values corresponding to these measurements.  The
log exposure values must be known a priori.  Tasks \fIspotlist\fR and
\fIdematch\fR are used to assemble these two data sets.
.NH 2
SPOTLIST
.PP
The first step is to calculate the density of
the calibration spots, each of which is a separate IRAF image or image
section.  The spot density is either the median of the spot pixels or
the mean of the pixels when pixels more then a user specified number of
standard deviations away from the mean have been rejected.  The numbers
in the spot image must be scaled to density; parameter \fBspotlist.scale\fR
is used such that density = input_value * scale.  Task \fIspotlist\fR also
calculates the standard deviation of each spot and reports
the number of good pixels, i.e., the number of pixels not rejected
when determining the mean density.  
The final product of this task is a record in the data base containing a
density for each spot.  The scale factor used is also written to the data
base; it will be read later in task \fIhdtoi\fR.
.NH 2
DEMATCH
.PP
Log exposure values must be matched to the measured density values.  These
log exposure values must be known a priori and will be read from a file.
Task \fIdematch\fR retrieves the proper exposure information by
matching the wedge number, emulsion type and filter used.  Once a match
has been made, the proper log exposure values are written to a record
in the database.  
.PP
A database of log exposure values for the NOAO standard wedges is maintained 
in a system file; the wedge/emulsion/filter combinations available are listed
in last section of this document.  This file can be replaced with one specific 
to any institution; the file name is supplied to task \fIdematch\fR as a 
parameter.  In this way the wedge file can be personalized to any application 
and not be lost when the system is updated.

.NH
Fitting the Curve
.PP
The HD curve, or characteristic curve, is a plot of density versus log 
exposure.  This curve is determined from the data points generated by
tasks \fIspotlist\fR and \fIdematch\fR.  The objective is to fit
a curve to these points, such that Log exposure = F(Density).  The
technique available in this package allows the independent variable of the
fit to be a transformation of the density (log opacitance, for example).
The log exposure and density values are
read from the database.  If multiple entries for a particular record are 
present in the database, the last one is used.
.NH 2
HDFIT
.PP
Task \fIhdfit\fR fits a characteristic curve to density and log exposure
values in preparation for transforming an image from density to intensity.
Five functional forms of the curve are available:
.nf

 	  Power Series
    	  Linear Spline
    	  Cubic Spline
    	  Legendre Polynomials
    	  Chebyshev Polynomials
 	
.fi
.LP 
It is possible to apply a transformation to the 
independent variable (density above fog) prior to the fit.  The traditional 
choice is to fit log exposure
as a function of the log opacitance, rather than density directly.  This is
sometimes referred to as the Baker, or Seidel, function.  Transforming
the density has the effect of stretching the low density data points, which
tend to be relatively oversampled. 
In the DTOI package, four independendent variables are currently available:
.nf

 	Density
 	Log Opacitance
 	K50 - (Kaiser* Transform with alpha = 0.50)
 	K75 - (Kaiser* Transform with alpha = 0.75)

.fi
.FS
* Margoshes and Rasberry, Spectrochimica Acta, Vol 24B, p497, (1969)
.FE
Any combination of transformation type and fitting function can be used and
changed interactively.  Two combinations of interest are discussed here.

The default fit is a power series fit where the independent variable is 
Log Opacitance.  That is:
.LP
.EQ

	"Log Exposure = " sum from k=0 to {ncoeff - 1} {A sub k Y sup k}

.EN
.sp 1
.EQ
	"where Y = Log Opacitance = "Log sub 10 (10 sup Density - 1)
.EN
.LP
A fit that is expected to best model a IIIA-J emulsion is a power series
fit to a K75 transform of the density.  That is,
.LP
.EQ

	"Log Exposure = "sum from k=0 to {ncoeff - 1} {A sub k Y sup k}

.EN
.sp 1
.EQ
"where Y = K75 transform = Density + 0.75 " Log sub 10 (1 - 10 sup -Density )
.EN
.LP
Over the expected small dynamic range in variables of the fit, legendre
and chebyshev functions offer no advantages over a simple power series
functional form.  The cubic and linear spline fits may follow the data very
closely, but with typically sparse data sets this is not desirable.  It
is expected that power series fit will serve satisfactorily in all cases.

.NH 3
Interactive Curve Fitting
.PP
Task \fIhdfit\fR can be run interactively or not.  In interactive mode,
points in the sample can be edited, added or deleted.  Weighting values
can be changed as well as the fog value, the type of transformation
and the fitting function chosen.  To obtain the best fit possible, interactive
fitting is recommended.  A complete list of the available commands
is printed here; this list is also available interactively with the 
keystroke '\fL?\fR'.
.TS
center;
c s s w(3.0i)
c l s.

	DTOI INTERACTIVE CURVE FITTING OPTIONS 

\fL?\fR	Print options
\fLa\fR	Add the point at the cursor position to the sample
\fLc\fR	Print the coordinates and fit of point nearest the cursor
\fLd\fR	Delete data point nearest the cursor
\fLf\fR	Fit the data and redraw or overplot
\fLg\fR	T{
Redefine graph keys.  Any of the following data types may be along
either axis:
T}
.T&
l l l.
 	\fLx\fR  Independent variable	\fLy\fR  Dependent variable
 	\fLf\fR  Fitted value	\fLr\fR  Residual (y - f)
 	\fLd\fR  Ratio (y / f)	\fLn\fR  Nonlinear part of y
 	\fLu\fR  Density above fog

Graph keys:	 	 
.T&
c l s.

\fLh\fR	h = (x,y)  transformed density vs. log exposure
\fLi\fR	i = (y,x)  log exposure vs. transformed density
\fLj\fR	j = (x,r)  transformed density vs. residuals
\fLk\fR	k = (x,d)  transformed density vs. the y(data)/y(fit) ratio
\fLl\fR	l = (y,u)  log exposure vs. density above fog (HD Curve)

\fLo\fR	Overplot the next graph
\fLq\fR	T{
Terminate the interactive curve fitting, updating the database file. 
T}
\fLr\fR	Redraw graph
\fLu\fR	Undelete the deleted point nearest the cursor
\fLw\fR	Set the graph window.  For help type 'w' followed by '?'.
\fLx\fR	Change the x value of the point nearest the cursor
\fLy\fR	Change the y value of the point nearest the cursor
\fLz\fR	Change the weight of the point nearest the cursor

.T&
l s s w(3.0i).
T{
The parameters are listed or set with the following commands which may be
abbreviated.  To list the value of a parameter type the command alone.
T}

.T&
l l s.

\fL:show \fR[\fIfile\fR]	Show the values of all the parameters
\fL:vshow \fR[\fIfile\fR]	Show the values of all the parameters verbosely
\fL:errors \fR[\fIfile\fR]	Print the errors of the fit (default STDOUT)
\fL:reset \fR	T{
Return to original conditions of x, y, wts and npts.
T}
\fL:ebars \fR[\fIerrors/weights\fR]	T{
The size of marker type '[hv]ebars' can show either standard deviations or
relative weights.
T}
\fL:function \fR[\fIvalue\fR]	T{
Fitting function (power, chebyshev, legendre, spline3, or spline1)
T}
\fL:transform \fR[\fIvalue\fR]	Set the transform type (none, logo, k50, k75)
\fL:fog \fR[\fIvalue\fR]	Change the fog level (or ":fog reset")
\fL:order \fR[\fIvalue\fR]	Fitting function order
\fL:quit \fR	Terminate HDFIT without updating database
\fL:/mark \fRstring	T{
Mark type (point, box, plus, cross, diamond, hline, vline, hebar, vebar, circle)
T}

.T&
l s s.
T{
Additional commands are available for setting graph formats and manipulating
the graphics.  Use the following commands for help.
T}

.T&
l l s.
\fL:/help\fR	Print help for graph formatting option
\fL:.help\fR	Print cursor mode help

.TE
.PP
The value of fog can be changed interactively if you have
reason to override the value written in the database by \fIspotlist\fR.
You can reset the fog to its original value with the command ":fog reset".
A common problem with defining the HD curve is that some of 
the calibration spot densities fall below fog.  This is caused by either
the low signal to noise at low densities or by making a poor choice of 
where to scan the fog level.  These points are rejected from the fit
when a transformation of the density is being made, as the transform cannot
be evaluated for negative density.  If the fog value or transformation 
type is interactively changed so this problem no longer exists, 
the spot densities are restored in the sample.

The parameters of the final fit are written to a database which then
contains the information
necessary to reinitialize the curfit package for applying the transformation
in \fIhdtoi\fR.  

.NH
Applying the Transform
.PP
.NH 2
HDTOI
.PP
Once the HD curve has been defined, it is applied to a density image
in task \fIhdtoi\fR.
Here the transformation is applied, as described by the fit parameters
stored in the database.  If more than one record of fit parameters is
present, the last one is used.  This means task \fIhdfit\fR can be
repeated until an acceptable solution is found; the last solution will
be used by \fIhdtoi\fR.  On output, a new output image is written; the 
input image is left intact.
.PP
The transformation is accomplished by using a look-up table.  All possible
input values, from the minimum to maximum values found in the image, are
converted to density using the scale value read from the database, and then 
to intensity using the fit parameters determined by \fIhdfit\fR.  The input
value is then the index into the intensity table: 
intensity = look_up_table (input_value).
.PP
A scaling factor can be applied to the final intensities, as typically
they will be < 1.0.  (The maximum log exposure in the NOAO wedge database
is 0.0)  By default, a saturated density pixel will be assigned the "ceiling"
intensity of 30000 and the other pixels are scaled accordingly.  
The user is responsible for choosing a ceiling value
that will avoid having significant digits truncated.
The precision 
of the transform is unaffected by scaling the
final intensities, although caution must be used if the output image
pixel type is an integer.  
.PP
The value of fog to be used is entered by the user, and can be either
a number or a list of file names from which to calculate the fog value.
The fog value is subtracted from the input image before the transformation
takes place.
Again, consider density values below fog.  Two choices are available for
these densities: the calculated intensity can be equal to the constant 
value 0.0 or equal -1.0 times the intensity determined for absolute (density).

.NH
Aligning Related HD curves
.PP
Calibration data sets from several plates can be combined once a shift
particular to each set has been removed.  "Different spot exposures 
define a series of HD curves which are parallel but mutually separated
by arbitrary shifts in log exposure, produced by differing lamp intensities
or exposure times.  Generally, Kodak spectroscopic plates can be
averaged if [1] they come from the same emulsion batch and box, [2]
they receive identical hypersensitizing, [3] they are exposed similarly and
[4] they receive the same development." * 
.FS
* "Averaging Photographic Characteristic Curves", John Kormendy, from
"ESO Workshop on Two Dimensional Photometry", Edited by P. Crane and
K.Kjar, p 69, (1980), an ESO Publication.
.FE
.NH 2
HDSHIFT
.PP
Procedure \fIhdshift\fR calculates and subtracts a zero point shift to 
bring several related HD curves into alignment.  The individual shifts 
are calculated by elimination of the first coefficient (Bevington, eqn 9-3):
.EQ

a0 = y bar - a sub 1 X bar - a sub 2 X bar sup 2 - ~ ...~ - a sub n X bar sup n

.EN
Here, the averages over y and X refer to individual calibration set averages; 
the coefficients a1, ... an were previously calculated using data from all 
calibration sets with task \fIhdfit\fR, and stored in the database.  The
a0 term is calculated individually for each database; this term represents
the zero point shift in log exposure and will be different for each database.

On output, the log exposure values in each database have been 
shifted to the zero point shift of the first database in the list.  The
log exposure records are now aligned and it would be appropriate
to run \fIhdfit\fR on the modified database list.
.NH
Testing the Transformation Algorithm
.PP
A test task is included to see if any numerical errors were introduced
during the density to intensity transformation.  It also evaluates
truncation errors produced when an output image with integer pixels,
rather than reals, is written.
.NH 2
SELFTEST
.PP
An intensity vector is generated from a density vector in two different
ways.  The first method uses the density vector and known coefficients
to compute the intensity.  The second method uses the curfit package
to generate a look up table of intensities as done in task \fIhdtoi\fR.  The
residual of the two vectors is plotted; ideally the difference between
the 'known' and 'calculated' intensity is zero.
.PP
Task \fIselftest\fR also plots intensity as a function of density for
both integer and real output pixels.  The user should investigate the
plot with the cursor zoom and expand capabilities to determine if
truncation errors are significant.
.NH
The Wedgefile Database
.PP
Task \fIdematch\fR reads a database and retrieves log exposure information 
for certain combinations of wedge number, photographic emulsion and filter.  
Those combinations included in the NOAO database are listed in the next 
section, although any calibration data can be included if the values are
known.  To modify the database, it is recommended that
you generate a new file rather than add records to the existing file.  This
way, the modifications will not be lost when a new version of the IRAF
system is released.  

In the database, the information for each wedge makes up a separate record;
each record starts with the word \fBbegin\fR.  Each record has a title field 
and can have multiple emulsion/filter fields.  The number of log exposure
values must be given, followed by the values written 8 per line.  The order 
of the exposure data can be either monotonically increasing or decreasing.  
Here is an example:
.DS
begin	115
	title	MAYALL 4-M PF BEFORE 15APR74 (CHROME) [MP1-MP968]
	IIIAJ/UG2	16
		  0.000 -0.160 -0.419 -0.671 -0.872 -1.153 -1.471 -1.765
		 -2.106 -2.342 -2.614 -2.876 -3.183 -3.555 -3.911 -4.058
	IIAO/UG2	16
		  0.000 -0.160 -0.418 -0.670 -0.871 -1.152 -1.468 -1.761
		 -2.102 -2.338 -2.609 -2.870 -3.176 -3.547 -3.901 -4.047

.DE
.NH 2
Contents of the NOAO Wedgefile
.LP
The following table lists the wedge/emulsion/filter combinations available in
the NOAO wedgefile database.  
.TS
center;
l l s s s
l l l l l.

\fBWedge  24 	CTIO SCHMIDT WESTON TUBE SENSITOMETER.            \fR
 	MONO/MONO					

.T&
l l s s s
l l l l l.
\fBWedge  48 	PALOMAR 48-INCH SCHMIDT STEP WEDGE.               \fR
 	MONO/MONO			

.T&
l l s s s
l l l l l.
\fBWedge  84 	OLD 84-INCH SPOT SENSITOMETER (1967)               \fR
 	MONO/MONO			

.TE
.TS
l l s s s
l l l l l.
\fBWedge 101 	SPOT BOX 4, KEPT IN SCHOENING-S LAB.              \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 115 	MAYALL 4-M PF BEFORE 15APR74 (CHROME) [MP1-MP968] \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4770	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 117 	CTIO 4-METER P.F.                                 \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4770	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 118 	CTIO 4-METER CASSEGRAIN                           \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470	MONO/6900		

.T&
l l s s s
l l l l l.
\fBWedge 119	SPOT BOX 5, KEPT AT MAYALL 4-METER.               \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 120 	SPOT BOX 6, KEPT AT 2.1-METER.                    \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 121 	SPOT BOX 8, KEPT IN SCHOENING'S LAB.              \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470        

.T&
l l s s s
l l l l l.
\fBWedge 122 	SPOT BOX 7, AVAILABLE AT KPNO NIGHT ASST'S OFFICE \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470C			
.TE
.TS
l l s s s
l l l l l.
\fBWedge 123 	MAYALL 4-M P.F. 15APR74 TO 21MAY74 [MP969-MP1051] \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4770	MONO/5200	MONO/5876        
 	MONO/6470        

.T&
l l s s s
l l l l l.
\fBWedge 129 	MAYALL 4-METER P.F. AFTER 21MAY74 [MP1052-->  ]   \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 130 	MAYALL 4-METER CASS CAMERA.                       \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4760	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 138 	TRAVELLING BOX AFTER 06JAN78.                    \fR
 	IIIAJ/UG2	IIAO/UG2	IIIAJ/*5113	IIAO/*5113       
 	IIAO/GG385	IIIAJ/CLEAR	IIIAJ/GG385	IIAD/GG495       
 	127/GG495	098/RG610	127/RG610	IVN/RG695       
 	MONO/4363	MONO/4770	MONO/5200	MONO/5876        
 	MONO/6470			

.T&
l l s s s
l l l l l.
\fBWedge 201 	TEN UCLA SPOTS (H. FORD, 10JAN78)                 \fR
 	MONO/MONO			
.TE