diff options
author | Joseph Hunkeler <jhunkeler@gmail.com> | 2015-07-08 20:46:52 -0400 |
---|---|---|
committer | Joseph Hunkeler <jhunkeler@gmail.com> | 2015-07-08 20:46:52 -0400 |
commit | fa080de7afc95aa1c19a6e6fc0e0708ced2eadc4 (patch) | |
tree | bdda434976bc09c864f2e4fa6f16ba1952b1e555 /sys/imio/doc | |
download | iraf-linux-fa080de7afc95aa1c19a6e6fc0e0708ced2eadc4.tar.gz |
Initial commit
Diffstat (limited to 'sys/imio/doc')
-rw-r--r-- | sys/imio/doc/IMH.hlp | 219 | ||||
-rw-r--r-- | sys/imio/doc/Notes | 177 | ||||
-rw-r--r-- | sys/imio/doc/bench.ms | 73 | ||||
-rw-r--r-- | sys/imio/doc/imfort.doc | 72 | ||||
-rw-r--r-- | sys/imio/doc/imio.2.ms | 331 | ||||
-rw-r--r-- | sys/imio/doc/imio.doc | 232 | ||||
-rw-r--r-- | sys/imio/doc/imio.hlp | 1185 | ||||
-rw-r--r-- | sys/imio/doc/imio.ms | 295 |
8 files changed, 2584 insertions, 0 deletions
diff --git a/sys/imio/doc/IMH.hlp b/sys/imio/doc/IMH.hlp new file mode 100644 index 00000000..0599843b --- /dev/null +++ b/sys/imio/doc/IMH.hlp @@ -0,0 +1,219 @@ +.help imio Mar86 "Image I/O Modifications" +.ce +\fBImage I/O Modifications to Support Multiple Data Formats\fR +.ce +Doug Tody +.ce +March 16, 1986 + +.nh +Introduction + + The primary purpose of this revision of IMIO was to add support for +multiple disk data formats. This was done by defining a new interface +called IMH, which hides the details of how images are stored on disk from +the IMIO code. IMIO is concerned only with image i/o based on an internal +image descriptor. IMIO calls IMH to perform all imagefile management +operations, but accesses the pixel data directly using the FIO interface. +The IMH interface initially supports only the old IRAF and SDAS image formats, +but may be extended to support other formats in the future if necessary. +The IMH interface should be reusable in the future when IMIO is layered +upon DBIO. + +A secondary purpose of this revision was to make several minor enhancements +to IMIO to better support groups of images. The image template syntax +has been extended to provide a special selection syntax for specifying the +subset of the images in a group or set of groups to be operated upon. +Minor changes were made to the filenames of the files used to store IRAF +format images to make identification of the header and pixel files easier. +The image database interface (IDB) has been extended to permit the storage +of one dimensional arrays in the image header. IMIO was modified to use the +static file driver rather than the regular binary file driver to access +pixel data. + +.nh +Image Templates +.nh 2 +Image Template Syntax + + An image template is an expression used to specify the set or group of +images to be operated upon. The current image template syntax is upwards +compatible with the planned DBIO record select/project syntax. The full +syntax is as follows. + + images [,images ...] + +where \fIimages\fR is an expression built up from some combination of the +following constructs: + +.nf + @listfile take strings from a listfile + pattern expand pattern against database + str // str concatenate strings + {select} select elements from a set + [section] append image section +.fi + + +These elements may appear in any order, although not all orderings make +sense. The \fIpattern\fR string may contain any of the standard pattern +matching characters, but the pattern matching meta-characters {} (ignore +case) and [] (character class) must be escaped to avoid interpretation as +the subset selection and image section operators. An \fIimages\fR string +may contain at most one list construct, i.e., listfile reference, pattern, +or selection set. The \fIselect\fR expression is limited to a range list +at present. The range list syntax uses the : range syntax, i.e., +"from[:to[:by]]". Some examples are given below. None of templates shown +in these examples need be quoted if entered in the CL command mode. + + +.nf + pix one image + pix.0013 one image + @pics list of images + pix[*,-*] one image section + @pics[*,-*] list of image sections + pix.* all pix.whatever images in cwd + pix.*//.flat same, but append ".flat" to image name + pix.*//.flat[*,5] same, but append image section too + pix{1,4,9:21} pix.0001, pix.0004, and 9 through 21 + pix{1,4,9:21}[1:10,5] same, but append section to each one +.fi + + +Note that \fIselect\fR expressions are expanded without checking to see if the +named images actually exist. Image names are formed by concatenating the +image number encoded as a four digit string, padding with zeroes at the left +as in the examples. + +.nh 2 +Image Template Procedures + + The image template package (IMT) is an existing package. The calling +sequences are identical to those of the FNT (FIO filename template) package +and have not been changed in this release of the package. Extensive changes +have however been made internally, and the new package has been installed +in IMIO. The old package has been removed from the XTOOLS library. To use +the new version of the package, all one need do is relink. + + +.ks +.nf + list = imtopen (template) + nimages = imtlen (list) + imtrew (list) + nchars|EOF = imtgetim (list, fname, maxch) + imtclose (list) +.fi +.ke + + +An image template is expanded into a list of image names or image sections +with \fBimtopen\fR. The list is not globally sorted, however sublists +generated by pattern matching are sorted before appending the sublist to +the final list. The number of images or image sections in a list is given by +\fBimtlen\fR. Images are read sequentially from the list with \fBimtgetim\fR, +which returns EOF when the end of the list is reached. The list may be +rewound with \fBimtrew\fR. An image template list should be closed with +\fBimtclose\fR to return the buffers used to store the list and its +descriptor. + +.nh +The Image Header Access Interface (IMH) + + The image header access interface (IMH) is a new interface in this release +of IMIO. The purposes of the IMH interface are to hide knowledge of how images +are stored on disk from the rest of the system, and to make it possible to +support multiple image storage formats. IMIO is not aware that there are +multiple image storage formats. When called to open an existing image, +IMH determines the image format and calls the appropriate lower level access +procedure to read the image header. A standard set of IMH callable interface +procedures are required for each supported storage format. + +The IMH package is intended as an internal IMIO package and should not normally +be called by packages other than IMIO. + + +.ks +.nf + im = imh_open (image, acmode, o_im) # open/create header + imh_opix (im, acmode) # open/create pixels + imh_update (im) # update header + imh_close (im) # close image + + y/n = imh_access (image, type) # test if image exists + imh_delete (image) # delete an image + imh_rename (oldname, newname) # rename an image + imh_copy (oldname, newname) # copy an image +.fi +.ke + + +The \fIimh_open\fR procedure will open an existing image, create a new +image, or make a new copy of an existing image (preserving the header +of the old image but not the pixels). A pointer to an IMIO binary image +descriptor is returned; when opening an existing image, the primary +function of \fIimh_open\fR is to map the disk image header, stored in +whatever format, into the IMIO fixed format binary descriptor. + +The \fIimh_opix\fR procedure must be called after the header has been opened +before any pixel i/o can be done to the image. In the case of a new image +the size attributes of the new image are not fixed until \fIimh_opix\fR is +called (giving the high level code time to set the size parameters in the +image descriptor). It is not necessary to call \fIimh_opix\fR if only the +header of an existing image is to be accessed. + +The \fIimh_access\fR procedure is provided to test if the named image +exists (no test is made to determine if the image is also accessible). +An integer code identifying the storage format of the image, e.g., old +IRAF or SDAS, is returned in the \fItype\fR argument. Currently, the type +of an image is indicated by the filename extension of the header file. + +The \fIimh_rename\fR and \fIimh_copy\fR procedures are in principle not +required since the operations can be performed at a high level with the +procedures already provided, but the IMH operators can carry out the +operations more efficiently and without the possibility of information being +lost, since they have knowledge of the physical storage format. + +.nh +Physical Data Formats + + The only two physical (disk) data formats currently supported are the +old IRAF format and the STScI SDAS format. The IMH interface makes it +relatively straightforward for other sites to interface their local format +if necessary, or to support multiple versions of the same format. + +IMH will automatically read images stored in any of the supported formats. +When making a NEW_COPY image, IMH will generate a new image in the same +format as the existing input image. When making a NEW_IMAGE type image, +the value of the environment variable \fBimtype\fR determines the type of +image to be created. The recognized values of \fBimtype\fR are shown below. + +.ks +.nf + not defined old iraf format + imtype = "oiraf" old iraf format + imtype = "sdas" SDAS/SOGS format +.fi +.ke + +The format in which an image is stored is indicated by the filename extension +of the header file. The filename extension is not specified in image names +or templates above the IMH interface, but is visible to the user in a +directory listing. + +.ks +.nf + .imh old iraf format header + .hhh SDAS/SOGS format header +.fi +.ke + +The current IMH interface is implemented in such a way that the semantics +of the two image storage formats are essentially equivalent, i.e., applications +programs should work consistently regardless of the storage format used. +In order to achieve this, the SDAS group format is fully supported only +when reading existing group format images. On output, images stored in +SDAS format are stored in separate files (gcount=1). The IRAF image template +is more flexible than the SDAS group format, simplifies programming, and +provides much the same basic capability. diff --git a/sys/imio/doc/Notes b/sys/imio/doc/Notes new file mode 100644 index 00000000..b7f6675c --- /dev/null +++ b/sys/imio/doc/Notes @@ -0,0 +1,177 @@ +IMIO modifications to support SDAS format imagefiles +---------------------------------------------------------------- + +1. EXISTING DATA FORMATS + + +1.1 IRAF Data Format (pre-DBIO) + +Characteristics: + + o Header: binary data structure + fits card image string buffer + o Pixels: binary pixhdr + pixels (can be block aligned) + o Many datatypes supported + o One image per imagefile + o Pixel file may be stored in different directory than header file + o Header file is protected from deletion + +Disadvantages: + + o Can lose track of pixel file if header is deleted. + o Since each image is stored in a separate pair of files, + directories can be large. + o The storage format is machine dependent. + +Modifications in this release: + + o Add .imh extension to image header file + o Add .pix extension to pixel file, make root name same as that + of the header file + o If IMDIR = "", put pixel file in same directory as the header. + This avoids use a pathname in the header, hence the images + are relocatable, but forces one to work on a scratch device. + + + +1.2 SDAS Data Format + +Characteristics: + + o Images are physically stored in group format + o FITS group header + binary image headers + o Pixfile format: [pixels + group header] * ngroups + nothing is aligned on block boundaries + o Pixel and header files stored in same directory + o Header file extension .hhh, pixel file extension .hhd + +Disadvantages: + + o Images cannot be added to a group; the number of images in a + group must be specified when the group is created. + o The individual images in a group cannot be deleted; only the + entire group can be deleted. + o The format of the image headers for the individual images in + a group is fixed at create time; new parameters cannot be + added to the individual image headers (new parameters can + however be added which apply to the group as a whole). + o The images in a group must all be of the same size and datatype + (this is probably not a serious disadvantage). + o The storage format is machine dependent. + + +2. IMAGEFILE ACCESS + + o Open/create image + o Make new_copy image + o Access pixel segment + o Close image + o Test if image exists (and determine type) + o Delete image + o Rename image + + +2.1 Open Image + + generate header filename + open/create header file + allocate image descriptor + if (existing image) + read image header into descriptor + else + initialize descriptor + return pointer to descriptor + + +2.2 Access Pixel Segment + + All image size parameters must be determined at pixfile creation + time. + + if (new segment) { + fill in descriptor + if (new pixel file) + allocate pixel file + else + open pixel file r/w + update header + } else + open pixel file + + +2.3 Close Image + + if (header has been modified) + update header + close pixfile + close header + + + +3. SPECIFICATIONS + +3.1 Image Header Access + + To minimize changes to existing code, the IMIO internal data structures +will not be modified. The principal change to the structure of the existing +interface is the replacement of the direct calls to the FIO open, close, read, +write, etc. procedures called to access the image header in mass storage by +calls to a new interface IH (Image Header access). The new interface will +hide the disk image format from IMIO. Interface subroutines will be provided +only for the IRAF and SDAS image formats, although in principle the interface +will be extensible to other formats as well. + +Ideally the IH interface should be coded using only relatively low level +VOS and kernel facilities (i.e., no high level FIO, no error handling) so that +it may be used by the IMFORT interface and called from host Fortran programs, +as well as by IMIO. + + + im = ihopen (image, group, acmode) + ihopix (im) + ihclose (im) + + +IHOPEN returns the standard IMIO image descriptor, consisting of the internal +IMIO fields, the binary image header structure IMHDR, and the "user area", +a sequence of FITS card images stored in memory a string buffer, i.e., +each card image is represented as a stripped, newline delimited sequence of +characters, with an EOS following the last card. + +The GROUP argument to IHOPEN permits access to the individual elements of +a group format imagefile. Group format is supported for both imagefile +formats, the principal difference being that the individual images are +stored in separate files in the old IRAF format, and in a single pair of +images in the SDAS format. + +The individual images in a group format imagefile appear as separate, +independent images in IMIO. Several images in a group format imagefile may +be simultaneously open (the files are physically opened only once). +The group header parameters are duplicated for each image in the group. +If the images are stored in the old IRAF format on disk the values of these +parameters may vary from image to image, otherwise (SDAS format) they are +the same for all members of the group. The number of images in a group is +fixed at image creation time. + + +3.2 Image Sections and Templates + + The image section notation recognized by IMMAP may include a group +specifier (set selection expression) as well as a section specifier. +The full syntax is "image{group}[section]", e.g., + + pix{3}[*,5] + +where { is the set selection operator, and [ is the familiar array subscript +or subsection operator. + +The image template notation has also been generalized to support group format +image data. The general form of a template element is + + image{groups}[section] + +where [section] applies to each image in the group. For example, the template + + aa{4},bb{1,3:5},cc{12:15}[*,-*] + +expands as image 4 of group AA, images 1, 3, 4, and 5 of group BB, and images +12 through 15 of group CC flipped in Y. diff --git a/sys/imio/doc/bench.ms b/sys/imio/doc/bench.ms new file mode 100644 index 00000000..05717208 --- /dev/null +++ b/sys/imio/doc/bench.ms @@ -0,0 +1,73 @@ +.OM +.TO +Steve Ridgway +.FR +Doug Tody +.SU +Performance of IRAF Image I/O +.PP +As Caty reported in her memo of 15 November, the timings of the \fIimarith\fR +task were surprisingly poor, i.e., approximately 20 cpu seconds for the +addition of two 200 column by 800 line short integer images, producing a +short integer image as output (a "short" integer is 16 bits). +A look at the code for \fIimarith\fR revealed +that the internal computations were being done in double precision floating, +regardless of the datatype of the images on disk. +I was not aware of this and I appreciate having it brought to my attention. +Fixing \fIimarith\fR took several hours and nearly cut the timings in half. +.PP +When I orginally implemented IMIO I planned to eventually make three major +optimizations (as noted in the program plan and system interface reference +manual): +.RS +.LP \(bu +Optimize the special case of line by line i/o with no automatic type +conversion, image sectioning, boundary extension, etc. +.LP \(bu +Provide direct access into the FIO file buffers when possible to eliminate +the memory to memory copy to and from the IMIO and FIO buffers. +.LP \(bu +Implement a optimal static file driver for UNIX to eliminate the overhead +of copying the data through the system buffer cache, and to permit +overlapped i/o. +.RE +.LP +I have gone ahead and implemented the first two optimizations; this took +a day and the changes were entirely internal to the interface, +requiring no changes to user code and no loss of machine independence. +After these changes were made to IMIO I ran several benchmarks with the +following results. All benchmarks were for images with 16 bit integer pixels. +.TS +center box tab(|); +ci ci ci ci ci ci ci +r n n n nb n n. +operation|open/close|line ovhead|kernel op|total user time|%opt|systime +- +(c=a+b)[200,800]|.38|1.43|1.69|3.50|48%|3.82 +(c=a+b)[800,800]|.38|1.43|6.94|8.75|79%|12.16 +minmax[800,800]|.05|0.59|11.39|12.03|95%|2.66 +.TE +.PP +The columns in the table show the operation tested by the benchmark (two image +additions, each involving three images, and a computation of the minimum and +maximum of a single image), the overhead involved in opening and closing the +images (same operation on a [1,1] image), the total overhead to process the +image lines, the time consumed by the kernel operation, the total user time +for the task, the degree of optimality (ratio of time spent in the kernel +vector operation to the total time for the task), +and the system (UNIX kernel) time required. +.PP +In short, the time required by the original benchmark has decreased from +20 seconds to 3.5 seconds, disregarding the system time. In this worst +case benchmark we still manage to come within 48% of the optimal time of +1.69 seconds for a VAX 11/750. +.PP +The short integer vector addition kernel operator was hand optimized in +assembler for these benchmarks to provide a true measure of the degree +of optimality. The actual unoptimized UNIX vector addition operator is +slightly slower. +The last column, labelled "systime", shows the cpu time consumed +by the UNIX kernel moving the pixels to and from disk; this is the time +that will be eliminated by the static file driver optimization. +Once the static file driver is optimized any further optimizations +will be difficult. diff --git a/sys/imio/doc/imfort.doc b/sys/imio/doc/imfort.doc new file mode 100644 index 00000000..6eef0dc6 --- /dev/null +++ b/sys/imio/doc/imfort.doc @@ -0,0 +1,72 @@ +Jun 19: IRAF images may now be read and written from Fortran programs. + The interface is simple and efficient, but limited in capability. + If a more sophisticated interface is required one may call Fortran + subroutines from SPP main programs (templates are available for + accessing 1 and 2 dimensional images in this fashion), or program + directly in SPP. + + 1. Documentation from the source file + + IMFORT -- Fortran interface to IRAF images. This interface permits a + Fortran program to read or write an existing IRAF image. There is + currently no provision for creating new images or deleting old images + from Fortran. The interface routines are as follows: + + im = imopen (image, mode, ndim, len_axes) + imclos (im) + + imget[sr] (im, buf, x1, x2, linenum) + imput[sr] (im, buf, x1, x2, linenum) + + where + input integer im, x1, x2, linenum + input character image, mode + output integer ndim, len_axes(7) + pixel buf(*) + + imgets,imputs are for short integer (integer*2) pixels + imgetr,imputr are for real pixels + + An image must be opened with IMOPEN before it can be accessed. Legal + access modes are 'r', 'w', and 'rw'. The number of dimensions and + the length of the axes are returned in ndim and len_axes; the latter + should be dimensioned for at least 7 dimensions. All coordinates are + 1-indexed. The variable "im" is an integer. The get and put routines + will perform datatype conversion if necessary. The imget and imput + routines will abort program execution if there is an error. + + + 2. Usage + + Source files (minimal documentation in imfort.c header): + + /iraf/sys/imio/mhdr.c.h + /iraf/sys/imio/imfort.c + + Libraries: + + /usr/lib/libiraf.a -liraf on f77 cmd line + /usr/lib/libvops.a -lvops on f77 cmd line + + e.g., + f77 myprog.f -liraf -lvops -o myprog + + or if called in SPP + + cl> xc myprog.x, lib=iraf + + + 3. Example + + integer im + integer axlen(7), ndim + integer imopen + integer*2 pix(1024) + + im = imopen ('/tmp2/iraf/images/m74', 'r', ndim, axlen) + write (*,*) ndim, axlen + call imgets (im, pix, 10,15, 5) + write (*,*) pix(1), pix(5) + stop + end + diff --git a/sys/imio/doc/imio.2.ms b/sys/imio/doc/imio.2.ms new file mode 100644 index 00000000..0727179f --- /dev/null +++ b/sys/imio/doc/imio.2.ms @@ -0,0 +1,331 @@ +.DA May 7, 1985 +.OM +.TO +distribution +.FR +Doug Tody +.SU +New release of image i/o, etc. +.PP +The new release of IMIO has been installed in the system for a week now and +appears to be bug free. This memo summarizes the changes/additions in this +new version of the interface, and introduces the new "image database" tools +\fIhedit\fR and \fIhselect\fR as well. +.NH +Summary of Changes in the Current Release +.PP +The following changes or additions have been made to the IMIO interface and +the \fIimages\fR package. +.RS +.IP \(bu +IMIO now has the ability to perform (optionally) automatic boundary extension +to satisfy out of bounds pixel references. +.IP \(bu +A preliminary database interface has been added to IMIO. +.IP \(bu +Image headers are now variable length and use only the amount of disk space +required to store the header (excluding byte packing). +.IP \(bu +Two new database utility tasks \fIhedit\fR and \fIhselect\fR have been +added to the \fIimages\fR package. Both use the new library subroutine +\fIevexpr\fR, now installed in the FMTIO package. +.IP \(bu +A new task \fIimshift\fR has been added to the \fIimages\fR package to +perform shifts of two dimensional images using full two dimensional +interpolation. The related tasks \fIgeomap\fR and \fIgeotran\fR are +currently being worked on. Some filtering and convolution tasks should +also be available soon. All of these tasks use the new boundary extension +feature of IMIO. +.RE +.PP +The new release of IMIO is upward compatible with previous versions and should +require no changes to or even recompilation of existing code. The basic image +header structure is unchanged hence existing images and image headers are still +accessible. Copying of old images still on disk with \fIimcopy\fR may however +be desirable to reduce disk consumption (the old headers were wasteful of +storage). +.PP +This release of IMIO introduces some database tools and concepts which +should help us understand the role the DBIO interface and DBMS package will +play in image processing applications in the near future. The current database +interface has serious functional limitations and is inefficient for operations +which operate upon entire databases (e.g., the \fIselect\fR operation), +but does provide a basic and much needed image database capability. +.NH +Planned Future Developments +.PP +This new release of IMIO is expected to remain unchanged until DBIO is +completed, at which time a new version of the interface will be released. +This next release is expected to be upward compatible with the current +interface except in cases where the applications task has detailed knowledge +of the current image header structure. Applications which directly access +the "user area" of the current header, or which use certain header fields +such as IM_HISTORY, will have to be modified as these data structures will +change in the next release. +.PP +Applications which use only \fIimmap\fR, \fIimunmap\fR, IM_PIXTYPE, +IM_NDIM, IM_LEN, and the basic i/o procedures should not have to be changed. +The new interface will provide different facilities to do the same things +but we can probably emulate the old interface to allow plenty of time to +convert the old code. Of course, the new interface will provide new facilities +which we did not formerly have and which we will want to use, and therefore +we will eventually have to modify all existing image tasks. +.PP +Perhaps more seriously, we are not going to be able to maintain the ability +to read the existing binary image files when the DBIO version of IMIO is +released. At that time, all disk resident images will have to be processed +to FITS format and thence into the new DBIO image format. We will keep the +old binary for the FITS writer task around for an indefinite period after +the changeover to make this possible. + +.NH +Modifications to the Current Interface +.NH 2 +Boundary extension +.PP +Automatic boundary extension is useful in applications such as filtering via +convolution, since the convolution kernel will extend beyond the boundary of +the image when near the boundary, and in applications which operate upon +subrasters, for the same reason. When reading from an image with boundary +extension in effect, IMIO will generate artificial values for the out of +bounds pixels using one of the techniques listed below. When writing to an +image with boundary extension in effect, the out of bounds pixels are +discarded. +.PP +By default, an out of bounds pixel reference will result in an error message +from IMIO. Consider an image line of length 5 pixels. The statement +.DS +\fL + buf = imgs1r (im, -1, 7) +\fR +.DE +references out of bounds by 2 pixels on either end of the image line, +referencing a total of 5+2+2=9 pixels. If boundary extension is enabled +and the get section completes successfully then \fIMemr[buf]\fR will reference +the pixel at X = -1, and \fIMemr[buf+2]\fR will reference the first inbounds +pixel. +.PP +When an image is first opened zero pixels of boundary extension are +selected, and any out of bounds references will result in an error. +To enable boundary extension \fIimseti\fR must be called on the open +image to specify the number of pixels of boundary extension permitted +before an out of bounds error occurs. +.DS +\fL + include <imset.h> + call imseti (im, IM_NBNDRYPIX, 2) +.DE +\fR +.LP +If boundary extension is enabled the type of boundary extension desired +should also be set. The possibilities are defined in \fI<imset.h>\fR and +are summarized below. +.DS +\fL + BT_CONSTANT return constant if out of bounds + BT_NEAREST return nearest boundary pixel + BT_REFLECT reflect back into image + BT_WRAP wrap around to other side + BT_PROJECT project about boundary +.ce +\fR +\fBTypes of Boundary Extension\fR +.DE +.LP +The type of boundary extension is set with the imset parameter IM_TYBNDRY. +If the BT_CONSTANT option is selected the constant value should be set with +an \fIimseti\fR or \fIimsetr\fR call to set the parameter IM_BNDRYPIXVAL. +Boundary extension works for images of any dimension up to 7 (the current +IMIO limit). A single IM_NBNDRYPIX value is used for all dimensions. +This value is used only for bounds checking, hence the value should be set +to the maximum out of bounds reference expected for any dimension. +Larger values do not "cost more" than small ones. An actual out of bounds +reference is however more expensive than an inbounds reference. +.NH 2 +Image Database Interface +.PP +The image database interface is the IMIO interface to the database +containing the image headers. In this implementation the image header is +a variable length binary structure. The first, fixed format, part of the +image header contains the standard fields in binary and is fixed in size. +This is followed by the so called "user area", a string buffer containing +a sequence of variable length, newline delimited FITS format keyword=value +header cards. When an image is opened a large user area is allocated to permit +the addition of new parameters without filling up the buffer. When the +header is subsequently updated on disk only as much disk space is used as +is needed to store the actual header. +.PP +The new header format is upwards compatible with the old image header format, +hence old images and programs do not have to be modified to use the latest +release of IMIO. In the future image headers will be maintained under DBIO, +but the routines in the image header database interface described in this +section are not exected to change. +The actual disk format of images will of course change when we switch +over to the DBIO headers. While the physical storage format of header will +change completely under DBIO, the logical schema will change very little, +i.e., our mental picture of an image header will be much as it is now. +The main difference will be the consolidation of many images into a few files, +and real support in the image header for bad pixels, history, and coordinate +transformations. In addition a number of restrictions on the "user fields" +will be lifted, the remaining distinctions between the standard and user +fields will disappear, and database operations will be much more efficient +than they are now. +.NH 3 +Library Procedures +.PP +The IMIO library procedures comprising the current image database interface +are summarized in the table below. +.DS +\fL + value = imget[bcsilrd_] (im, field) + imgstr (im, field, outstr, maxch) + imput[bcsilrd_] (im, field, value) + impstr (im, field, value) + imadd[bcsilrd_] (im, field, def_value) + imastr (im, field, def_value) + imaddf (im, field, datatype) + imdelf (im, field) + y/n = imaccf (im, field) + + list = imofnl[su] (im, template) + nchars/EOF = imgnfn (list, fieldname, maxch) + imcfnl (list) + +where + pointer im, list + char[] field, outstr, datatype, template, fieldname +\fR +.ce +\fBImage Database Interface Procedures\fR +.DE +.PP +New parameters will typically be added to the image header with either +one of the typed \fIimadd\fR procedures or with the lower level \fIimaddf\fR +procedure. +The former procedures permit the parameter to be created and the value +initialized all in one call, while the latter only creates the parameter. +In addition, the typed \fIimadd\fR procedures may be used to update the values +of existing parameters, i.e., it is not considered an error if the parameter +already exists. The principal limitation of the typed procedures is that +they may only be used to add or set parameters of a standard datatype. +The \fIimaddf\fR procedure will permit creation of parameters with more +descriptive datatypes (abstract datatypes or domains) when the interface is +recut upon DBIO. There is no support in the current interface for domains. +.PP +The value of any parameter may be fetched with one of the \fIimget\fR functions. +\fIBe careful not to confuse \fBimgets\fI with \fBimgstr\fI +(or \fBimputs\fI with \fBimpstr\fI) when +fetching or storing the string value of a field\fR. Full automatic type +conversion is provided. Any field may be read or written as a string, +and the usual type conversions are permitted for the numeric datatypes. +.PP +The \fIimaccf\fR function may be used (like the FIO \fIaccess\fR procedure) +to determine whether a field exists. Fields are deleted with \fIimdelf\fR; +it is an error to attempt to delete a nonexistent field. +.PP +The field name list procedures \fIimofnl[su]\fR, \fIimgnfn\fR, +and \fIimcfnl\fR procedures are similar to the familiar file template +facilities, except that the @file notation is not supported. The template +is expanded upon an image header rather than a directory. Unsorted lists +are the most useful for image header fields. If sorting is enabled each +comma delimited pattern in the template is sorted separately, rather than +globally sorting the entire template after expansion. Minimum match is +permitted when expanding the template, another difference from file +templates. Only actual, full length field names are placed in the output +list. +.NH 3 +Standard Fields +.PP +The database interface may be used to access any field of the image header, +including the following standard fields. Note that the nomenclature has +been changed slightly to make it more consistent with FITS. Additional +standard fields will be defined in the future. These names and their +usage may change in the next release of IMIO. +.DS +\fI + keyword type description +\fL + i_ctime l time of image creation + i_history s history string buffer + i_limtime l time when limits (minmax) were last updated + i_maxpixval r maximum pixel value + i_minpixval r minimum pixel value + i_mtime l time of last modify + i_naxis i number of axes (dimensionality) + i_naxis[1-7] l length of an axis ("i_naxis1", etc.) + i_pixfile s pixel storage file + i_pixtype i pixel datatype (SPP integer code) + i_title s title string +\fR +.ce +\fBStandard Header Fields\fR +.DE +.PP +The names of the standard fields share an "i_" prefix to reduce the possibility +of collisions with user field names, to identify the standard fields in +sorted listings, to allow use of pattern matching to discriminate between the +standard fields and user fields, and so on. For the convenience of the user, +the "i_" prefix may be omitted provided the resultant name does not match the +name of a user parameter. It is however recommended that the full name be +used in all applications software. +.NH 3 +Restrictions +.PP +The use of FITS format as the internal format for storing fields in this +version of the interface places restrictions on the size of field names and +of the string value of string valued parameters. Field names are currently +limited to eight characters or less and case is ignored (since FITS requires +upper case). The eight character limit does not apply to the standard fields. +String values are limited to at most 68 characters. If put string is passed +a longer string it will be silently truncated. Trailing whitespace and +newlines are chopped when a string value is read. + +.NH +Database Utility Tasks +.PP +Two image database utility tasks have been implemented, \fIhedit\fR and +\fIhselect\fR. \fIHedit\fR is the so called header editor, used to modify, +add, or delete selected fields of selected images. The \fIhselect\fR task +is used to select images that satisfy a selection criteria given as a boolean +expression, printing a subset of the fields of these images on the standard +output in list form. Manual pages are attached. +.PP +Both of these tasks gain most of their power from use of the \fIevexpr\fR +utility procedure, now available in FMTIO. The \fIevexpr\fR procedure takes +as input an algebraic expression (character string), parses and evaluates +the expression, and returns as output the value of the expression. +.DS +\fL + include <evexpr.h> + pointer evexpr() + + o = evexpr (expr, getop, ufcn) + +where + o Is a pointer to an operand structure + expr Is a character string + getop Is either NULL or the \fIlocpr\fL address + of a user supplied procedure called during + expression evaluation to get the value of + an external operand. + ufcn Is either NULL or the \fIlocpr\fL address + of a user supplied procedure called during + expression evaluation to satisfy a call to + an external function. +\fR +.DE +The operand structure is defined in \fB<evexpr.h>\fR. The best documentation +currently available for the operators and functions provided by \fIevexpr\fR +will be found in the manual page(s) for \fIhedit\fR. Additional documentation +will be found with the sources. The expression evaluation procedure is +probably the single largest procedure in the system (in terms of kilobytes +added to an executable) and should not be used unless it is needed, but it can +greatly increase the power of a task in the right application. +.CT +IRAF +Larry Goad +George Jacoby +Richard Wolff +Steve Ridgway (fyi) +Jeanette Barnes (fyi) +Ed Anderson (fyi) diff --git a/sys/imio/doc/imio.doc b/sys/imio/doc/imio.doc new file mode 100644 index 00000000..daa72a52 --- /dev/null +++ b/sys/imio/doc/imio.doc @@ -0,0 +1,232 @@ + IRAF IMIO OVERVIEW + 7 May 1986 + + + +1. DATA MANAGEMENT ROUTINES + +include <imhdr.h> + + im = immap (image, mode, oim) + imunmap (im) + + imdelete (image) + imrename (oldname, newname) + +where + struct imhdr *im, *oim; image header/descriptor + char image[]; image name or image section + int mode; ro, wo, rw, new_image, new_copy + +important header parameters: + + im->im_naxis number of axes + im->im_axlen[i] axis lengths + im->im_pixtype pixel datatype + im->im_datamin min pixel value + im->im_datamax max pixel value + im->im_title title string + + +Existing images are normally opened either read only or read write. +New images are opened either new_image or new_copy. In the latter case, +the third argument is a pointer to the image descriptor of an existing +image, with the new image inheriting the non-data attributes of the +existing image header. This latter feature is important for data +independence. + +The IMIO interface supports images of up to naxis=7. In a sense, all images +are multidimensional, with the higher, unused axis lengths set to 1. +An N dimensional image may therefore be accessed by a program coded to +operate upon an M dimensional image. + +The image section facility greatly inceases the flexibility of the IMIO +interface. Image sections are specified as part of the image name input +to IMOPEN, and are not visible to the applications program, which sees +a somewhat smaller image, or an image of lesser dimensionality. Some examples +are shown below. + + + section refers to + + pix[] whole image + pix[i,j] the pixel value (scalar) at [i,j] + pix[*,*] whole image, two dimensions + pix[*,-*] flip y-axis + pix[*,*,b] band B of three dimensional image + pix[*,*:s] subsample in y by S + pix[*,l] line L of image + pix[c,*] column C of image + pix[i1:i2,j1:j2] subraster of image + pix[i1:i2:sx,j1:j2:sy] subraster with subsampling + + +Sections are implemented by defining a linear transformation upon the +pixel coordinates input when image i/o takes place. All image data +transfers can be represented as subrasters defined by corner points +pointed to by the vectors VS and VE, each of length NAXIS. If an image +section is specified, the IMIO i/o routines merely transform these +vectors into PVS and PVE, the physical coordinates of the referenced +subraster, before doing any i/o. + + +2. IMIO OPTIONS + + IMIO options may be set and queried with the IMSET and IMSTAT procedures, +shown below. + + imseti (im, option, int_value) + imsetr (im, option, real_value) + + int = imstati (im, option) + real = imstatr (im, option) + +The options currently supported are shown below (from <imset.h>). + + +# IMSET.H -- Definitions for IMIO user settable options + +define IM_ADVICE 1 # RANDOM or SEQUENTIAL +define IM_NBUFS 2 # number of input buffers +define IM_COMPRESS 3 # align lines on device blocks? +define IM_NBNDRYPIX 4 # width of boundary region +define IM_TYBNDRY 5 # type of boundary extension +define IM_FLAGBADPIX 6 # set bad pix to INDEF +define IM_PIXFD 7 # pixfile fd (special devices) +define IM_WHEADER 8 # update image header at unmap time +define IM_BNDRYPIXVAL 9 # for option IM_CONSTANT + + +# Types of Boundary Extension + +define BT_CONSTANT 1 # return constant if out of bounds +define BT_NEAREST 2 # return nearest boundary pixel +define BT_REFLECT 3 # reflect back into image +define BT_WRAP 4 # wrap around to other side +define BT_PROJECT 5 # project about boundary + + +The most useful options are the multiple input buffers and boundary extension, +used to implement filtering operators. + + +3. IMIO I/O ROUTINES + + There are two basic approaches used in image interfaces: images may be +mapped into virtual memory, or accessed via conventional file i/o. IMIO +provides both but emphasizes the latter, since it is more portable, more +efficient for sequential image operations, and because it provides data +independence. All buffering is handled internally by the interface to +simplify the interface (externally), and to provide the control necessary +for sophisticated features and optimizations. + +IMIO currently provides three classes of i/o routines: [1] get/put line +random, [2] get/put line sequential, and [3] get/put subraster random. + + +3.1 Get/Put Line Random (for images of known dimension) + + ptr = im[gp]l[123][usilrdx] (im, line [, band [, ...]]) +e.g., + (short *) = imgl1s (im) # get line from 1d short image + (short *) = imgl2s (im, lineno) # get line from 2d short image + (real *) = imgl2r (im, lineno) # get line from 2d real image + (short *) = impl2s (im, lineno) # put line to 2d short image + (real *) = imgl3r (im, line, band) # get from 3d image + + +3.2 Get/Put Line Sequential (for images of any dimension) + + int = im[gp]nl[usilrdx] (im, ptr, v) +e.g., + (EOF?) = imgnlr (im, ptr, v) # get next line, real + (EOF?) = impnls (im, ptr, v) # put next line, short + +Here, STAT is either EOF or not EOF, with EOF being returned when the last +line of the image has been read. The output argument PTR is set to point +to the buffer containing the input pixels, or the buffer into which the +output pixels are to be written. The vector V, of length IM_MAXDIM, points +to the next line of the image to be read. It is set initially to [1,1,1,...] +by the user (assuming the entire image is to be accessed), and is automatically +updated by IMIO in each call. + + +3.3 Get/Put Subraster Random + + ptr = im[gp]s[123][usilrdx] +e.g., + (short *) = imgs2s (im, x1,x2, y1,y2) # get 2d subraster, short + (real *) = imps1r (im, x1,x2) # put 1d subraster, real + +These routines (and indeed all the i/o routines) can be used for either +sequential or random accesses. The subraster routines must be used to +reference outside the boundary in X. + + +3.4 Other I/O Routines + + Other, lower level routines are provided for unusual applications for which +the above routines are not suited. + + ptr = im[pg]gs[usilrdx] (im, vs, ve, ndim) + +The above puts/gets a general section of any dimension. The vectors VS and VE +define the starting and ending corners of the subraster to be accessed. +An IMFLUSH routine is provided for flushing the output buffer (remember the +delayed write). + + +In all cases, no buffers are allocated until i/o takes place, allowing IMSET +calls to set options after the image has been opened. In the case of a new +image (or new copy image), the pixel file is not allocated until i/o takes +place, giving the user time to set the number of axes, size of each axis, +pixel type, etc. after the image has been opened. + + +4. STORAGE FORMATS + + IMIO stores images on disk in line storage mode, like a multidimensional +Fortran array. Image lines are normally padded out to an integral number +of disk blocks to increase i/o efficiency. We store the header information +separately from the pixels, since the header is variable length. The pixel +storage file is preallocated and fixed in size. We call this a "static file". +A special FIO driver is provided for static files to provide optimal i/o. +Since the file is not dynamically extended at run time and the physical +blocks allocated for the file do not move about, it is possible to bypass +the host files system to directly access the data with a low level interface. + + +5. EXAMPLE (SPP) + + +# IMCOPY -- Copy an image. The header information is preserved. The output +# image has the same size, dimensionality, and pixel type as the input image. +# An image section may however be used to copy a subsection of the input image. + +procedure imcopy (in_image, out_image) + +char in_image[ARB] +char out_image[ARB] + +int npix +long v1[IM_MAXDIM], v2[IM_MAXDIM] +pointer in, out, l1, l2 +pointer immap(), imgnlr(), impnlr() + +begin + # Open/create the images. + in = immap (in_image, READ_ONLY, 0) + out = immap (out_image, NEW_COPY, im_a) + + # Initialize position vectors to line 1, column 1, band 1, ... + call amovkl (long(1), v1, IM_MAXDIM) + call amovkl (long(1), v2, IM_MAXDIM) + npix = IM_LEN(in,1) + + # Copy the image. + while (imgnlr (in, l1, v1) != EOF && impnlr (out, l2, v2) != EOF) + call amovr (Memr[l1], Memr[l2], npix) + + call imunmap (in) + call imunmap (out) +end diff --git a/sys/imio/doc/imio.hlp b/sys/imio/doc/imio.hlp new file mode 100644 index 00000000..8eb96159 --- /dev/null +++ b/sys/imio/doc/imio.hlp @@ -0,0 +1,1185 @@ +.help imio May83 "Image I/O Routines" +.sh +The Image Header + + The major difference between the prototype IMIO interface, and the final +interface, concerns the way in which the image header is implemented and +accessed. In the prototype version, we will simply read the entire header +into core and access it as an ordinary (dynamically allocated) structure. + + +.nf + ptr = immap (fname, mode, hdrsize/hdrptr) + imunmap (hdrptr) +.fi + + +The final resolution of how image headers are implemented depends on how +we decide to implement virtual structures in the spp language. The immap +calls, and the techniques used to access the fields of the image header, +can be expected to change. + +.sh +Pixel I/O + + The calling sequences for the i/o routines, described below, hopefully will +not have to be changed. We will eventually add GETPIX and PUTPIX statements +to the subset preprocessor language, to automatically generate the appropriate +low level calls. + +A generic, polymorphic GETPIX or PUTPIX statement is translated into a +reference to a low level Fortran function. The transformation is governed +by the following subprogram name generating function: + + +.rj (108 total) +GETPIX, PUTPIX --> im[gp][pls][123][silrdx] + + +.ks +.nf +For example (get, type real): + + ptr = imgp1r (hdrptr, x, npix) # get pixels + ptr = imgp2r (hdrptr, x, y, npix) + ptr = imgp3r (hdrptr, x, y, z, npix) + + ptr = imgl1r (hdrptr) # get line + ptr = imgl2r (hdrptr, y) + ptr = imgl3r (hdrptr, y, z) + + ptr = imgs1r (hdrptr, x1, x2) # get section + ptr = imgs2r (hdrptr, x1, x2, y1, y2) + ptr = imgs3r (hdrptr, x1, x2, y1, y2, z1, z2) +.fi +.ke + + +The IM?P?? procedures access a list of pixels, the coordinates of which +are given by the X, Y, Z, etc. arrays given as arguments. Note that random +access of individual pixels is provided as a special case (npix=1). + +The IM?L?? routines access the lines of an image, and the IM?S?? routines +operate on general, but connected, subsections of images. + + +.sh +Restrictions Imposed by the Initial Prototype: + + IMMAP, IMMAPNC, IMUNMAP will be implemented for image headers that are +simple binary structures (not self describing), subject to the restriction +that a file may contain only a single header. An arbitrary selection of user +defined fields will follow the standard header. The entire header will +be read into core and accessed as a simple incore structure. + +The pixels, and other variable size image substructures, will be stored +in separate files, as in the general plan. All of the standard data types +will be implemented in the disk space. The initial implementation will +support only type REAL pixels in program space. + +The following i/o routines will be implemented in the first release of +the prototype: + +.rj (12 total) + im[gp][sc][123][r] + +In words, we will be able to read and write lines and sections, with the +applications program manipulating type REAL pixels internally. The full +range of data types will be permitted in the image file as stored on disk. +Up to three dimensional images are permitted. + +.sh +IMSET Options + + The prototype need not provide multiple buffering and boundary extension +initially. + +.sh +Implementation + + Little effort should be made to make the prototype optimal. All +buffering will be locally allocated, and data will be copied to and from +the FIO buffers (the FIO buffers will not be directly accessed). Special +cases will not be optimized. The most general entry points are IMGGSR +and IMPGSR (get/put general section). Initially, all of the other entry +points can be defined in terms of these. + + +.ks +.nf +Structure of the input procedures (type REAL): + + imgl1r + imgl2r + imgl3r + imgs1r + imgs2r + imgs3r + imggsr + imggsc + imgibf + imopsf + calloc + imcssz + realloc + malloc + mfree + imsslv + imrdpx + imsoob + imnote + seek + read + imflip + imupkr + + (datatype dependent) | (datatype independent) +.fi +.ke + + + +The output procedures are structured somewhat differently, because the +transfer of a section occurs sometime after a "put section" returns, +rather than immediately as in the input procedures. Since the output +is buffered for a delayed write, we must have an IMFLUSH entry point, and +IMUNMAP must flush the output buffer before unmapping an image. + + +.ks +.nf +Structure of the output procedures (type REAL): + + impl1r + impl2r + impl3r + imps1r imunmap + imps2r | + imps3r | + impgsr | + imflush + imflsr + imflsh + imflip + imwrpx + imsoob + imnote + imwrite + fstatus + seek + write + impakr + + imgobf + imopsf + calloc + imcssz + realloc + malloc + mfree + + (datatype dependent) | (datatype independent) +.fi +.ke + + +.sh +Semicode for the Basic I/O Routines + + The IMGGS? and IMPGS? procedures get and put general N-dimensional +image sections of a specific datatype. There is no intrinsic limit on +the maximum number of dimensions, and the full range (8) of disk datatypes +are easily supported. The subscript for a particular dimension may run +either forward or backward. The semicode is written generically, allowing +code to be machine generated for all program space datatypes (6). + +We do not address the problems of boundary extension and multiple buffering +here, but these features can be easily added in the future. This version +of IMIO assumes that pixels are stored on disk in line storage mode, with +no interlacing of bands. + + +IMGGS? gets a general section, and converts it to the datatype indicated +by the type suffix. + + +.ks +.nf +pointer procedure imggs$t (imdes, vs, ve) + +imdes pointer to image descriptor structure +vs,ve coordinates of starting and ending points + +begin + bp = imggsc (imdes, vs, ve, TY_PIXEL, totpix) + if (imdes.pixtype != TY_PIXEL) + call imupk$t (*bp, *bp, totpix, imdes.pixtype) + return (coerce (bp, TY_CHAR, TY_PIXEL)) +end +.fi +.ke + + + +IMGGSC gets a general section from an imagefile into a buffer. Upon +exit, the buffer contains the requested section, with the pixels still +in the same datatype they were in the imagefile. The buffer is made +large enough to accommodate the pixels in either datatype. + + +.ks +.nf +pointer procedure imggsc (imdes, vs, ve, dtype, totpix) + +imdes pointer to image descriptor structure +vs,ve coordinates of starting and ending points +dtype datatype of pixels required by calling program +bp pointer to CHAR buffer to hold pixels + +begin + # Get an (input) buffer to put the pixels into. Prepare the + # section descriptor vectors V, VINC. + + bp = imgibf (imdes, vs, ve, dtype) + call imsslv (imdes, vs, ve, v, vinc, npix) + + # Extract the pixels. IMRPIX reads a contiguous array of + # pixels into the buffer at the specified offset, incrementing + # the offset when done. The pixels are type converted if + # necessary. + + offset = 0 + + repeat { + call imrdpx (imdes, *(bp+offset), v, npix) + if (vinc[1] < 0) + call imflip (*(bp+offset), npix, sizeof(imdes.pixtype)) + offset = offset + npix + + for (d=2; d <= imdes.ndim; d=d+1) { + v[d] += vinc[d] + if ((v[d] - ve[d] == vinc[d]) && d < imdes.ndim) + v[d] = vs[d] + else { + d = 0 + break + } + } + } until (d >= imdes.ndim) + + totpix = offset + return (bp) +end +.fi +.ke + + + + + +Prepare the section descriptor vectors V and VINC. V is a vector specifying +the coordinates at which the next i/o transfer will take place. VINC is +a vector specifying the loop step size. + + +.ks +.nf +procedure imsslv (imdes, vs, ve, v, vinc, npix) + +begin + # Determine the direction in which each dimension is to be + # traversed. + + do i = 1, imdes.ndim + if (vs[i] <= ve[i]) + vinc[i] = 1 + else + vinc[i] = -1 + + # Initialize the extraction vector (passed to IMRDS? to read a + # contiguous array of pixels). Compute length of a line. + + do i = 1, imdes.ndim + v[i] = vs[i] + + if (vs[1] > ve[1]) { + v[1] = ve[1] + npix = vs[1] - ve[1] + 1 + } else + npix = ve[1] - vs[1] + 1 +end +.fi +.ke + + + + + +The put-section procedure must write the contents of the output buffer +to the image, using the section parameters saved during the previous call. +The new section parameters are then saved, and the buffer pointer is +returned to the calling program. The calling program subsequently fills +the buffer, and the sequence repeats. + + +.ks +.nf +pointer procedure impgs$t (imdes, vs, ve) + +imdes pointer to image descriptor structure +vs,ve coordinates of starting and ending points + +begin + # Flush the output buffer, if appropriate. IMFLUSH calls + # one of the IMFLS? routines, which write out the section. + + call imflush (imhdr) + + # Get an (output) buffer to put the pixels into. Save the + # section parameters in the image descriptor. Save the epa + # of the typed flush procedure in the image descriptor. + + bp = imgobf (imdes, vs, ve, TY_PIXEL) + imdes.flush_epa = loc (imfls$t) + + return (bp) +end +.fi +.ke + + + +Flush the output buffer, if a put procedure has been called, and the +buffer has not yet been flushed. The output buffer is flushed automatically +whenever a put procedure is called, when an image is unmapped, or when +the applications program calls IMFLUSH. + + +.ks +.nf +procedure imfls$t (imdes) + +begin + # Ignore the flush request if the output buffer has already been + # flushed. + + if (imdes.flush == YES) { + bdes = imdes.obdes + bp = bdes.bufptr + + # Convert datatype of pixels, if necessary, and flush buffer. + if (imdes.pixtype != TY_PIXEL) + call impak$t (*bp, *bp, bdes.npix, imdes.pixtype) + call imflsh (imdes) + + imdes.flush = NO + } +end +.fi +.ke + + +.ks +.nf +procedure imflsh (imdes) + +begin + # Determine the direction in which each dimension is to be + # traversed. + + bdes = imdes.obdes + call imsslv (imdes, bdes.vs, bdes.ve, v, vinc, npix) + + # Write out the pixels. IMWRPX writes a contiguous array of + # pixels at the specified offset. + + offset = 0 + + repeat { + if (vinc[1] < 0) + call imflip (*(bp+offset), npix, sizeof(imdes.pixtype)) + call imwrpx (imdes, *(bp+offset), v, npix) + offset = offset + npix + + for (d=2; d <= imdes.ndim; d=d+1) { + v[d] += vinc[d] + if ((v[d] - ve[d] == vinc[d]) && d < imdes.ndim) + v[d] = vs[d] + else { + d = 0 + break + } + } + } until (d >= imdes.ndim) +end +.fi +.ke + + + + +Read a contiguous array of NPIX pixels, starting at the point defined by +the vector V, into the callers buffer. + + +.ks +.nf +procedure imrdpx (imdes, buf, v, npix) + +begin + # Check that the access does not reference out of bounds. + + if (imsoob (imdes, v, npix)) + call imerr (imname, subscript_out_of_range) + + # Seek to the point V in the pixel storage file. Compute size + # of transfer. + + call seek (imdes.pfd, imnote (imdes, v)) + nchars = npix * sizeof (imdes.pixtype) + + # Read in the data. + if (read (imdes.pfd, buf, nchars, junk) != nchars) + call imerr (imname, missing_pixels) +end +.fi +.ke + + + +Write a contiguous array of NPIX pixels, starting at the point defined by +the vector V, into the pixel storage file. + + +.ks +.nf +procedure imwrpx (imdes, buf, v, npix) + +begin + # Check that the access does not reference out of bounds. + + if (imsoob (imdes, v, npix)) + call imerr (imname, subscript_out_of_range) + + # Seek to the point V in the pixel storage file. Note that + # when writing to a new image, the next transfer may occur + # at a point beyond the current end of file. If so, write + # out zeros until the desired offset (which is in bounds) + # is reached. + + file_offset = imnote (imdes, v) + if (file_offset > imdes.file_size) + [write zeros until the desired offset is reached] + else + call seek (imdes.pfd, file_offset) + + # Compute size of transfer. If transferring an entire line, + # increase size of transfer to the physical line length, + # to avoid having to enblock the data. NOTE: buffer must + # be large enough to guarantee no memory violation. + + if (v[1] == 1 && npix == imdes.len[1]) + nchars = imdes.physlen[1] * sizeof (imdes.pixtype) + else + nchars = npix * sizeof (imdes.pixtype) + + call write (imdes.pfd, buf, nchars) + imdes.file_size = max (imdes.file_size, file_offset+nchars) +end +.fi +.ke + + + + +IMNOTE computes the physical offset of a particular pixel in the +pixel storage file. If the disk datatype is UBYTE, this is the offset +of the char containing the subscripted byte. + + +.ks +.nf +long procedure imnote (imdes, v) + +begin + pixel_offset = v[1] + for (i=2; i <= imdes.ndim; i=i+1) + pixel_offset += (v[i]-1) * imdes.physlen[i] + + char_offset0 = (pixel_offset-1) * sizeof (imdes.pixtype) + return (imdes.pixoff + char_offset0) +end +.fi +.ke + + + +Convert a vector of any datatype to type PIXEL ($t). The input and +output vectors may be the same, without loss of data. The input and +output datatypes may be the same, in which case no conversion is +performed. + + +.ks +.nf +procedure imupk$t (a, b, npix, dtype) + +begin + switch (dtype) { + case TY_USHORT: + call achtu$t (a, b, npix) + case TY_SHORT: + call achts$t (a, b, npix) + case TY_INT: + call achti$t (a, b, npix) + case TY_LONG: + call achtl$t (a, b, npix) + case TY_REAL: + call achtr$t (a, b, npix) + case TY_DOUBLE: + call achtd$t (a, b, npix) + case TY_COMPLEX: + call achtx$t (a, b, npix) + default: + call syserr (unknown_datatype_in_imagefile) + } +end +.fi +.ke + + + +Convert a vector of type PIXEL ($t) to any datatype. The input and +output vectors may be the same, without loss of data. The input and +output datatypes may be the same, in which case no conversion is +performed. + + +.ks +.nf +procedure impak$t (a, b, npix, dtype) + +begin + switch (dtype) { + case TY_USHORT: + call acht$tu (a, b, npix) + case TY_SHORT: + call acht$ts (a, b, npix) + case TY_INT: + call acht$ti (a, b, npix) + case TY_LONG: + call acht$tl (a, b, npix) + case TY_REAL: + call acht$tr (a, b, npix) + case TY_DOUBLE: + call acht$td (a, b, npix) + case TY_COMPLEX: + call acht$tx (a, b, npix) + default: + call syserr (unknown_datatype_in_imagefile) + } +end +.fi +.ke + + +.sh +Data Structure Management + + When an image is mapped, buffer space is allocated for a copy of +the image header, and for the image descriptor (used by IMIO while an +image is mapped). When the first i/o transfer is done on an image, +either an input or an output data buffer will be created. The size of +this buffer is governed by the size of the transfer, and by the datatypes +of the pixels on disk and in program space. + +If a new image is being written, the pixel storage file is created at +the time of the first PUTPIX operation. The physical characteristics +of the new image, defined by the image header of the new image, are +unalterable once the first i/o operation has occurred. Accordingly, +the number of dimensions, length of the dimensions, datatype of the +pixels on disk, and so on must be set (in the image header structure) +before writing to the new image. + +The only exception to this rule may be the addition of new lines to a +two dimensional image stored in line storage mode, or the addition of +new bands to a multiband image stored in band sequential (noninterlaced) +mode. It is not always possible to modify the dimensions or size of +an existing image. + +It is possible to preallocate space for an image (using FALOC). This +may result in a more nearly contiguous file, and may make writing a +new image slightly more efficient, since it will not be necessary +to write blocks of zeros in IMPGS?. Preallocation will occur +automatically in systems where it is desirable. + +.sh +Pixel Buffer Management + + There may be any number of input buffers per image, but only a single +output buffer. By default there is only a single input buffer. The input +and output buffers are distinct: the same buffer is never used for both +input and output (unlike FIO). + +The size of a buffer may range from one pixel, to the entire image (or +larger if boundary extension is in use). If multiple buffers are in use, +all buffers do not have to be the same size. The size of a buffer may +vary from one GETPIX or PUTPIX call to the next. + +If multiple input buffers are in use, buffers are allocated in a strictly +round robin fashion, one per GETPIX call. Several buffers may contain +data from the same part of the image. Once the desired number of buffers +have been filled, a buffer "goes away" with each subsequent GETPIX call. + + +IMGIBF gets an input buffer. When called to get a line or section, +the vectors VS and VE specify the subsection to be extracted. +This information is saved in the buffer descriptor, along with the +datatype of the pixels and the dimension of the section. + +When IMGIBF is called to get a list of pixels, VS and VE would have to be +replaced by a set of NPIX such vectors, to fully specify the section. +It is impractical to save this much information in the buffer descriptor, +so when creating a buffer to contain a list of pixels, VS and VE are faked +to indicate a one dimensional section of the appropriate size. + + + + +.ks +.nf +pointer procedure imgibf (imdes, vs, ve, dtype) + +imdes image descriptor +vs,ve define the number of pixels to be buffered +dtype the datatype of the pixels in the program + +begin + # If first input transfer, allocate and initialize array of + # input buffer descriptors. + + if (imdes.ibdes == NULL) { + call imopsf (imdes) + call calloc (imdes.ibdes, LEN_BDES * imdes.nbufs, TY_STRUCT) + } + + # Compute pointer to the next input buffer descriptor. + # Increment NGET, the count of the number of GETPIX calls. + + bdes = &imdes.ibdes [mod (imdes.nget, imdes.nbuf) + 1] + imdes.nget += 1 + + # Compute the size of the buffer needed. Check buffer + # descriptor to see if the old buffer is the right size. + # If so, use it, otherwise make a new one. + + nchars = imcssz (imdes, vs, ve, dtype) + + if (nchars < bdes.bufsize) + call realloc (bdes.bufptr, nchars, TY_CHAR) + else if (nchars > bdes.bufsize) { + call mfree (bdes.bufptr, TY_CHAR) + call malloc (bdes.bufptr, nchars, TY_CHAR) + } + + # Save section coordinates, datatype in buffer descriptor, and + # return buffer pointer to calling program. + + bdes.bufsize = nchars + bdes.dtype = dtype + bdes.npix = totpix + + do i = 1, imdes.ndim { + bdes.vs[i] = vs[i] + bdes.ve[i] = ve[i] + } + + return (coerce (bdes.bufptr, TY_CHAR, dtype) +end +.fi +.ke + + + + +.ks +.nf +pointer procedure imgobf (imdes, vs, ve, dtype) + +imdes image descriptor +vs,ve define the number of pixels to be buffered +dtype the datatype of the pixels in the program + +begin + # If first write, and if new image, create pixel storage file, + # otherwise open pixel storage file. Allocate and initialize + # output buffer descriptor. + + if (imdes.obdes == NULL) { + call imopsf (imdes) + call calloc (imdes.obdes, LEN_BDES, TY_STRUCT) + } + + bdes = imdes.obdes + + # Compute the size of buffer needed. Add a few extra chars + # to guarantee that there won't be a memory violation when + # writing a full physical length line. + + nchars = imcssz (imdes, vs, ve, dtype) + + (imdes.physlen[1] - imdes.len[1]) * sizeof (imdes.pixtype) + + if (nchars < bdes.bufsize) + call realloc (bdes.bufptr, nchars, TY_CHAR) + else if (nchars > bdes.bufsize) { + call mfree (bdes.bufptr, TY_CHAR) + call malloc (bdes.bufptr, nchars, TY_CHAR) + } + + # Save section coordinates, datatype of pixels in buffer + # descriptor, and return buffer pointer to calling program. + + bdes.bufsize = nchars + bdes.dtype = dtype + bdes.npix = totpix + + do i = 1, imdes.ndim { + bdes.vs[i] = vs[i] + bdes.ve[i] = ve[i] + } + + return (coerce (bdes.bufptr, TY_CHAR, dtype) +end +.fi +.ke + + + +Given two vectors describing the starting and ending coordinates +of an image section, compute and return the amount of storage needed +to contain the section. Sufficient storage must be allocated to +hold the largest datatype pixels which will occupy the buffer. + + + +.ks +.nf +long procedure imcssz (imdes, vs, ve, dtype) + +begin + pix_size = max (sizeof(imdes.pixtype), sizeof(dtype)) + npix = 0 + + do i = 1, imdes.ndim + if (vs[i] <= ve[i]) + npix *= ve[i] - vs[i] + 1 + else + npix *= vs[i] - ve[i] + 1 + + return (npix * pix_size) +end +.fi +.ke + + +.sh +Mapping and unmapping Image Structures + + An imagefile must be "mapped" to an image structure before the +structure can be accessed. The map operation associates a file with +a defined structure. + +The IMMAP procedure must allocate a buffer for the image header +structure, and for the image descriptor structure. If an existing +imagefile is being mapped, the header is copied into memory from +the imagefile. If a new image is being mapped, the header structure +is allocated and initialized. + +If an image is being mapped as a "new copy", a new header +structure is created which is a copy of the header of an image which +has already been mapped. The entire image header, including any +application specific fields, is copied. + +After copying an image header for a NEW_COPY image, the header field +containing the name of the pixel storage file is cleared. A "new copy" +image structure does not inherit any pixels. Any similar substructures +which describe the attributes of the pixels (i.e., the blank pixel +list, the histogram) must also be initialized. + +Note that the "image descriptor" buffer allocated below actually +contains the image descriptor, followed by the standard image header +(at offset IMHDR_OFF), followed by any user fields. If an existing +image structure is being mapped, the caller supplies the length of +the user area of the header as the third argument to IMMAP. + +IMMAP returns a pointer to the first field of the standard header +as the function value. The image descriptor is invisible to the +calling program. + + + +.ks +.nf +pointer procedure immap (fname, mode, hdr_arg) + +begin + # Add code here to handle section suffixes in imagefile + # name strings (e.g. "image[*,5]"). + + # Open image header file. + hfd = open (fname, mmap[mode], BINARY_FILE) + + # Allocate buffer for image descriptor/image header. Note + # the dual use of the HDR_ARG argument. + + if (mode == NEW_COPY) + sz_imhdr = hdr_arg.sz_imhdr + else + sz_imhdr = (LEN_IMHDR + int(hdr_arg)) * SZ_STRUCT + + call calloc (imdes, SZ_IMDES + sz_imhdr, TY_STRUCT) + imhdr = imdes + IMHDR_OFF + + [initialize the image descriptor, including the default + image section (optionally set by user with suffix above).] + + # Initialize the mode dependent fields of the image header. + switch (mode) { + case NEW_COPY: + call im_init_newcopy (imdes, hdr_arg) + case NEW_IMAGE: + [initialize the image header] + default: + call seek (hfd, BOFL) + n = read (hfd, Memi[imhdr], sz_imhdr) + if (n < SZ_IMHDR || strne (IM_MAGIC(imhdr), "imhdr")) { + call mfree (imdes) + call imerr (fname, file_not_an_imagefile) + } else if (mode == READ_ONLY) + call close (hfd) + } + + [initialize those fields of the image header which are not + dependent on the mode of access.] + + return (imhdr_pointer) +end +.fi +.ke + + + + +.ks +.nf +procedure imunmap (imhdr) + +begin + imdes = imhdr - IMHDR_OFF + + # Flush the output buffer, if necessary. + call imflush (imhdr) + + # Append the bad pixel list. + if (the bad pixel list has been modified) { + if (file_size < blist_offset) + [write out zeros until the offset of the bad pixel + list is reached] + [append the bad pixel list] + [free buffer space used by bad pixel list] + } + + call close (imdes.pfd) + + # Update the image header, if necessary (count of bad pixels, + # minimum and maximum pixel values, etc.). + + if (imdes.update == YES) { + if (no write permission on image) + call imerr (imname, cannot_update_imhdr) + call imuphdr (imdes) + call close (imdes.hfd) + } + + # Free buffer space. + for (i=1; i <= imdes.nbufs; i=i+1) + call mfree (imdes.ibdes[i].bufptr, TY_CHAR) + call mfree (imdes.obdes.bufptr, TY_CHAR) + call mfree (imdes, TY_STRUCT) +end +.fi +.ke + + +IMFLUSH indirectly references a typed flush procedure, the entry point +address of which was saved in the image descriptor at the time of the +last IMPGS? call. The problem here is that IMFLUSH must work properly +regardless of the data type of the pixels in the output buffer. To +ensure this, and to avoid having to link in the full matrix of 48 type +conversion routines, we call LOC in the put-section procedure to reference +the appropriate typed flush routine. + + + +.ks +.nf +procedure imflush (imhdr) + +begin + if (imdes.flush == YES) + call zcall1 (imdes.flush_epa, imdes) +end +.fi +.ke + + + +The following procedure is called by the IMGOBF and IMGIBF routines +to open the pixel storage file, during the first PUTPIX operation on +a file. + + +.ks +.nf +procedure imopsf (imdes) + +begin + switch (imdes.mode) { + case READ_ONLY, READ_WRITE, WRITE_ONLY, APPEND: + imdes.pfd = open (imdes.pixfile, imdes.mode, BINARY_FILE) + if (read (imdes.pfd, pix_hdr, SZ_PIXHDR) < SZ_IMMAGIC) + call imerr (imname, cannot_read_pixel_storage_file) + else if (strne (pix_hdr.im_magic, "impix")) + call imerr (imname, not_a_pixel_storage_file) + + case NEW_COPY, NEW_FILE, TEMP_FILE: + # Get the block size for device "imdir$", and initialize + # the physical dimensions of the image, and the absolute + # file offsets of the major components of the pixel storage + # file. + + dev_block_size = fdevblk ("imdir$") + [initialize im_physlen, im_pixels, im_hgmoff fields + in image header structure] + + # Open the new pixel storage file (preallocate space if + # enabled on local system). Call FADDLN to tell FIO that + # the pixfile is subordinate to the header file (for delete, + # copy, etc.). Save the physical pathname of the pixfile + # in the image header, in case "imdir$" changes. + + call mktemp ("imdir$im", temp, SZ_FNAME) + call fpathname (temp, imhdr.pixfile, SZ_PATHNAME) + + if (preallocation of imagefiles is enabled) + call falloc (imhdr.pixfile, sz_pixfile) + imdes.pfd = open (imdes.pixfile, NEW_FILE, BINARY_FILE) + call faddln (imdes.imname, imdes.pixfile) + + # Write small header into pixel storage file. Allows + # detection of headerless pixfiles, and reconstruction + # of header if it gets lost. + + [write pix_hdr header structure to pixel storage file] + + default: + call imerr (imname, illegal_access_mode) + } +end +.fi +.ke + + +.sh +Data Structures + + An imagefile consists of two separate files. The first file contains +the image header. In the prototype, there may be only a single header per +header file. The header consists of the standard image header, followed +by an arbitrary number of user defined fields. + +The standard part of the image header has a fixed structure. All the variable +size components of an image are stored in the pixel storage file. The +name of the pixel storage file, and the offsets to the various components +of the image, are stored in the image header. The name of the image header +file is in turn stored in the header area of the pixel storage file, +making it possible to detect headerless images. + +The pixel storage file contains a small header, followed by the pixels +(aligned on a block boundary), optionally followed by a fixed size +histogram, and a variable size bad pixel list. + + +.ks +.nf + Structure of an Imagefile + + --------- --------- + | <---- | + standard ----> header + image | + header PIXELS + | | + user histogram (optional) + fields | + | bad + \|/ pixel (optional) + ---------- list + | + \|/ + --------- +.fi +.ke + + +The image header file, which is small, will reside in the users own +directory. The pixel storage file is generated and manipulated +transparently to the applications program and the user, and resides +in the temporary files system, in the logical directory "imdir$". + +Storing the parts of an image in two separate files does cause problems. +The standard file operators, like DELETE, COPY, RENAME, and so on, +either cannot be used to manipulate imagefiles, or must know about +imagefiles. + +To solve this problem, without requiring FIO to know anything about IMIO +or VSIO data structures, two operators will be added to FIO. The first +will tell FIO that file 'A' has a subordinate file 'B' associated with +it. Any number of subordinate files may be associated with a file. +The information will be maintained as a list of file names in an invisible +text file in the same directory as file 'A'. + +The second operator will delete the link to a subordinate file. The FIO +procedures DELETE and RENAME will check for subordinate files, as will CL +utilities like COPY. + +.sh +The Standard Image Header + + The standard fields of an image header describe the physical +characteristics of the image (required to access the pixels), plus +a few derived or historic attributes, which are commonly associated +with images as used in scientific applications. + + +.ks +.nf +struct imhdr { + char im_magic[5] # contains the string "imhdr" + long im_hdrlen # length of image header + int im_pixtype # datatype of the pixels + int im_ndim # number of dimensions + long im_len[MAXDIM] # length of the dimensions + long im_physlen[MAXDIM] # physical length (as stored) + long im_pixels # offset of the pixels + long im_hgmoff # offset of hgm pixels + long im_blist # offset of bad pixel list + long im_szblist # size of bad pixel list + long im_nbpix # number of bad pixels + long im_cdate # date of image creation + long im_mdate # date of last modify + real im_max # max pixel value + real im_min # min pixel value + struct histogram im_hgm # histogram descriptor + struct coord_tran im_coord # coordinate transformations + char im_pixfile[SZ_PATHNAME] # name of pixel storage file + char im_name[SZ_IMNAME] # image name string + char im_history[SZ_IMHIST] # history comment string +} +.fi +.ke + + + +The histogram structure, if valid, tells where in the pixel storage file +the histogram is stored, and in addition summarizes the principal +attributes of the histogram. All of these quantities are directly +calculable, except for the last three. The modal value is determined +by centering on the (major) peak of the histogram. LCUT and HCUT define +an area, centered on the modal value, which contains a certain fraction +of the total integral. + + +.ks +.nf +struct histogram { + int hgm_valid # YES if histogram is valid + int hgm_len # number of bins in hgm + long hgm_npix # npix used to compute hgm + real hgm_min # min hgm value + real hgm_max # max hgm value + real hgm_integral # integral of hgm + real hgm_mean # mean value + real hgm_variance # variance about mean + real hgm_skewness # skewness of hgm + real hgm_mode # modal value of hgm + real hgm_lcut # low cutoff value + real hgm_hcut # high cutoff value +} +.fi +.ke + + + +The coordinate transformation descriptor is used to map pixel coordinates +to some user defined virtual coordinate system, (useful when displaying the +contents of an image). For lack of a significantly better scheme, we have +simply adopted the descriptor defined by the FITS standard. + + +.ks +.nf +struct coord_tran { + real im_bscale # pixval scale factor + real im_bzero # pixval offset + real im_crval[MAXDIM] # value at pixel + real im_crpix[MAXDIM] # index of pixel + real im_cdelt[MAXDIM] # increment along axis + real im_crota[MAXDIM] # rotation angle + char im_bunit[SZ_BUNIT] # pixval ("brightness") units + char im_ctype[SZ_IMCTYPE,MAXDIM] # coord units +} +.fi +.ke + + + +The image and buffer descriptors are used internally by IMIO while +doing i/o on a mapped image. The image descriptor structure is +allocated immediately before the image header, is transparent to the +applications program, and is used to maintain runtime data, which +does not belong in the image header. + + +.ks +.nf +struct image_descriptor { + long file_size # size of pixfile + long nget # number getpix calls + int nbufs # number of in buffers + int flush # flush outbuf? + int update # update header? + int pfd # pixfile fd + int hfd # header file fd + int flush_epa # epa of imfls? routine + struct buffer_descriptor *ibdes # input bufdes + struct buffer_descriptor *obdes # output bufdes + char imname[SZ_FNAME] # imagefile name +} +.fi +.ke + + + +.ks +.nf +struct buffer_descriptor { + char *bufptr # buffer pointer + int dtype # datatype of pixels + long npix # number of pixels in buf + long bufsize # buffer size, chars + long vs[MAXDIM] # section start vector + long ve[MAXDIM] # section end vector +} +.fi +.ke diff --git a/sys/imio/doc/imio.ms b/sys/imio/doc/imio.ms new file mode 100644 index 00000000..2302a8ff --- /dev/null +++ b/sys/imio/doc/imio.ms @@ -0,0 +1,295 @@ +.ce +\fBThe IRAF Image I/O Interface\fR +.ce +\fIDesign Strategies\fR +.ce +\fIStatus and Plans\fR +.sp +.ce +Doug Tody +.ce +November 1983 +.sp 3 +.NH +Introduction +.PP +Bulk data arrays are accessed in IRAF SPP programs via the Image I/O +(IMIO) interface. IMIO is used to create, read, and write IRAF +\fBimagefiles\fR. The term \fBimage\fR refers to data arrays of one, two, +or more dimensions. Each "imagefile" actually consists of two files: +the \fBheader file\fR and the \fBpixel storage file\fR. Seven disk datatypes +are currently supported. +.PP +The IMIO calling sequences are summarized in the \fIProgrammer's Crib +Sheet\fR. There is as yet no Reference Manual or User's Guide for the package. +Our intention in this document is merely to introduce IMIO, to summarize its +capabilities, and note what is planned for the future. +.NH +Structure +.PP +The basic structure of an applications program which uses IMIO is shown +below. In the current implementation of IMIO the image header is a simple +binary structure, but this will change when DBIO (the database interface) +is implemented. The pixel storage file is accessed via FIO (the IRAF File +I/O interface) which permits arbitrarily large buffers and double or multiple +buffered i/o. All buffers are dynamically allocated and deallocated using +the facilities provided by the MEMIO interface. + +.DS +.cs 1 22 +Command Language + (applications program) + IMIO + DBIO + FIO + OS | + MEMIO | (operating system) + OS | + + + (system independent) | (system dependent) +.DE +.cs 1 + +.NH +Summary of What is Provided by the Current Interface +.PP +The IMIO interface code is mostly concerned with pixel buffer allocation and +manipulation, and with mapping requests to read and write image sections +into file i/o calls. FIO handles all low level i/o. The efficiency of FIO +for sequential image access stems from the fact that the FIO buffers may +be made as large as desired transparently to the outside world (i.e., IMIO), +the number of FIO buffers is variable, and full read-ahead and write-behind +are implemented (provided the OS provides asynchronous i/o facilities). +.PP +IMIO currently provides the following functions/features: + +.RS +.IP (1) +7 disk datatypes (ushort, silrdx). +.IP (2) +6 in-core datatypes (the standard silrdx). +.IP (3) +Images of up to 7 dimensions are currently supported. The maximum +dimensionality is a sysgen parameter. +.IP (4) +Fully automatic multidimensional buffer allocation, resizing, +and deallocation. There is no fixed limit on the size of a buffer (a subraster +may actually exceed the size of the image if boundary extension is employed). +The size of an image is limited only by the resources of the machine. +.IP (5) +An arbitrary number of input buffers (default 1) may be used to access an +image. Buffers are allocated in a round robin fashion, and need not be the +same size, dimension, or datatype. This feature is especially useful for +convolutions, block averaging, and similar operators. +.IP (6) +Fully automatic type conversion on both input and output. Conversion occurs +only when data is accessed, so one need not type convert the entire image +to access a subraster. +.IP (7) +IMIO implements general image sections (described below), coordinate flip, +and subsampling. +.IP (8) +The dimensionality of the image expected by the applications code and the +actual dimension of an image need not agree. If an operator expects a one +dimensional image, for example, it may be used to operate on any line, column, +or pillar of a three dimensional image, on both input and output (see +discussion on image sections below). +.IP (9) +Both "compressed" and "block aligned" storage modes are supported, with IMIO +automatically selecting the optimal choice during image creation (if the +packing efficiency is not above a certain threshold then image lines are +not block aligned). The device blocksize is determined at runtime and +devices with different blocksizes may coexist. +.IP (10) +IMIO may be advised if i/o is to be either highly sequential or highly +random; the buffering strategy will be modified to increase i/o efficiency. +.IP (11) +Pixel storage files may reside on special devices if desired. For example, +the current \fBdisplay\fR routine accesses the image display device as a random +access imagefile via the standard IMIO interface. This was easy to do +because FIO is device independent and allows new devices to be interfaced +dynamically at run time (other examples of special "devices" are the CL, +magtapes, and strings). +.IP (12) +The image header file, which is small, is normally placed in the user's +own directory system. The pixel storage file, on the other hand, is often +very large and is normally placed in a different filesystem. This is +transparent to the user, and has the advantage that bulk data does not +have to be backed up on tape when the user disk is backed up, and throughput +is often higher because the pixel filesystem can be optimized for large +transfers and more nearly contiguous files. +.IP (13) +An image opened with the mode "new_copy" inherits the full image header +of an existing image, including all user defined fields, minus the pixels +and minus all fields which depend on the actual values of the pixels. +.RE + +.PP +The basic i/o facilities are described in the crib sheet. In short, we +have procedures to get or put pixels, lines, or sections. The put calls +are identical to the get calls and all buffer allocation and manipulation +is performed by IMIO. The pixel access routines access a list of pixels +(described by one, two, or more integer arrays giving the coordinates of +the pixels, which are fetched in storage order to minimize seeks). +An additional set of calls are available for accessing all of the lines +in an image sequentially in storage order, regardless of the dimensionality +of the image (as in the FITS reader). +.NH +Planned Enhancements to IMIO +.PP +The following enhancements are currently planned for IMIO; they are +arranged more or less with the highest priority items first. The DBIO +header, boundary extension facilities, and bad pixel list features are +of the highest priority and will be implemented within the next few months. + +.RS +.IP (1) +Replacement of the current rather rigid binary header by the highly +extensible yet efficient DBIO header. +.IP (2) +Automatic boundary extension by any of the following techniques: +nearest neighbor, reflection, projection, wrap around, indefinite, +constant, apodize. Useful for convolutions and subraster extraction +near the boundary of an image. +.IP (3) +Bad pixel list manipulation. A list of bad pixels will optionally be +associated with each image. The actual value of each "bad" pixel in the +image will be a reasonable, artificially generated value. Programs which +do not need to know about bad pixels, such as simple pointwise image +operators, will see only reasonable values. IMIO will provide routines to +merge (etc.) bad pixel lists in simple pointwise image operations. +Operators which need to be able to deal with bad pixels, such as surface +fitting routines, will advise IMIO to replace the bad pixels with the +value INDEF upon input. +.IP (4) +Implement the pixel access routines (\fBimgp__\fR and \fBimpp__\fR). +Currently only the line and section routines are implemented. The section +routines may be used to access individual pixels, but this involves quite +a bit of overhead and disk seeks are not optimized. +.IP (5) +Optimization to the get/put line procedures to work directly +out of the FIO buffers when possible for increased efficiency. +.IP (6) +IMIO (and FIO) dynamically allocate all buffers. Eventually we will add +an "advice" option permitting buffers to be allocated in a region +of memory which is \fIshared\fR with a bit-mapped array processor. +The VOPS primitives, already used extensively for vector operations, +will be interfaced to the AP and applications sofware will then make use +of the AP without modification and without introducing any device +dependence. Note that CSPI is currently marketing a 7 Mflop bit-mapped +AP for the VAX, and Masscomp provides a similar device for their 680000 based +supermicro. +.IP (6) +Support for the unsigned byte disk datatype. +.DE + +.PP +Long range improvements include language support for image sections in +the successor to the SPP (subset) language compiler, and extensions for +block storage mode of images on disk. Currently all images are stored on +disk in line storage mode (i.e., like a Fortran array). +.NH +Image Sections +.PP +Image sections are used to specify the region of an image to be operated +upon. The essential idea is that when the user passes the name of an +image to a task, a special notation is employed which specifies the section +of the image to be operated upon. The image section is decoded by IMIO +at "immap" time and is completely transparent to the applications code +(when a section is used, the image appears smaller to the applications +program). If no section is specified then the entire image is accessed. +.PP +For example, suppose we want to display the image "pix" in frame 1 of the +image display, using all the default parameters: + +.nf + cl> display pix, 1 +.fi + +This works fine as long as "pix" is a one or two dimensional image. If it +is a three dimensional image, we will see only the first band. To display +some other band, we must specify a two-dimensional \fIsection\fR of the +three dimensional image: + +.nf + cl> display pix[*,*,5], 1 + cl> display pix[5], 1 +.fi + +Either command above would display band 5 of the three dimensional image +(higher dimensional images are analogous). To display a dimensional image +with the columns flipped: + +.nf + cl> display pix[*,\(mi*], 1 +.fi + +This command flips the y-axis. To display a subraster: + +.nf + cl> display pix[30:40,310:300], 1 +.fi + +would display the indicated eleven pixel square subraster. To display a +2048 square image on a 512 square display by means of subsampling: + +.nf + cl> display pix[*:4,*:4], 1 +.fi +.NH +Use of Virtual Memory +.PP +The current implementation of IMIO does not make use of any virtual memory +facilities. We have had little incentive to do so because 4.1BSD Berkeley +UNIX does not have a very good implementation of virtual memory (few systems +do, it seems - DG/AOS, which is what CTIO runs, does not have a +good implemenation either). Various strategies can, however, be employed +to take advantage of virtual memory on a machine which provides good +virtual memory facilities. +.PP +One technique is to use IMIO to "extract a subraster" which is in fact +the entire image. The current implementation of IMIO would copy rather +than map the image, but \fIif\fR no type conversion were required, +if no section was specified, if the image was not block-aligned, +and if referencing out of bounds was not required, +IMIO could instead map the image directly into virtual memory. +This would be an easy enhancement to make to IMIO because all data is +accessed with pointers. The code fragment in the following example +demonstrates how this is done in the current version of IMIO. + +.DS +.cs 1 22 +int ncols, nlines +pointer header, raster +pointer immap(), imgs2r() + +begin + # Open or "map" the image. "Imagefile" is a file name + # or a file name with section subscript appended. + + header = immap (imagefile, READ_ONLY, 0) + + ncols = IM_LEN (header, 1) + nlines = IM_LEN (header, 2) + + # Read or map entire image into memory. Pixels are + # converted to type real if necessary. + + raster = imgs2r (header, 1, ncols, 1, nlines) + + # Call SPP or Fortran subroutine to process type real + # image. Note how the pointer "raster" is dereferenced. + + call subroutine (Memr[raster], ncols, nlines) + ... +.DE +.cs 1 + +.PP +Another, slightly different approach would be to allocate a single FIO +buffer and map it onto the entire file. This would require no modifications +to IMIO, rather one would modify the "file fault" code in FIO. +This scheme would more efficiently support random access (to image lines or +subrasters) on a virtual machine without introducing a real dependence +on virtual memory. |