From fa080de7afc95aa1c19a6e6fc0e0708ced2eadc4 Mon Sep 17 00:00:00 2001 From: Joseph Hunkeler Date: Wed, 8 Jul 2015 20:46:52 -0400 Subject: Initial commit --- sys/imio/doc/imio.ms | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 295 insertions(+) create mode 100644 sys/imio/doc/imio.ms (limited to 'sys/imio/doc/imio.ms') diff --git a/sys/imio/doc/imio.ms b/sys/imio/doc/imio.ms new file mode 100644 index 00000000..2302a8ff --- /dev/null +++ b/sys/imio/doc/imio.ms @@ -0,0 +1,295 @@ +.ce +\fBThe IRAF Image I/O Interface\fR +.ce +\fIDesign Strategies\fR +.ce +\fIStatus and Plans\fR +.sp +.ce +Doug Tody +.ce +November 1983 +.sp 3 +.NH +Introduction +.PP +Bulk data arrays are accessed in IRAF SPP programs via the Image I/O +(IMIO) interface. IMIO is used to create, read, and write IRAF +\fBimagefiles\fR. The term \fBimage\fR refers to data arrays of one, two, +or more dimensions. Each "imagefile" actually consists of two files: +the \fBheader file\fR and the \fBpixel storage file\fR. Seven disk datatypes +are currently supported. +.PP +The IMIO calling sequences are summarized in the \fIProgrammer's Crib +Sheet\fR. There is as yet no Reference Manual or User's Guide for the package. +Our intention in this document is merely to introduce IMIO, to summarize its +capabilities, and note what is planned for the future. +.NH +Structure +.PP +The basic structure of an applications program which uses IMIO is shown +below. In the current implementation of IMIO the image header is a simple +binary structure, but this will change when DBIO (the database interface) +is implemented. The pixel storage file is accessed via FIO (the IRAF File +I/O interface) which permits arbitrarily large buffers and double or multiple +buffered i/o. All buffers are dynamically allocated and deallocated using +the facilities provided by the MEMIO interface. + +.DS +.cs 1 22 +Command Language + (applications program) + IMIO + DBIO + FIO + OS | + MEMIO | (operating system) + OS | + + + (system independent) | (system dependent) +.DE +.cs 1 + +.NH +Summary of What is Provided by the Current Interface +.PP +The IMIO interface code is mostly concerned with pixel buffer allocation and +manipulation, and with mapping requests to read and write image sections +into file i/o calls. FIO handles all low level i/o. The efficiency of FIO +for sequential image access stems from the fact that the FIO buffers may +be made as large as desired transparently to the outside world (i.e., IMIO), +the number of FIO buffers is variable, and full read-ahead and write-behind +are implemented (provided the OS provides asynchronous i/o facilities). +.PP +IMIO currently provides the following functions/features: + +.RS +.IP (1) +7 disk datatypes (ushort, silrdx). +.IP (2) +6 in-core datatypes (the standard silrdx). +.IP (3) +Images of up to 7 dimensions are currently supported. The maximum +dimensionality is a sysgen parameter. +.IP (4) +Fully automatic multidimensional buffer allocation, resizing, +and deallocation. There is no fixed limit on the size of a buffer (a subraster +may actually exceed the size of the image if boundary extension is employed). +The size of an image is limited only by the resources of the machine. +.IP (5) +An arbitrary number of input buffers (default 1) may be used to access an +image. Buffers are allocated in a round robin fashion, and need not be the +same size, dimension, or datatype. This feature is especially useful for +convolutions, block averaging, and similar operators. +.IP (6) +Fully automatic type conversion on both input and output. Conversion occurs +only when data is accessed, so one need not type convert the entire image +to access a subraster. +.IP (7) +IMIO implements general image sections (described below), coordinate flip, +and subsampling. +.IP (8) +The dimensionality of the image expected by the applications code and the +actual dimension of an image need not agree. If an operator expects a one +dimensional image, for example, it may be used to operate on any line, column, +or pillar of a three dimensional image, on both input and output (see +discussion on image sections below). +.IP (9) +Both "compressed" and "block aligned" storage modes are supported, with IMIO +automatically selecting the optimal choice during image creation (if the +packing efficiency is not above a certain threshold then image lines are +not block aligned). The device blocksize is determined at runtime and +devices with different blocksizes may coexist. +.IP (10) +IMIO may be advised if i/o is to be either highly sequential or highly +random; the buffering strategy will be modified to increase i/o efficiency. +.IP (11) +Pixel storage files may reside on special devices if desired. For example, +the current \fBdisplay\fR routine accesses the image display device as a random +access imagefile via the standard IMIO interface. This was easy to do +because FIO is device independent and allows new devices to be interfaced +dynamically at run time (other examples of special "devices" are the CL, +magtapes, and strings). +.IP (12) +The image header file, which is small, is normally placed in the user's +own directory system. The pixel storage file, on the other hand, is often +very large and is normally placed in a different filesystem. This is +transparent to the user, and has the advantage that bulk data does not +have to be backed up on tape when the user disk is backed up, and throughput +is often higher because the pixel filesystem can be optimized for large +transfers and more nearly contiguous files. +.IP (13) +An image opened with the mode "new_copy" inherits the full image header +of an existing image, including all user defined fields, minus the pixels +and minus all fields which depend on the actual values of the pixels. +.RE + +.PP +The basic i/o facilities are described in the crib sheet. In short, we +have procedures to get or put pixels, lines, or sections. The put calls +are identical to the get calls and all buffer allocation and manipulation +is performed by IMIO. The pixel access routines access a list of pixels +(described by one, two, or more integer arrays giving the coordinates of +the pixels, which are fetched in storage order to minimize seeks). +An additional set of calls are available for accessing all of the lines +in an image sequentially in storage order, regardless of the dimensionality +of the image (as in the FITS reader). +.NH +Planned Enhancements to IMIO +.PP +The following enhancements are currently planned for IMIO; they are +arranged more or less with the highest priority items first. The DBIO +header, boundary extension facilities, and bad pixel list features are +of the highest priority and will be implemented within the next few months. + +.RS +.IP (1) +Replacement of the current rather rigid binary header by the highly +extensible yet efficient DBIO header. +.IP (2) +Automatic boundary extension by any of the following techniques: +nearest neighbor, reflection, projection, wrap around, indefinite, +constant, apodize. Useful for convolutions and subraster extraction +near the boundary of an image. +.IP (3) +Bad pixel list manipulation. A list of bad pixels will optionally be +associated with each image. The actual value of each "bad" pixel in the +image will be a reasonable, artificially generated value. Programs which +do not need to know about bad pixels, such as simple pointwise image +operators, will see only reasonable values. IMIO will provide routines to +merge (etc.) bad pixel lists in simple pointwise image operations. +Operators which need to be able to deal with bad pixels, such as surface +fitting routines, will advise IMIO to replace the bad pixels with the +value INDEF upon input. +.IP (4) +Implement the pixel access routines (\fBimgp__\fR and \fBimpp__\fR). +Currently only the line and section routines are implemented. The section +routines may be used to access individual pixels, but this involves quite +a bit of overhead and disk seeks are not optimized. +.IP (5) +Optimization to the get/put line procedures to work directly +out of the FIO buffers when possible for increased efficiency. +.IP (6) +IMIO (and FIO) dynamically allocate all buffers. Eventually we will add +an "advice" option permitting buffers to be allocated in a region +of memory which is \fIshared\fR with a bit-mapped array processor. +The VOPS primitives, already used extensively for vector operations, +will be interfaced to the AP and applications sofware will then make use +of the AP without modification and without introducing any device +dependence. Note that CSPI is currently marketing a 7 Mflop bit-mapped +AP for the VAX, and Masscomp provides a similar device for their 680000 based +supermicro. +.IP (6) +Support for the unsigned byte disk datatype. +.DE + +.PP +Long range improvements include language support for image sections in +the successor to the SPP (subset) language compiler, and extensions for +block storage mode of images on disk. Currently all images are stored on +disk in line storage mode (i.e., like a Fortran array). +.NH +Image Sections +.PP +Image sections are used to specify the region of an image to be operated +upon. The essential idea is that when the user passes the name of an +image to a task, a special notation is employed which specifies the section +of the image to be operated upon. The image section is decoded by IMIO +at "immap" time and is completely transparent to the applications code +(when a section is used, the image appears smaller to the applications +program). If no section is specified then the entire image is accessed. +.PP +For example, suppose we want to display the image "pix" in frame 1 of the +image display, using all the default parameters: + +.nf + cl> display pix, 1 +.fi + +This works fine as long as "pix" is a one or two dimensional image. If it +is a three dimensional image, we will see only the first band. To display +some other band, we must specify a two-dimensional \fIsection\fR of the +three dimensional image: + +.nf + cl> display pix[*,*,5], 1 + cl> display pix[5], 1 +.fi + +Either command above would display band 5 of the three dimensional image +(higher dimensional images are analogous). To display a dimensional image +with the columns flipped: + +.nf + cl> display pix[*,\(mi*], 1 +.fi + +This command flips the y-axis. To display a subraster: + +.nf + cl> display pix[30:40,310:300], 1 +.fi + +would display the indicated eleven pixel square subraster. To display a +2048 square image on a 512 square display by means of subsampling: + +.nf + cl> display pix[*:4,*:4], 1 +.fi +.NH +Use of Virtual Memory +.PP +The current implementation of IMIO does not make use of any virtual memory +facilities. We have had little incentive to do so because 4.1BSD Berkeley +UNIX does not have a very good implementation of virtual memory (few systems +do, it seems - DG/AOS, which is what CTIO runs, does not have a +good implemenation either). Various strategies can, however, be employed +to take advantage of virtual memory on a machine which provides good +virtual memory facilities. +.PP +One technique is to use IMIO to "extract a subraster" which is in fact +the entire image. The current implementation of IMIO would copy rather +than map the image, but \fIif\fR no type conversion were required, +if no section was specified, if the image was not block-aligned, +and if referencing out of bounds was not required, +IMIO could instead map the image directly into virtual memory. +This would be an easy enhancement to make to IMIO because all data is +accessed with pointers. The code fragment in the following example +demonstrates how this is done in the current version of IMIO. + +.DS +.cs 1 22 +int ncols, nlines +pointer header, raster +pointer immap(), imgs2r() + +begin + # Open or "map" the image. "Imagefile" is a file name + # or a file name with section subscript appended. + + header = immap (imagefile, READ_ONLY, 0) + + ncols = IM_LEN (header, 1) + nlines = IM_LEN (header, 2) + + # Read or map entire image into memory. Pixels are + # converted to type real if necessary. + + raster = imgs2r (header, 1, ncols, 1, nlines) + + # Call SPP or Fortran subroutine to process type real + # image. Note how the pointer "raster" is dereferenced. + + call subroutine (Memr[raster], ncols, nlines) + ... +.DE +.cs 1 + +.PP +Another, slightly different approach would be to allocate a single FIO +buffer and map it onto the entire file. This would require no modifications +to IMIO, rather one would modify the "file fault" code in FIO. +This scheme would more efficiently support random access (to image lines or +subrasters) on a virtual machine without introducing a real dependence +on virtual memory. -- cgit