path: root/sys/imio/doc/imio.ms

.ce
\fBThe IRAF Image I/O Interface\fR
.ce
\fIDesign Strategies\fR
.ce
\fIStatus and Plans\fR
.sp
.ce
Doug Tody
.ce
November 1983
.sp 3
.NH
Introduction
.PP
Bulk data arrays are accessed in IRAF SPP programs via the Image I/O
(IMIO) interface.  IMIO is used to create, read, and write IRAF
\fBimagefiles\fR.  The term \fBimage\fR refers to data arrays of one, two,
or more dimensions.  Each "imagefile" actually consists of two files:
the \fBheader file\fR and the \fBpixel storage file\fR.  Seven disk datatypes
are currently supported.
.PP
The IMIO calling sequences are summarized in the \fIProgrammer's Crib
Sheet\fR.  There is as yet no Reference Manual or User's Guide for the package.
Our intention in this document is merely to introduce IMIO, to summarize its
capabilities, and note what is planned for the future.
.NH
Structure
.PP
The basic structure of an applications program which uses IMIO is shown
below.  In the current implementation of IMIO the image header is a simple
binary structure, but this will change when DBIO (the database interface)
is implemented.  The pixel storage file is accessed via FIO (the IRAF File
I/O interface) which permits arbitrarily large buffers and double or multiple
buffered i/o.  All buffers are dynamically allocated and deallocated using
the facilities provided by the MEMIO interface.

.DS
.cs 1 22
Command Language
        (applications program)
                IMIO
                        DBIO
                        FIO
                                OS   |
                        MEMIO        |  (operating system)
                                OS   |


        (system independent)  |  (system dependent)
.DE
.cs 1

.NH
Summary of What is Provided by the Current Interface
.PP
The IMIO interface code is mostly concerned with pixel buffer allocation and
manipulation, and with mapping requests to read and write image sections
into file i/o calls.  FIO handles all low level i/o.  The efficiency of FIO
for sequential image access stems from the fact that the FIO buffers may
be made as large as desired transparently to the outside world (i.e., IMIO),
the number of FIO buffers is variable, and full read-ahead and write-behind
are implemented (provided the OS provides asynchronous i/o facilities).
.PP
IMIO currently provides the following functions/features:

.RS
.IP (1)
7 disk datatypes (ushort, silrdx).
.IP (2)
6 in-core datatypes (the standard silrdx).
.IP (3)
Images of up to 7 dimensions are currently supported.  The maximum
dimensionality is a sysgen parameter.
.IP (4)
Fully automatic multidimensional buffer allocation, resizing,
and deallocation.  There is no fixed limit on the size of a buffer (a subraster
may actually exceed the size of the image if boundary extension is employed).
The size of an image is limited only by the resources of the machine.
.IP (5)
An arbitrary number of input buffers (default 1) may be used to access an
image.  Buffers are allocated in a round robin fashion, and need not be the
same size, dimension, or datatype.  This feature is especially useful for
convolutions, block averaging, and similar operators.
.IP (6)
Fully automatic type conversion on both input and output.  Conversion occurs
only when data is accessed, so one need not type convert the entire image
to access a subraster.
.IP (7)
IMIO implements general image sections (described below), coordinate flip,
and subsampling.
.IP (8)
The dimensionality of the image expected by the applications code and the
actual dimension of an image need not agree.  If an operator expects a one
dimensional image, for example, it may be used to operate on any line, column,
or pillar of a three dimensional image, on both input and output (see
discussion on image sections below).
.IP (9)
Both "compressed" and "block aligned" storage modes are supported, with IMIO
automatically selecting the optimal choice during image creation (if the
packing efficiency is not above a certain threshold then image lines are
not block aligned).  The device blocksize is determined at runtime and
devices with different blocksizes may coexist.
.IP (10)
IMIO may be advised if i/o is to be either highly sequential or highly
random; the buffering strategy will be modified to increase i/o efficiency.
.IP (11)
Pixel storage files may reside on special devices if desired.  For example,
the current \fBdisplay\fR routine accesses the image display device as a random
access imagefile via the standard IMIO interface.  This was easy to do
because FIO is device independent and allows new devices to be interfaced
dynamically at run time (other examples of special "devices" are the CL,
magtapes, and strings).
.IP (12)
The image header file, which is small, is normally placed in the user's
own directory system.  The pixel storage file, on the other hand, is often
very large and is normally placed in a different filesystem.  This is
transparent to the user, and has the advantage that bulk data does not
have to be backed up on tape when the user disk is backed up, and throughput
is often higher because the pixel filesystem can be optimized for large
transfers and more nearly contiguous files.
.IP (13)
An image opened with the mode "new_copy" inherits the full image header
of an existing image, including all user defined fields, minus the pixels
and minus all fields which depend on the actual values of the pixels.
.RE

.PP
The basic i/o facilities are described in the crib sheet.  In short, we
have procedures to get or put pixels, lines, or sections.  The put calls
are identical to the get calls and all buffer allocation and manipulation
is performed by IMIO.  The pixel access routines access a list of pixels
(described by one, two, or more integer arrays giving the coordinates of
the pixels, which are fetched in storage order to minimize seeks).
An additional set of calls are available for accessing all of the lines
in an image sequentially in storage order, regardless of the dimensionality
of the image (as in the FITS reader).
.NH
Planned Enhancements to IMIO
.PP
The following enhancements are currently planned for IMIO; they are
arranged more or less with the highest priority items first.  The DBIO
header, boundary extension facilities, and bad pixel list features are
of the highest priority and will be implemented within the next few months.

.RS
.IP (1)
Replacement of the current rather rigid binary header by the highly
extensible yet efficient DBIO header.
.IP (2)
Automatic boundary extension by any of the following techniques:
nearest neighbor, reflection, projection, wrap around, indefinite,
constant, apodize.  Useful for convolutions and subraster extraction
near the boundary of an image.
.IP (3)
Bad pixel list manipulation.  A list of bad pixels will optionally be
associated with each image.  The actual value of each "bad" pixel in the
image will be a reasonable, artificially generated value.  Programs which
do not need to know about bad pixels, such as simple pointwise image
operators, will see only reasonable values.  IMIO will provide routines to
merge (etc.) bad pixel lists in simple pointwise image operations.
Operators which need to be able to deal with bad pixels, such as surface
fitting routines, will advise IMIO to replace the bad pixels with the
value INDEF upon input.
.IP (4)
Implement the pixel access routines (\fBimgp__\fR and \fBimpp__\fR).
Currently only the line and section routines are implemented.  The section
routines may be used to access individual pixels, but this involves quite
a bit of overhead and disk seeks are not optimized.
.IP (5)
Optimization to the get/put line procedures to work directly
out of the FIO buffers when possible for increased efficiency.
.IP (6)
IMIO (and FIO) dynamically allocate all buffers.  Eventually we will add
an "advice" option permitting buffers to be allocated in a region
of memory which is \fIshared\fR with a bit-mapped array processor. 
The VOPS primitives, already used extensively for vector operations,
will be interfaced to the AP and applications sofware will then make use
of the AP without modification and without introducing any device
dependence.  Note that CSPI is currently marketing a 7 Mflop bit-mapped
AP for the VAX, and Masscomp provides a similar device for their 680000 based
supermicro.
.IP (6)
Support for the unsigned byte disk datatype.
.DE

.PP
Long range improvements include language support for image sections in
the successor to the SPP (subset) language compiler, and extensions for
block storage mode of images on disk.  Currently all images are stored on
disk in line storage mode (i.e., like a Fortran array).
.NH
Image Sections
.PP
Image sections are used to specify the region of an image to be operated
upon.  The essential idea is that when the user passes the name of an
image to a task, a special notation is employed which specifies the section
of the image to be operated upon.  The image section is decoded by IMIO
at "immap" time and is completely transparent to the applications code
(when a section is used, the image appears smaller to the applications
program).  If no section is specified then the entire image is accessed.
.PP
For example, suppose we want to display the image "pix" in frame 1 of the
image display, using all the default parameters:

.nf
	cl> display pix, 1
.fi

This works fine as long as "pix" is a one or two dimensional image.  If it
is a three dimensional image, we will see only the first band.  To display
some other band, we must specify a two-dimensional \fIsection\fR of the
three dimensional image:

.nf
	cl> display pix[*,*,5], 1
	cl> display pix[5], 1
.fi

Either command above would display band 5 of the three dimensional image
(higher dimensional images are analogous).  To display a dimensional image
with the columns flipped:

.nf
	cl> display pix[*,\(mi*], 1
.fi

This command flips the y-axis.  To display a subraster:

.nf
	cl> display pix[30:40,310:300], 1
.fi

would display the indicated eleven pixel square subraster.  To display a
2048 square image on a 512 square display by means of subsampling:

.nf
	cl> display pix[*:4,*:4], 1
.fi
.NH
Use of Virtual Memory
.PP
The current implementation of IMIO does not make use of any virtual memory
facilities.  We have had little incentive to do so because 4.1BSD Berkeley
UNIX does not have a very good implementation of virtual memory (few systems
do, it seems - DG/AOS, which is what CTIO runs, does not have a
good implemenation either).  Various strategies can, however, be employed
to take advantage of virtual memory on a machine which provides good
virtual memory facilities.
.PP
One technique is to use IMIO to "extract a subraster" which is in fact
the entire image.  The current implementation of IMIO would copy rather
than map the image, but \fIif\fR no type conversion were required,
if no section was specified, if the image was not block-aligned,
and if referencing out of bounds was not required,
IMIO could instead map the image directly into virtual memory.
This would be an easy enhancement to make to IMIO because all data is
accessed with pointers.  The code fragment in the following example
demonstrates how this is done in the current version of IMIO.

.DS
.cs 1 22
int     ncols, nlines
pointer header, raster
pointer immap(), imgs2r()

begin
        # Open or "map" the image.  "Imagefile" is a file name
        # or a file name with section subscript appended.

        header = immap (imagefile, READ_ONLY, 0)

        ncols  = IM_LEN (header, 1)
        nlines = IM_LEN (header, 2)

        # Read or map entire image into memory.  Pixels are
        # converted to type real if necessary.

        raster = imgs2r (header, 1, ncols, 1, nlines)
        
        # Call SPP or Fortran subroutine to process type real
        # image.  Note how the pointer "raster" is dereferenced.

        call subroutine (Memr[raster], ncols, nlines)
            ...
.DE
.cs 1

.PP
Another, slightly different approach would be to allocate a single FIO
buffer and map it onto the entire file.  This would require no modifications
to IMIO, rather one would modify the "file fault" code in FIO.
This scheme would more efficiently support random access (to image lines or
subrasters) on a virtual machine without introducing a real dependence
on virtual memory.