Enhanced Image List Template Package April 15, 2011 The enhanced image list package provides new capabilities for handling image lists, but remains backwards compatible with tasks currently using the IMT interface. The enhancements allow for expansion of MEF files into lists of extensions using the @-file operator, as well as selection of images within more general lists by means of modifiers (e.g. a simple expression such as the extname/extver or explicit extension number, or more complex boolean expressions to allow selection by header keyword). In addition, tables may now take the @-file operator to use a column containing image references as an input list. ======================================================================== TODO: - Describe syntax for use with tables and selection by row values - Describe remote image specification caching mechanism ======================================================================== Template Strings ---------------- The FNT template package supports the following forms of pattern strings: alpha, *.x, data* // .pix, [a-m]*, @list_file, nite%1%2%.1024.imh i.e. simple filenames, wildcard expansion in filenames, concatenation of filenames, @files, substitution in filenames, or a comma-delimited list of the above. The image template package (IMT) extends these patterns to allow image names followed by a cluster index or image section in [] brackets. These patterns remain unchanged in the new version of the package to allow backward compatability with existing applications. Lists of these types represent *explicit* collections of images, i.e. a collection based on the image name (wildcards) or as a result of processing by some task (e.g. expansion of an MEF file to create an input @-file of expanded extension specifications). The enhanced version of the IMT package further abstracts the concept of image collections to include data objects such as MEF files or tables containing a list of image references that *implicitly* defines the list (e.g. the expanded MEF extension specification or the complete column of image references). Further, we allow this list (which might be quite broad) to be refined using modifiers or selectors on the list and thus dynamically create the list without requiring the use (and management) of intermediate files. For example, @file* expand all files beginning w/ 'file' @file//".fits" append ".fits" to contents of 'file' @mef.fits expand all (image) extensions of an MEF file @mef.fits[SCI] select SCI extensions from MEF file @mef.fits[SCI,2][noinherit] select v2 SCI extns, add kernel param @mef.fits[1-16x2] select range of extensions from MEF file @mef.fits[+1-8] create a list of extensions for an MEF *.fits[1:100,1:100] append section to all FITS images @@file[1:100,1:100] append section to expanded MEFs in 'file' *.fits[filter?='V'] select images w/ FILTER keyword containing 'V' @*.fits[gain<3.0] select image extns where GAIN keyword < 3.0 *.fits[filter?='V';gain<2.0] select using multiple OR's expressions Template Syntax --------------- The previous syntax and behavior of image templates is unchanged in this version, new functionality is provided by (optional) new syntax now supported in the template pattern string. Briefly, - wildcard filename expansion may now be applied to @-files - the use of an '@' operator is now permitted on MEF files. By default, all image extensions in the file will be included in the list, modifiers may be used to select specific extensions or to indicate a range of extensions to be used. - the use of an '@@' operator to indicate expansion of the contents of an @-file. For example, an @-file of MEF image names can be expanded to list of all the file extensions using "@@file", whereas just using "@file" would list the names of the MEF files as before. - modifier expressions enclosed in square brackets may be appended to an image template string (or @-file) to either constrain the list (e.g. only a range of MEF extensions, only images with a certain keyword value, etc) or to append extra information to the image specification (e.g. to add an image section to all images in the list). Multiple modifier expressions may be used The allowed syntax for a template string can be described roughly in the following way: [@@ | @] [extname] [;...] [] [
] [@@ | @] [extname,extver][;...] [] [
] [@@ | @] [index_range] [;...] [] [
] <-------- selectors -------> <------ modifiers -----> The specification may be the name of a file, and image, or a table. The behavior of the @-file and @@-file operators will depend on the type of but the @-file usage remains backward compatible when used with text files. The use of a modifier/selection on an MEF file will automatically trigger expansion of the extensions in the image and so an '@' operator is not strictly required, however only those extensions matching the selection expression will be present in the final image list. Note that the use of index ranges and extname/extver selectors are mutually exclusive, selector expressions may be added to either. @-file Operations ----------------- The @-file operator is unchanged from previous versions when used with text files of image names. Modifier/selector expressions however can now be applied to the contents of the @-file to select from the list only those images that match the selector expression, or to augment the name in the list with an additional image syntax such as a section or kernel parameters. @@-file Operations ------------------ The @@-file operator is new syntax meant to allow the contents of an @-file to be expanded automaticaly, e.g. as if there were an @-file of @-file names. Primarily this can be used to create a list of MEF image names in which an @-file would return the names of the MEF, while the @@-file syntax could be used to expand each MEF into individual extension specifications. Extension Indices ----------------- The [index_range] modifier may be used to specify an explicit set of extensions to be used. Index ranges are specified as a comma-delimited list of strings specifying individual range segments as described in the RANGES help page. The use of a '+' operator before an index range indicates the range list should be expanded without checking that the extension exists in the MEF itself. Otherwise, only those extensions present in the MEF will be included in the list. Selection Expressions --------------------- Selection expressions may be used to restrict a template list to only those images that match some boolean expression, e.g. to provide for selection based on a header keyword value. Expressions follow the same guidelines as in the HSELECT task 'expr' parameter (see the help page for details). Multiple expressions may be specified if they are separated by a semicolon however they are evaluated as a single expression of OR'd values rather than as individual expressions. This is significant when considering that expressions may contain keywords not present in all images being checked, for instance *.fits[filter?='V';gain<3.0] would evaluate as if the expression had been written (filter?='V' || gain < 3.0) If a particular image lacks either the 'filter' or 'gain' keyword the entire expression will evaluate to false because of an error even if one of the two clauses would otherwise have been true. [NOTE: This behavior will be changed in a future version.] Image Sections -------------- Image sections may be added to an image specification by adding a separate modifier string. The section will be added once selection of the list by the selector expressions is complete. An example of where this might be used is in automatically specifying the bias section for all images in a list, e.g. @mef.fits[1:128,*] all extensions in the image @mef.fits[1-16x2][1:128,*] only 'left' amplifiers of a mosaic @mef.fits[2-16x2][1024:1128,*] only 'right' amplifiers of a mosaic m31*.fits[345:528,200:300] same section in all registered images No check is made that the image section is valid for the given image. Kernel Parameters ----------------- A comma-delimited list of image kernel parameters may be added to any image specification by adding the keywords to a separate modifier. For example, @mef.fits[1-8][noinherit,padline=30] would expand the file 'mef.fits' to include extensions 1 thru 8 and add the kernel parameters, generating a list such as mef.fits[1][noinherit,padline=30] mef.fits[2][noinherit,padline=30] : : : : mef.fits[8][noinherit,padline=30] No check is made to verify that the image kernel keywords are appropriate for the image type. Supplying an incorrect kernel parameter will likely result in the task throwing an error when opening the image. -------------------------------------------------------------------------------- Appendix 1: Examples file file* @file @file* @file[2] extension @file[SCI] extname @file[SCI,2] extname+extver @file[2][noinherit] extension + ikiparams @file[SCI][noinherit] extname + ikiparams @file[SCI,2][noinherit] extname+extver + ikiparams @file[2][1:20,2:30] extension + section @file[SCI][1:20,2:30] extname + section @file[SCI,2][1:20,2:30] extname+extver + section @file[2][noinherit][1:20,2:30] extension + ikiparams + section @file[SCI][noinherit][1:20,2:30] extname + ikiparams + section @file[SCI,2][noinherit][1:20,2:30] extname+extver + ikiparams + section @file[noinherit] ikiparams @file[noinherit][1:123,2:234] ikiparams + sections @file[1:123,2:234] sections @file[1:123,2:234] sections mef*.fits[filter?='V'] selection expression mef*.fits[filter?='V';filter?='B'] selection expressions (OR) mef*.fits[filter?='V'||filter?='B'] selection expressions (OR) mef*.fits[gain>0.5&&gain<2.0] selection expressions (AND) Expressions will evaluate to 'false' if there is an error such as "keyword not found", meaning that no images will match when one or more keywords may not be present. Best to use a comma-delimited list in this case. Concatenation @file // foo append @file* // foo append wildcards @file // [2] append modifiers foo // @file prepend foo // @file* prepend wildcards foo // @file[2] prepend modifiers Prior Behavior: foo // bar.fits ==> foobar.fits foo.fits // bar ==> foobar.fits foo // @file1 ==> foosif1.fits,foomef1.fits @file1 // bar ==> sif1foo.fits,mef1foo.fits