1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
|
Enhanced Image List Template Package
April 15, 2011
The enhanced image list package provides new capabilities for handling
image lists, but remains backwards compatible with tasks currently using
the IMT interface. The enhancements allow for expansion of MEF files into
lists of extensions using the @-file operator, as well as selection of
images within more general lists by means of modifiers (e.g. a simple
expression such as the extname/extver or explicit extension number, or more
complex boolean expressions to allow selection by header keyword). In
addition, tables may now take the @-file operator to use a column
containing image references as an input list.
========================================================================
TODO:
- Describe syntax for use with tables and selection by row values
- Describe remote image specification caching mechanism
========================================================================
Template Strings
----------------
The FNT template package supports the following forms of pattern strings:
alpha, *.x, data* // .pix, [a-m]*, @list_file, nite%1%2%.1024.imh
i.e. simple filenames, wildcard expansion in filenames, concatenation of
filenames, @files, substitution in filenames, or a comma-delimited list of
the above. The image template package (IMT) extends these patterns to
allow image names followed by a cluster index or image section in []
brackets. These patterns remain unchanged in the new version of the
package to allow backward compatability with existing applications. Lists
of these types represent *explicit* collections of images, i.e. a
collection based on the image name (wildcards) or as a result of processing
by some task (e.g. expansion of an MEF file to create an input @-file of
expanded extension specifications).
The enhanced version of the IMT package further abstracts the concept
of image collections to include data objects such as MEF files or tables
containing a list of image references that *implicitly* defines the list
(e.g. the expanded MEF extension specification or the complete column of
image references). Further, we allow this list (which might be quite
broad) to be refined using modifiers or selectors on the list and thus
dynamically create the list without requiring the use (and management) of
intermediate files. For example,
@file* expand all files beginning w/ 'file'
@file//".fits" append ".fits" to contents of 'file'
@mef.fits expand all (image) extensions of an MEF file
@mef.fits[SCI] select SCI extensions from MEF file
@mef.fits[SCI,2][noinherit] select v2 SCI extns, add kernel param
@mef.fits[1-16x2] select range of extensions from MEF file
@mef.fits[+1-8] create a list of extensions for an MEF
*.fits[1:100,1:100] append section to all FITS images
@@file[1:100,1:100] append section to expanded MEFs in 'file'
*.fits[filter?='V'] select images w/ FILTER keyword containing 'V'
@*.fits[gain<3.0] select image extns where GAIN keyword < 3.0
*.fits[filter?='V';gain<2.0] select using multiple OR's expressions
Template Syntax
---------------
The previous syntax and behavior of image templates is unchanged in
this version, new functionality is provided by (optional) new syntax now
supported in the template pattern string. Briefly,
- wildcard filename expansion may now be applied to @-files
- the use of an '@' operator is now permitted on MEF files. By
default, all image extensions in the file will be included in the
list, modifiers may be used to select specific extensions or to
indicate a range of extensions to be used.
- the use of an '@@' operator to indicate expansion of the contents
of an @-file. For example, an @-file of MEF image names can be
expanded to list of all the file extensions using "@@file", whereas
just using "@file" would list the names of the MEF files as before.
- modifier expressions enclosed in square brackets may be appended
to an image template string (or @-file) to either constrain the
list (e.g. only a range of MEF extensions, only images with a certain
keyword value, etc) or to append extra information to the image
specification (e.g. to add an image section to all images in the
list). Multiple modifier expressions may be used
The allowed syntax for a template string can be described roughly in
the following way:
[@@ | @] <file> [extname] [<expr>;...] [<ikparams>] [<section>]
[@@ | @] <file> [extname,extver][<expr>;...] [<ikparams>] [<section>]
[@@ | @] <file> [index_range] [<expr>;...] [<ikparams>] [<section>]
<-------- selectors -------> <------ modifiers ----->
The <file> specification may be the name of a file, and image, or a
table. The behavior of the @-file and @@-file operators will depend on
the type of <file> but the @-file usage remains backward compatible when
used with text files.
The use of a modifier/selection on an MEF file will automatically
trigger expansion of the extensions in the image and so an '@' operator is
not strictly required, however only those extensions matching the selection
expression will be present in the final image list. Note that the use of
index ranges and extname/extver selectors are mutually exclusive, selector
expressions may be added to either.
@-file Operations
-----------------
The @-file operator is unchanged from previous versions when used with
text files of image names. Modifier/selector expressions however can now
be applied to the contents of the @-file to select from the list only those
images that match the selector expression, or to augment the name in the
list with an additional image syntax such as a section or kernel parameters.
@@-file Operations
------------------
The @@-file operator is new syntax meant to allow the contents of an
@-file to be expanded automaticaly, e.g. as if there were an @-file of
@-file names. Primarily this can be used to create a list of MEF image
names in which an @-file would return the names of the MEF, while the
@@-file syntax could be used to expand each MEF into individual extension
specifications.
Extension Indices
-----------------
The [index_range] modifier may be used to specify an explicit set of
extensions to be used. Index ranges are specified as a comma-delimited
list of strings specifying individual range segments as described in the
RANGES help page.
The use of a '+' operator before an index range indicates the range
list should be expanded without checking that the extension exists in the
MEF itself. Otherwise, only those extensions present in the MEF will be
included in the list.
Selection Expressions
---------------------
Selection expressions may be used to restrict a template list to only
those images that match some boolean expression, e.g. to provide for
selection based on a header keyword value. Expressions follow the same
guidelines as in the HSELECT task 'expr' parameter (see the help page
for details). Multiple expressions may be specified if they are separated
by a semicolon however they are evaluated as a single expression of
OR'd values rather than as individual expressions. This is significant
when considering that expressions may contain keywords not present in
all images being checked, for instance
*.fits[filter?='V';gain<3.0]
would evaluate as if the expression had been written
(filter?='V' || gain < 3.0)
If a particular image lacks either the 'filter' or 'gain' keyword the
entire expression will evaluate to false because of an error even if one
of the two clauses would otherwise have been true.
[NOTE: This behavior will be changed in a future version.]
Image Sections
--------------
Image sections may be added to an image specification by adding a
separate modifier string. The section will be added once selection of
the list by the selector expressions is complete. An example of where
this might be used is in automatically specifying the bias section for
all images in a list, e.g.
@mef.fits[1:128,*] all extensions in the image
@mef.fits[1-16x2][1:128,*] only 'left' amplifiers of a mosaic
@mef.fits[2-16x2][1024:1128,*] only 'right' amplifiers of a mosaic
m31*.fits[345:528,200:300] same section in all registered images
No check is made that the image section is valid for the given image.
Kernel Parameters
-----------------
A comma-delimited list of image kernel parameters may be added to any
image specification by adding the keywords to a separate modifier. For
example,
@mef.fits[1-8][noinherit,padline=30]
would expand the file 'mef.fits' to include extensions 1 thru 8 and add the
kernel parameters, generating a list such as
mef.fits[1][noinherit,padline=30]
mef.fits[2][noinherit,padline=30]
: : : :
mef.fits[8][noinherit,padline=30]
No check is made to verify that the image kernel keywords are appropriate
for the image type. Supplying an incorrect kernel parameter will likely
result in the task throwing an error when opening the image.
--------------------------------------------------------------------------------
Appendix 1: Examples
file
file*
@file
@file*
@file[2] extension
@file[SCI] extname
@file[SCI,2] extname+extver
@file[2][noinherit] extension + ikiparams
@file[SCI][noinherit] extname + ikiparams
@file[SCI,2][noinherit] extname+extver + ikiparams
@file[2][1:20,2:30] extension + section
@file[SCI][1:20,2:30] extname + section
@file[SCI,2][1:20,2:30] extname+extver + section
@file[2][noinherit][1:20,2:30] extension + ikiparams + section
@file[SCI][noinherit][1:20,2:30] extname + ikiparams + section
@file[SCI,2][noinherit][1:20,2:30] extname+extver + ikiparams + section
@file[noinherit] ikiparams
@file[noinherit][1:123,2:234] ikiparams + sections
@file[1:123,2:234] sections
@file[1:123,2:234] sections
mef*.fits[filter?='V'] selection expression
mef*.fits[filter?='V';filter?='B'] selection expressions (OR)
mef*.fits[filter?='V'||filter?='B'] selection expressions (OR)
mef*.fits[gain>0.5&&gain<2.0] selection expressions (AND)
Expressions will evaluate to 'false' if there is an error such as
"keyword not found", meaning that no images will match when one or
more keywords may not be present. Best to use a comma-delimited list
in this case.
Concatenation
@file // foo append
@file* // foo append wildcards
@file // [2] append modifiers
foo // @file prepend
foo // @file* prepend wildcards
foo // @file[2] prepend modifiers
Prior Behavior:
foo // bar.fits ==> foobar.fits
foo.fits // bar ==> foobar.fits
foo // @file1 ==> foosif1.fits,foomef1.fits
@file1 // bar ==> sif1foo.fits,mef1foo.fits
|
