1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
|
Content-type: text/html
<HTML><HEAD><TITLE>Manpage of VODATA</TITLE>
</HEAD><BODY>
<H1>VODATA</H1>
Section: User Commands (1)<BR>Updated: July 2007<BR><A HREF="#index">Index</A>
<A HREF="http://localhost/cgi-bin/man/man2html">Return to Main Contents</A><HR>
<A NAME="lbAB"> </A>
<H2>NAME</H2>
vodata - query and access VO data services
<A NAME="lbAC"> </A>
<H2>SYNOPSIS</H2>
<B>vodata</B> [<I><flags></I>] [ <<I>resource</I>> [[ <<I>objname</I>> [ <<I>sr</I>> ]]] ]
<P>
<B>vodata</B> [<I><flags></I>] [ <<I>resource</I>> [[ <<I>ra</I>> <<I>dec</I>> [ <<I>sr</I>> ]]] ]
<P>
<B>vodata</B> [<I><flags></I>] [ <<I>url</I>> ]
<P>
<A NAME="lbAD"> </A>
<H2>OPTIONS</H2>
The <I>vodata</I> task accepts the following options:
<DL COMPACT>
<DT><B>-h,--help</B>
<DD>
Print a help summary to the terminal and exit. No processing is done
following this flag.
<DT><B>-v,--verbose</B>
<DD>
Verbose output. The output will be more verbose than normal but exactly
what is printed depends on whether other flags are enabled to changed the
basic task behavior.
<DT><B>--vverbose</B>
<DD>
Very-verbose output. Even more output.
<P>
<DT>The following flags control the major behavior of the task, i.e. the type <DD>
of output to present.
<DT><B>-a, --all</B>
<DD>
Perform an action based on all available data. When used as part of a
data query, this flag causes the <<I>resource</I>> argument to be used in
a substring match of Directory <I>ShortName</I> or <I>Identifier</I> fields
to create the actual list of resources to be queried. If the <I>ShortName</I>
of a <I>TABULARSKYSERVICE</I> from Vizier is given, the <<I>resource</I>>
will typically expand to include all tables associated with the paper,
and providing a means to access all of these tables from a single query.
<DT><B>-c, --count</B>
<DD>
Print only a count of the matching records found and do not save any
results. The standard output for the task is to echo some of the input
parameters and print a table of results showing progress and the number
of matching records. If this flag is set, the output written to the
screen will be the same, however the data will not be saved locally.
<DT><B>-g, --get</B>
<DD>
Get the data referenced by the results of a data query. This typically
only applies to Simple Image Access service in which the result of a
query include a column of "<I>access references</I>" to the actual data
that must be resolved separately. Setting this flag will cause all data
references to be resolved by the task once all of the data queries have
been completed.
<P>
Access references are appended to a master "access list" as each query
completes. In general the order in which these are retrieved cannot be
guaranteed. Data downloads can be done in parallel by setting the number
of concurrent max downloads using the <I>--maxdownloads=<N></I> flag, the
default is to download one file at a time. If this flag is followed
with a comma-delimited list of numbers, only those rows in the result
table will be accessed.
<DT><B>-m, --meta</B>
<DD>
Print only the column metadata for the named services. The output will be
a list of the columns return by a data query to the service, but will not
save the actual data. A default position and search size will be used for
the query: In the case of Cone services a negative size is used, for SIAP
services the FORMAT=METADATA flag is used in the query, and for tabular
Vizier services the entire table is accessed. Compliant VO services will
respond quickly with only the column metadata, tabular services may respond
more slowly due to the need to transfer the data. Adding the <I>-v</I> or
<I>--verbose-<N></I> options will increment the VERBOSE level of services
and may return more metadata if available, to access these extra columns
the same level of verbosity must be set during a data query.
<P>
<DT>The following flags specify data query options:<DD>
<DT><B>-b <bpass> or --bandpass=<bpass></B>
<DD>
Constrain the query by bandpass. The argument following the flag must
be one of the allowed bandpass specification string. Setting the flag
will constrain any Directory search used to only those resources where
the spectral coverage matches the given bandpass. Aliases for bandpasses
are allowed, see below.
<DT><B>-i <file></B>
<DD>
Specify a file containing the remaining positional command-line input. The
command line is thought of as having the following components: the options
beginning with a '-' character and their associated arguments, one or more
<<I>resource</I>> names giving the service to invoke, an object name or
position, and an optional query size. The '-i' flag allows everything
except the options to be specified from a file (or the standard input if
the '-' argument is used), creating in effect a means to interactively
specify the e.g. resource/object without restarting the task, or to take
these values from a file or input stream to create multiple independent
queries. If either the resource or object name/position has already been
specified they do not need to be specified again.
<P>
The format for the command file is the same as for the <<I>resource</I>>,
<<I>objname</I>>, <<I>ra</I>> <<I>dec</I>>, <<I>url</I>>, or <<I>sr</I>> described
below and as they would appear on the command line, all input lines are
terminated with a newline, the file or input stream is terminated with an
EOF. An example of how this
may be used would be the using a command file such as:
<P>
<PRE>
2mass-psc m31,m51 0.5
chandra ngc4258,m51 0.25
</PRE>
<P>
The task will process this file as if the two lines had been invoked as
separate commands. The advantage is that this input can be created
dynamically by another task, and we can group resource and object lists
into independent queries. See the <I>Examples</I> below for other uses.
<DT><B>-o <obj|file> or --object=<obj|file></B>
<DD>
Specify the object name to use in a query. Object names are resolved
automatically to J2000 equatorial coordinates. The argument to this
flag may be the name of a single object, a comma-delimited list of
object names, the name of a file containing object names, or the
reserved value '-'.
<P>
The reserved value '-' tells the task to take this information from the
standard input, processing doesn't begin until the object list has been
fully read.
<DT><B>-p <pos|file> or --pos=<pos|file></B>
<DD>
Specify the position to use in a query. Positions are composed of two
values assumed to be equatorial J2000 coordinates.
Values specified as a floating-point decimal are assumed to be in units
of degrees, sexagesimal values may also be used and are assumed to be
equatorial RA and Dec. If the <I><pos></I> arg is used only one set of
coordinates may be given on the command-line and must be delimited by a comma,
however the argument may also be the name of a file containing coordinates
to be processed, or the reserved value '-'.
<P>
The reserved value '-' tells the task to take this information from the
standard input, processing doesn't begin until the position list has been
fully read.
<DT><B>-r <radius> or --sr=<radius>[<units>]</B>
<DD>
Set the search radius. The default search size is 0.1 degrees unless
specified on the command-line and argument are assumed to be
in degrees, setting the size in other units is permitted using the
<I>-sr</I> flag. To specify the <units> for the <I>--sr</I> flag, the
argument should be suffixed by an 's' to specify arcseconds, an 'm' to set
arcminutes, and 'd' to set the size in degrees. By default, all
queries will be done using the same search size. Variable search
sizes accomplished using the '-i' flag described above.
<DT><B>-s <service> or --svc=<service></B>
<DD>
Specify the service or url to invoke. In most cases the service, i.e.
the <<I>resource</I>> argument will be taken from the commandline based on
it's position. The exception is when the user want to specify a service
URL directly (e.g. to test a local service) because it isn't known to the
Directory, or to use the reserved values '-' or 'any'. Use of '-' tells
the task to read the service list from the standard input; use of
the word 'any' is a means to telling the task to dynamically create the
resource list from other options (e.g. "any image service" by using the
<DT><B>-t <type> or --type=<type></B>
<DD>
Constrain the query by service type. The list of allowed service types
is given below. The actual string used in a Directory resource record
may be used if known, otherwise common use is to specify 'image' to
access Simple Image Access (SIAP) services, 'catalog' for Cone searches,
or 'table' for Vizier tabular data.
<P>
<DT>The following flags are specific to the writing of HTML or KML files:<DD>
<DT><B>--webnoborder</B>
<DD>
Disable the shaded border drawn around an HTML table.
<DT><B>--webnocolor</B>
<DD>
Disable the coloring for an HTML table.
<DT><B>--webnoheader</B>
<DD>
Disable the HTML page header written to the output file.
<P>
<DT><B>--kmlmax=<N></B>
<DD>
Specify the max number of placemarks to write. The default is 50, ordering
is not guaranteed. Setting the sampling will automatically increase the
maximum number of results returned.
<DT><B>--kmlsample=<N></B>
<DD>
Specify the sampling of the result to be every <I><N></I> rows. The
default is to write all rows to the output file. If set, this value will
be used as a multiplier for the max number of placemarks automatically.
<DT><B>--kmlgroup=object </B>
<DD>
Group the results of a multi-resource/multi-object query into a single
hierarchical KML file grouped by the object or position index (default);
<DT><B>--kmlgroup=service </B>
<DD>
Group the results of a multi-resource/multi-object query into a single
hierarchical KML file grouped by the service name.
<DT><B>--kmlgroup=both </B>
<DD>
Groups the results of a multi-resource/multi-object query into a single
hierarchical KML file. The two top-level folders will be 'By Source' and
<DT><B>--kmlnolabel</B>
<DD>
Disable the labelling of placemarks. By default, the ID_MAIN ucd for
each point will be used as a label.
<DT><B>--kmlnoregion</B>
<DD>
Disable the drawing of the region bounding box in a KML file.
<DT><B>--kmlnoverbose</B>
<DD>
Disable the writing of verbose information to the KML placemarks. By
default, each placemark will contain all information from that result.
<P>
<P>
<DT>Input Options:<DD>
<DT><B>--cols=</B><I>col_str</I>
<DD>
Use columns specified in <I>col_str</I> to read the <I>ra</I>, <I>dec</I>
and <I>id</I> values respectively. <I>col_str</I> is a comma-delimited
list where the <I>id</I> column is optional and will not be used if not
present as the third element in the list. Other columns may be given as
a single integer or as a range of the form <I>start</I>-<I>end</I> indicating
the values in the <I>start</I> thru <I>end</I> columns should be combined
into a single value.
<DT><B>-d, --delim=</B><I>delim</I>
<DD>
Use the <I>delim</I> as the input table delimiter. By default, a space, tab,
comma, vertical bar ('|'), or semicolon may be used as a delimiter for the
input table. If no explicit delimiter is specified, the first occurance of
any one of these will be used. The reserved words <I>comma</I>,
<I>space</I>, <I>tab</I>, or <I>bar</I> may be used in place of a specific
character.
<P>
<DT><B>--ecols=</B><I>col_str</I>
<DT><B>--ecols=</B><I>col_str</I>
<DD>
Use the explicit columns specified in <I>col_str</I> in the input table. This
option should only be used with formatted text tables where the desired
values will always be in the same columns of the file. Note that 'column' in
this case refers to a specific character column in a text file. Columns
may be a single integer or a range, and is a comma-delimited list as with
the <I>--cols</I> option.
<DT><B>-f, --force</B>
<DD>
Force the input table to be used even in the number of columns varies on
each line. The assumption here is that any variation (e.g empty columns)
occurs after the <I>ra</I>, <I>dec</I> and <I>id</I> columns in the table.
<DT><B>--hskip=</B><I><N></I>
<DD>
Skip <I><N></I> header lines in the input file. This option is only needed
when the lines to be skipped do not begin with the normal '#' comment
character.
<DT><B>--nlines=</B><I><N></I>
<DD>
Use only <I><N></I> lines of the input table.
<DT><B>--sample=</B><I><N></I>
<DD>
Sample the table every <I><N></I> lines. Setting the sample will not affect
the <I>nlines</I> used.
<P>
<P>
<DT>Output Options:<DD>
<DT><B>-1,--one</B>
<DD>
Save the results into a single file regardless of format. This option will
be set automatically if the output is being written to the standard output.
If the output format is something other than KML or XML, all results will
be concatenated into individual files of the form
"<svc>_<pid>.<extn>" so that each file will contain the object results from
each service where the columns will be the same.
<DT><B>-A,--ascii</B>
<DD>
Save the results as a whitespace delimited ascii table. If an output file
is created it will have a ".txt" extension appended automatically.
<DT><B>-C,--csv</B>
<DD>
Save the results as a comma-separated-value (CSV) table. If an output file
is created it will have a ".csv" extension appended automatically.
<DT><B>-H,--html</B>
<DD>
Save the results as an HTML table. If an output file
is created it will have a ".html" extension appended automatically. See
above for the <I>--webnoheader</I> option that can be used to disable the HTML page
header.
<DT><B>-I,--inventory</B>
<DD>
Query the <I>Inventory Service</I> rather than the data services directly.
This will return simply a count of the results found, but when presented
with a table of resources and sources can be used to do a simple
crossmatch of the sources found in the catalogs available through the
service.
<DT><B>-K,--KML</B>
<DD>
Save the results as a Google Earth/Sky KML placemark file. If an output file
is created it will have a ".kml" extension appended automatically. See
above for additional options that control the content of the file.
<DT><B>-R, -V or --raw, --votable</B>
<DD>
Save the results as a raw VOTable. If an output file is created it will
have a ".vot" extension appended automatically.
<DT><B>-T,--tsv</B>
<DD>
Save the results as a tab-separated-value (TSV) table. If an output file
is created it will have a ".tsv" extension appended automatically.
<DT><B>-O <root> or --output=<root></B>
<DD>
Set the root of the output name. The reserved value '-' tells the task to
write to the standard output.
<DT><B>-X,--xml</B>
<DD>
Save the results wrapped XML file of the raw VOTable results. If an output
file is created it will have a ".xml" extension appended automatically. The
XML document will gather all the individual VOTable result files to a single
XML document, where each entry is wrapped by the element <I><VOTABLE_ENTRY></I>.
There will be three attributes: <I>svc</I> will be the data service name,
<I>obj</I> will be the object name (if supplied), and the <I>index</I> attribute
giving an index into the results. This index list is created by looping over
each service, and for each service, looping over the object/position list.
<DT><B>-e,--extract</B>
<DD>
Extract positional and access information to extra output files. By
default both files will be written, using <I>--extract=pos</I> will write only the
positional information file, using <I>--extract=urls</I> will write the access URLs only.
Access URLs are written one-per-line to a file with the same root name as
the main output but with a ".urls" extension; Positional information is
written to a file with a ".pos" extension and will contain three columns
made up of the identifier (the column with the <I>ID_MAIN</I> ucd), RA and
Dec (the <I>POS_EQ_RA_MAIN</I> and <I>POS_EQ_DEC_MAIN</I> ucd columns
respectively). If these ucds appear more than once in a table, the first
occurrance will be used.
<P>
Additionally, the <I>--extract=headers</I> and <I>--extract=kml</I> flags may be to to specify the
HTML and KML output be written to files with ".html" and ".kml"
extensions respectively. The <I>--extract=KML</I> flag will cause multi-resource
and/or multi-object queries to be collected into a single KML file.
The format-specific <I>--kml<opt></I> and <I>--web<opt></I> flags will apply to these files.
A <I>--extract=xml</I> flag will force the output format to be raw VOTable and gather
the results to a single XML document (see the <I>-X</I> option).
<P>
Note that the URLs file can be used to later access the data (perhaps
after sub-selecting from the table based on some criteria) by calling
the task again using the filename as the only argument.
<DT><B>-n,--nosave</B>
<DD>
If enabled, this flag tells the task not to save results to local disk.
Status and result information will continue to be printed to the screen,
but no data are saved to disk.
<DT><B>-q,--quiet</B>
<DD>
Quiet mode. Suppress any extraneous output and warning messages.
<DT><B>-u,--url</B>
<DD>
Force the specified URL to be downloaded.
<P>
<P>
<DT>DAL2 Query Options:<DD>
<DT><B>--band=</B><I>band_string</I>
<DD>
The spectral bandpass is given in range-list format. For a numerical
bandpass the units are wavelength in vacuum in units of meters.
The spectral rest frame may optionally be qualified as either
<I>source</I> or observer, specified as a range-list
qualifier. Bandpass names are often not useful for spectra (they are
probably more useful for image or time series data) but there are cases where
they are useful for spectra, for example for a velocity spectrum of a
specific emission line.
<DT><B>--time=</B><I>time_string</I>
<DD>
The time coverage (epoch) specified in range-list form as defined in section
8.7.2, in ISO 8601 format. If the time system used is not specified UTC is
assumed. The value specified may be a single value or an open or closed
range. If a single value is specified it matches any spectrum for which the
time coverage includes the specified value. If a two valued range is given, a
dataset matches if any portion of it overlaps the given temporal region.
<P>
<P>
<P>
</DL>
<A NAME="lbAE"> </A>
<H2>DESCRIPTION</H2>
The <B>vodata</B> task allows a user to query and access VO data for multiple
resources and objects from a desktop or scripting environment. By design,
the task interface is meant to provide the following features:
<P>
<DL COMPACT>
<DT><B>-</B>
<DD>
Resources (i.e. data services) may be referred to using a more familiar
<I>ShortName</I> designation, or an IVO identifier, either of which will be
resolved to a specific <I>ServiceURL</I> internally using the Directory.
<DT><B>-</B>
<DD>
Object names may be used to specify the location of a data query, the
position will be resolved internally using the <I>Sesame</I> web service.
<DT><B>-</B>
<DD>
Output files may be created in a variety of common formats easily manipulated
with other desktop tools.
<DT><B>-</B>
<DD>
Multiple resources and objects shall be queried in parallel when possible to
optimize the task.
<DT><B>-</B>
<DD>
Data referenced in a query response should be accessible by the task
automatically.
<DT><B>-</B>
<DD>
The command-line interface should be as friendly and as flexible as possible
to allow the task to be used in multiple ways.
<P>
</DL>
<P>
The task should quickly become familiar to users and is meant operate in
concert with the <B>vodirectory</B> and <B>vosesame</B> tasks to allow novice
users to begin to explore for data resources to be used in the final query.
Some of the flexibility of the task is shown in the Examples section below.
Major concepts of the task are detailed below as well.
<P>
<BR>
<A NAME="lbAF"> </A>
<H3>Argument Parsing</H3>
<P>
The meaning of the various command-line arguments is detailed below:
<P>
<DL COMPACT>
<DT><B></B><I><resource></I>
<DD>
The <I>ShortName</I> or <I>Identifier</I> of a data resource to be queried, a
comma-delimited list of either, or the name of a file containing either.
These names will be resolved to a data service URL using the Directory. The
<I>-s</I> option may be used to specify a non-registered <I>ServiceURL</I> that
the task may use, however the <I>-t</I> option is then also required to
specify the type of service.
<DT><B></B><I><objname></I>
<DD>
The name of an object, a comma-delimited list of object names, or the name of
a file containing object names. The coordinates of each object will be
resolved to a position prior to processing using the <I>Sesame</I> name
resolver service. An error will be returned if an object name cannot be
resolved, and that object will be skipped.
<DT><B></B><I><ra> <dec></I>
<DD>
The J2000 equatorial RA and Dec position to the searched. Values given as
floating point values are assumed to be in decimal degrees, sexagesimal
values are assumed to be equatorial RA/Dec positions. Sexagesimal values may
be of the form <I>hh:mm:ss.s</I> or <I>hh:mm.m</I> for RA, or
<I>dd:mm:ss.s</I> or <I>dd:mm.m</I> for Dec. Only one coordinate pair may be
specified on the commandline.
<DT><B></B><I><sr></I>
<DD>
The search size for the data query specified in decimal degrees. The default
size of 0.1 degrees will be used if this is not specified on the command line.
The <I>-rs</I> and
<I>-rm</I> options may be used specify the size in arc seconds and minutes
respectively. The <I>-i</I> option may be used to specify command-line input
options, where each command-line can include a different value for the search
size, otherwise only one value is allowed.
<DT><B></B><I><url></I>
<DD>
A single URL, or the name of a file containing URLs listed one per line.
<P>
<P>
</DL>
<A NAME="lbAG"> </A>
<H3>Multi-Thread and Multi-Process Data Querying</H3>
<P>
All data queries require at least one <I>resource</I> and one <I>source</I>
to be successful. The <I>resource</I> defines a specific data service to be
queried, and the <I>source</I> is either an explicit position on the sky or
the name of an object that can be resolved to a position. Additional
parameters to the query are used to specify other options, but in essence
each data query is translated to a single URL that must be accessed by the
client task. In a complex query, lists of resource and/or objects create a
potentially large matrix of queries that must be made (i.e.
<I>N-services</I> by <I>N-objects</I> in total). Because a large fraction of
the time spent in waiting for a query to finish is in waiting for the
server to respond, we are able to run multiple queries simultaneously
without saturating our network bandwidth in most cases.
<P>
The <B>vodata</B> task will parallelize the list of services to be queried
by running a separate processing <I>thread</I> (i.e. a lightweight process
running in parallel within the main application) for each of the services
to be called. This allows queries to different servers to be run in
parallel, and since these servers will often reside on multiple machines
the client won't impact any one data provider too badly. In addition,
the list of objects to be queried at each service will be broken up into
multiple child processes and called simultaneously. This allows, for
example, 10 objects to be queried from 3 services (a total of 30 queries)
simultaneously.
<P>
The <I>--maxthreads=<N>mt</I> option can be used to set the max number of threads to be
created for processing the resource list (the default is 20). If the
resource list is larger than this value, the list will be processed with no
more than the max number running at any one time until all resources have
been queried. Similarly, the <I>--maxprocs=<N></I> option can be used to set the
number of child processes to be created to process the object list (the
default is 10). When setting these values it is important to remember that
the total number of <I>potential</I> processes running on your machine will
be the product of these to values. The default values were empirically found
to work reasonably well on most modern machines.
<P>
Additionally, it is worth considering the potential strain that can be put
on data providers' machines before changing these settings. The large
majority of Cone services for example come from a single server at HEASARC
and overloading the server with hundreds of requests to multiple resources
it provides may result in a failed request and what would appear to be no
data. One should consider using the <I>-i</I> flag as a means to query a
large object list against a resource list such that only the object
processing is parallelized and the server load is minimized (See the example
below).
<P>
<P>
<A NAME="lbAH"> </A>
<H3>Output Filename Generation</H3>
<P>
The <I>-O</I> option may be used to specify the root part of output files
created by a data query. However, to guarantee that a multi-service, and/or
multi-object query doesn't overwrite a single output file, the filename root
will also include the <I>pid</I> (process ID) of the task that created it.
For a single service and object query no <I>pid</I> will be used as part of
the filename. This scheme guarantees unique output files across the various
processing scenarios, with similar root names for multiple files associated
with a specific query.
<P>
Output tables may be created in a number of formats and will likewise have
extensions indicating the table type. The <I>-e</I>/<I>--extract</I> option
may create additional files for each query, and the <I>-g</I>/<I>--get</I>
option to access data will similarly create additional files. The structure
of an output filename is:
<P>
<PRE>
<I><root>[_<pid>].<extn></I>
</PRE>
The meaning of <I><pid></I> and <I><extn></I> have been discussed above. If
the <I>-O</I> option was set then the <I><root></I> part of the name will
simply be the argument given to set the root name. Otherwise, the <I><root></I>
element will be of the form:
<PRE>
<I><svc>_<type>_<objname></I>
<I><svc>_<type>_<index></I>
</PRE>
The <I><svc></I> is derived from the service name used, the <I><type></I> is
a single-character code to indicate the type of service used ('I' for image,
object name or the index in a list of positions of no object was specified.
<P>
<A NAME="lbAI"> </A>
<H3>Verbosity</H3>
<P>
The <I>-v</I> and <I>-vv</I> options serve a dual purpose: within the task they
set the level of output verbosity in terms of what is reported during
processing (Similarly, the <I>-q</I> option can be used to turn off output
reporting entirely). These flags will however also increase the value of the
<B>VERBOSE</B> parameter sent to services during a data query. The default
value is at least 1, with the highest level being 3. Using the <I>-v</I> flag
sets <B>VERBOSE=2</B> and <I>-vv</I> sets <B>VERBOSE=3</B>.
<P>
The VERBOSE level can be important in accessing result columns that may only
be returned at the highest level. When using the <I>-m</I>
(or <I>--meta</I>) flag to print
the column metadata, the verbose options will also affect the results and it
is important that the same verbosity be set when doing the actual data
query and access.
<P>
<A NAME="lbAJ"> </A>
<H3>Bandpass and Service Type Aliases</H3>
<P>
The type constraint (<I>-t</I> or <I>--type</I>) accepts only the
following arguments:
<PRE>
catalog Cone search services
image Simple Image Access services
spectra Simple Spectral Access services
table Vizier services
<literal> ResourceType from Directory record
</PRE>
<P>
The bandpass constraint (<I>-b</I> or <I>--bandpass</I>) accepts only
the following arguments:
<PRE>
Radio Millimeter Infrared (IR)
Optical Ultraviolet (UV) X-Ray (xray)
Gamma-Ray (GR)
</PRE>
Values in parenthese are acceptable aliases. All matches are cases
insensitive.
<P>
<P>
<A NAME="lbAK"> </A>
<H3>Range-List Parameters</H3>
<P>
Some parameters (for example BAND and TIME) may allow a parameter value to be
specified as a numeric range. Such range-valued parameters use the forward
slash (<I>/</I>) character as the separator between elements of the
range specification (as in the ISO 8601 date specification after which this
convention is patterned). For example, <I>5E-7/8E-7</I> would
specify a range consisting of all values from 5E-7 to 8E-7, inclusive. If a
third field is specified it is a step size for traversing the indicated
range. If a parameter permits a step size the semantics of the step size are
defined by the specific parameter.
<P>
An open range may be specified by omitting either range value. If the first
value is omitted the range is open toward lower values. If the second value
is omitted the range is open toward higher values. Omitting both values
indicates an infinite range which accepts all values. For example,
<I>/5</I> is an open range which accepts all values less than or
equal to 5. To specify all values less than 5, <I>/4</I> would be
used (for an integer valued range). Range values are limited to numeric
values or ISO dates.
<P>
A list may be qualified by appending the character <I>;</I> (semicolon)
followed by a qualifier string. For example <I>1E-7/3E-6;source</I> could
specify a spectral bandpass in the rest frame of the source. List and
range syntax may be combined, e.g., to indicate a list of scalar or
range-valued parameter values. Such a range list may be ordered or
unordered, and may contain either numeric or string data. An ordered
list is one which requires values to be processed in a specified order,
and to ensure this the range list is sorted or ordered by the service as
necessary before being used. It is the responsibility of the service to
sort an ordered range list, hence the client can input ranges or range
values in any order for an ordered range list and the result will be
the same. The sequence in which items in an unordered list occur on the
other hand is significant, as since there is no intrinsic ordering for
the list which can be enforced by the service, items will be processed
by the service in the order they are input by the client.
<P>
TIME and BAND are typical examples of ordered range lists. Since a dataset
matches the query if it contains data in any of the specified ranges,
logically it does not matter in what order the ranges are given, or whether
the first element of a range is less than the second, or whether ranges
overlap; the result should be the same in all cases. Hence the range list has
an intrinsic ordering irrespective of how ranges are input. Unless otherwise
specified in the definition of a given parameter, range lists are assumed to
be ordered.
<P>
<BR>
<P>
<A NAME="lbAL"> </A>
<H2>VOCLIENT DAEMON PROCESSING</H2>
All VO-CLI tasks are built upon the VOClient interface an rely on a
separate <I>voclientd</I> process to provide the VO functionality. The
voclientd task is distributed as part of VO-CLI and will be started
automatically by each task if it is not already running. If problems are
encountered, you may want to manually start the voclientd in a separate
window before running the task so you can monitor the output for error
messages.
<P>
<A NAME="lbAM"> </A>
<H2>RESOURCE CACHING</H2>
Directory resolution is a common activity of VO-CLI tasks and so results
will be cached in the $HOME/.voclient/cache/regResolver directory based on
the search term, service type and bandpass parameters. Defining the
<I>VOC_NO_CACHE</I> environment variable will cause the task to ignore the
cache.
<P>
<A NAME="lbAN"> </A>
<H2>EXAMPLES</H2>
<P>
<DL COMPACT>
<DT>1)<DD>
Query the GSC 2.3 catalog for stars a) within the 0.1 degree
default search size around NGC 1234: b) around all positions
contained in file 'pos.txt': c) for the list of objects given
on the command line: d) query a list of services for a list
of positions: e) print a count of results that would be returned
from 3 services for each position in a file:
<PRE>
% vodata gsc2.3 ngc1234 (a)
% vodata gsc2.3 pos.txt (b)
% vodata gsc2.3 m31,m51,m93 (c)
% vodata svcs.txt pos.txt (d)
% vodata hst,chandra,gsc2.3 pos.txt (e)
</PRE>
<P>
<DT>2)<DD>
Query all (142) image services having data of the subdwarf galaxy
IC 10, print a count of the results only:
<PRE>
% vodata -c -t image any IC10
% vodata --count --type=image any IC10
</PRE>
Note that we use the reserved word '<I>any</I>' for the service name and
constrain by image type. The task will automatically query the Directory to
create the list of services to be queried.
<P>
<DT>3)<DD>
Print a count of X-ray catalog data around Abell2712:
<PRE>
% vodata -c -t catalog -b x-ray any abell2712
% vodata --count --type=catalog --bandpass=x-ray any abell2712
</PRE>
In this case we constrain both the service type as well as the spectral
coverage published for the resource in the Directory. We use the reserved
<I>
any</I>' service name to query multiple services and use the '<I>-c</I>'
flag to print a count without saving results. The object name is resolved
to coordinates internally. (Note: this example may take a while to run).
<P>
<P>
<DT>4)<DD>
Print the column metadata returned by the RC3 catalog service:
<PRE>
% vodata --meta rc3 or vodata -m rc3
</PRE>
The output will print the result using the default VERBOSE level, adding
the <I>-v</I> will set the query parameter VERBOSE=2, adding <I>-vv</I> will
set VERBOSE=3 (to print all available columns). When accessing data the
same <I>-v</I> flags will be required to retrieve columns at that
<I>VERBOSE</I> level.
<P>
<P>
<DT>5)<DD>
Use the Directory to query for resources using the search terms
"cooling flow". Upon examining the output the user notices a
Vizier paper titled "<I>Cooling Flows in 207 clusters of Galaxies</I>"
that looks interesting. Use the <B>vodata</B> task to download all
tables associated with this paper, save tables in the default
CSV format:
<PRE>
% vodirectory cooling flow
% vodata -O white97 -all J/MNRAS/292/419/
% vodata --output=white97 --all J/MNRAS/292/419/
</PRE>
All 7 tables will be written to the current directory to files
having a root name 'white97' (chosen based on the author and
publication date).
<P>
<P>
<DT>6)<DD>
Find a suitable XMM image service, get a (brief) count of the
XMM images available for 3c273, and if there aren't too many,
download the images and save the extracted access URLs:
<PRE>
% vodirectory -rv -t image xmm
ShortName ResourceType Title
------------------------------------------------------....
XMM-Newton SIAP/ARCHIVE XMM-Newton Archive ....
% vodata -cq xmm-newton 3c273
xmm-newton 27 I XMM-Newton Archive ....
% vodata --count --quiet xmm-newton 3c273
xmm-newton 27 I XMM-Newton Archive ....
% vodata --get xmm-newton 3c273
.... will query and download 27 images.
</PRE>
<P>
<DT>7)<DD>
Query for the images available from 2MASS at a given position,
extract the positions and service URLs to separate files:
<PRE>
% vodata -e -O 2mass -t image 2mass 12:34:56.7 -23:12:45.2
% vodata -e --output=2mass --type=image 2mass 12:34:56.7 -23:12:45.2
</PRE>
The query produces files with the root name '2mass', and exten- sions of
"<I>.csv</I>" (the main response), "<I>.pos</I>" (the extracted pos- itions),
and "<I>.urls</I>" (the access references). The user inspects the files and
notices that the references return both FITS and HTML files, but she only
wants the FITS image date and uses <B>vodata</B> to download only those:
<PRE>
% grep fits 2mass_I_001_15998.urls > images.txt
% vodata images.txt
or
% grep fits 2mass_I_001_15998.urls | vodata -i -
</PRE>
In both cases we pass URLs to the task which bypasses the query and directly
access the images. However, in the first case we process the entire list
and are able to take advantage of the <I>-maxdownloads=<N></I> option to
increase the number of simultaneous downloads. In the second case, the
<I>-i</I> flag causes the task to interpret each line of the input stream
as a separate command and so the data are always downloaded one at a time
(which is the default download behavior anyway).
<P>
<DT>8)<DD>
Use <B>vodata</B> as a test client for a locally-installed SIAP service:
<PRE>
% vodata -t image -s <A HREF="http://localhost/siap.pl">http://localhost/siap.pl</A> 180.0 0.0
% vodata --type=image --svc=<A HREF="http://localhost/siap.pl">http://localhost/siap.pl</A> 180.0 0.0
</PRE>
In this case we force the ServiceURL using the '<I>-s</I>' flag, but since we
can't do a Directory query to discover what type of service this is, we must
use the '<I>-t</I>' flag to indicate it is an image service.
<P>
<P>
<DT>9)<DD>
Create a local table containing the Abell catalog. Begin with a
Directory query to find likely services using the <B>vodirectory</B> task, print
a verbose description of each resource and page the results with <I>less</I>:
<PRE>
% vodirectory -v -v --type=catalog abell | less
</PRE>
The verbose results indicate a number of services with differing
requirements for what is included. We decide to use the service from
HEASARC since it contains southern hemisphere data and constraints we are
interested in. Try an all-sky query to retrieve the entire catalog, use
the service identifier to be specific about where we want to go:
<PRE>
% vodata -e <A HREF="ivo://nasa.heasarc/abell">ivo://nasa.heasarc/abell</A> 0.0 0.0 180.0
% vodata --extract <A HREF="ivo://nasa.heasarc/abell">ivo://nasa.heasarc/abell</A> 0.0 0.0 180.0
</PRE>
We use the '<I>-e</I>' flag to extract the positions to a separate file with
a "<I>.pos</I>" extension so that we can use these in later queries.
However, the <I>.pos</I> file additionally contains the ID from the original
catalog in column 1. Strip this column so we're left with only RA and DEC
and query for Chandra observations at each position:
<PRE>
% cut -c6- *.pos | vodata <A HREF="ivo://nasa.heasarc/chanmaster">ivo://nasa.heasarc/chanmaster</A> -p -
% cut -c6- *.pos | vodata <A HREF="ivo://nasa.heasarc/chanmaster">ivo://nasa.heasarc/chanmaster</A> --pos=-
</PRE>
Here we used the unix '<I>cut</I>' utility to remove the first column and
pipe the resulting positions to the task, using the '<I>-p -</I>' option to
indicate positions should be read from stding, and the IVO identifier of
the Chandra observation master log we also discovered from the Directory.
<P>
<P>
<DT>10)<DD>
Interactively query for a count of Chandra observations of Messier
objects:
<PRE>
% vodata -cq chandra -i -
m31
chandra 335 I Chandra X-Ray Observatory Data Archive
: : : : :
</PRE>
Note that by using the '<I>-i</I>' flag we execute each query as if we'd put
the object name on the command line. Using the '<I>-o</I>' flag would
instead read all of the objects from the stdin but then execute the queries
in parallel following an EOF to terminate the input.
<P>
<P>
<DT>11)<DD>
Use the <I>STILTS</I> task '<I>tpipe</I>' to select rows from a VOTable
of QSOs (made with an earlier query) where the redshift is > 0.2. Output
only the positions and pipe this to <B>vodata</B> to generate a new query to
see whether HST has observed any of these objects:
<PRE>
% stilts tpipe ifmt=votable qso_survey.vot
cmd='select "Z > 0.2"'
cmd='keepcols "RA DEC"' | vodata -p - hstpaec
</PRE>
Note that we use the '<I>-p -</I>' flag to tell the task to take it's list of
positions from the piped input. The positions are used to query the HST
Planned and Archived Exposure Catalog (<I>hstpaec</I>)
<P>
<P>
<DT>12)<DD>
The user has a task called '<I>wcsinfo</I>' that takes a list of images
and outputs a text table containing the plate center and size in degrees.
Use this task as part of a query for 2MASS point sources contained in each
image:
<PRE>
% wcsinfo *.fits | vodata 2mass-psc -i -
</PRE>
Here we specify the desired service (<I>2mass-psc</I>) on the commandline as
usual, and allow the remainder of the args (i.e. the position and search
size) to be read from the stdin. This allows for variable search sizes but
processes the positions serially. If the sizes are all the same (say 25
arcmin), multiple queries can be done simultaneously using just a position
file, e.g.
<PRE>
% wcsinfo -pos_only *.fits > centers.txt
% vodata --sr=25m 2mass-psc centers.txt
</PRE>
<P>
<P>
<P>
<DT>13)<DD>
Query a large list of objects against a number of ASCA resources.
Because the ASCA catalogs are served by the same machine, use the <I>'-i'</I>
option and a command file to process only each resource sequentially while
still parallelizing the object lists:
<P>
<PRE>
% cat cmds.txt
ASCA survey.tbl
ASCA\ GIS survey.tbl
ASCA\ GPS survey.tbl
: :
% vodata -i cmds.txt
</PRE>
</DL>
<P>
Note that we've needed to escape the space in the resource name in some
cases. To avoid this, use of the IVO identifier for each resource is usually
preferred.
<P>
<DL COMPACT>
<DT>14)<DD>
Query the VO for GALEX data of M51. Because the <I>ShortName</I> GALEX
is not unique, we must either specify the IVO identifier of a
specific service to query, or if we're interested in results from all
supported data services with <I>galex</I> in the name:
<PRE>
% vodata -a galex M51
% vodata --all galex M51
</PRE>
The results come from the Cone and SIAP services both called <I>GALEX</I>,
as well as an additional SIAP service called 'GALEX_Atlas'. Note that the
service names are case insensitive in either case.
<P>
<P>
<DT>15)<DD>
Process a list of hundreds of positions against the GSC2.3 catalog:
<PRE>
% vodata gsc2.3 positions.txt
</PRE>
<P>
<P>
<DT>16)<DD>
Process a list of hundreds of positions against the GSC2.3 catalog, assume
that the input table has a 5 line header of text and three columns
(id,ra,dec). Note that the <I>cols</I> option requires the optional id
be specified last:
<PRE>
% vodata --cols=2,3,1 --hskip=5 gsc2.3 positions.txt
</PRE>
<P>
</DL>
<A NAME="lbAO"> </A>
<H2>BUGS</H2>
Some services don't repond properly to the metadata query and will print
a "no attributes found" message.
<P>
<A NAME="lbAP"> </A>
<H2>TODO</H2>
<P>
- Additional command-line options should be allowed to be specified in a
command file.
<P>
- Support for SSAP, SAMP, TAP
<P>
<P>
<A NAME="lbAQ"> </A>
<H2>Revision History</H2>
June 2007 - This task is new.
<A NAME="lbAR"> </A>
<H2>Author</H2>
Michael Fitzpatrick (<A HREF="mailto:fitz@noao.edu">fitz@noao.edu</A>), June 2007
<A NAME="lbAS"> </A>
<H2>SEE ALSO</H2>
<A HREF="http://localhost/cgi-bin/man/man2html?1+voclient">voclient</A>(1), <A HREF="http://localhost/cgi-bin/man/man2html?1+voclientd">voclientd</A>(1), <A HREF="http://localhost/cgi-bin/man/man2html?1+vosesame">vosesame</A>(1), <A HREF="http://localhost/cgi-bin/man/man2html?1+vodirectory">vodirectory</A>(1)
<P>
<HR>
<A NAME="index"> </A><H2>Index</H2>
<DL>
<DT><A HREF="#lbAB">NAME</A><DD>
<DT><A HREF="#lbAC">SYNOPSIS</A><DD>
<DT><A HREF="#lbAD">OPTIONS</A><DD>
<DT><A HREF="#lbAE">DESCRIPTION</A><DD>
<DL>
<DT><A HREF="#lbAF">Argument Parsing</A><DD>
<DT><A HREF="#lbAG">Multi-Thread and Multi-Process Data Querying</A><DD>
<DT><A HREF="#lbAH">Output Filename Generation</A><DD>
<DT><A HREF="#lbAI">Verbosity</A><DD>
<DT><A HREF="#lbAJ">Bandpass and Service Type Aliases</A><DD>
<DT><A HREF="#lbAK">Range-List Parameters</A><DD>
</DL>
<DT><A HREF="#lbAL">VOCLIENT DAEMON PROCESSING</A><DD>
<DT><A HREF="#lbAM">RESOURCE CACHING</A><DD>
<DT><A HREF="#lbAN">EXAMPLES</A><DD>
<DT><A HREF="#lbAO">BUGS</A><DD>
<DT><A HREF="#lbAP">TODO</A><DD>
<DT><A HREF="#lbAQ">Revision History</A><DD>
<DT><A HREF="#lbAR">Author</A><DD>
<DT><A HREF="#lbAS">SEE ALSO</A><DD>
</DL>
<HR>
This document was created by
<A HREF="http://localhost/cgi-bin/man/man2html">man2html</A>,
using the manual pages.<BR>
Time: 20:46:31 GMT, May 21, 2009
</BODY>
</HTML>
|
