diff options
Diffstat (limited to 'vendor/voclient/doc/OLD/vodata.man')
| -rw-r--r-- | vendor/voclient/doc/OLD/vodata.man | 986 |
1 files changed, 986 insertions, 0 deletions
diff --git a/vendor/voclient/doc/OLD/vodata.man b/vendor/voclient/doc/OLD/vodata.man new file mode 100644 index 00000000..2f27f606 --- /dev/null +++ b/vendor/voclient/doc/OLD/vodata.man @@ -0,0 +1,986 @@ +.\" @(#)vodata.1 1.0 June-07 MJF +.TH VODATA 1 "July 2007" "NVO VO-CLI Project" +.SH NAME +vodata \- query and access VO data services +.SH SYNOPSIS +\fBvodata\fP [\fI<flags>\fP] [ <\fIresource\fP> [[ <\fIobjname\fP> [ <\fIsr\fP> ]]] ] + +\fBvodata\fP [\fI<flags>\fP] [ <\fIresource\fP> [[ <\fIra\fP> <\fIdec\fP> [ <\fIsr\fP> ]]] ] + +\fBvodata\fP [\fI<flags>\fP] [ <\fIurl\fP> ] + +.SH OPTIONS +The \fIvodata\fP task accepts the following options: +.TP 8 +.B \-h,\--help +Print a help summary to the terminal and exit. No processing is done +following this flag. +.TP 8 +.B \-v,\--verbose +Verbose output. The output will be more verbose than normal but exactly +what is printed depends on whether other flags are enabled to changed the +basic task behavior. +.TP 8 +.B \--vverbose +Very-verbose output. Even more output. + +.TP 0 +The following flags control the major behavior of the task, i.e. the type +of output to present. +.TP 8 +.B \-a, --all +Perform an action based on all available data. When used as part of a +data query, this flag causes the <\fIresource\fP> argument to be used in +a substring match of Directory \fIShortName\fR or \fIIdentifier\fR fields +to create the actual list of resources to be queried. If the \fIShortName\fP +of a \fITABULARSKYSERVICE\fP from Vizier is given, the <\fIresource\fP> +will typically expand to include all tables associated with the paper, +and providing a means to access all of these tables from a single query. +.TP 8 +.B \-c, --count +Print only a count of the matching records found and do not save any +results. The standard output for the task is to echo some of the input +parameters and print a table of results showing progress and the number +of matching records. If this flag is set, the output written to the +screen will be the same, however the data will not be saved locally. +.TP 8 +.B \-g, --get +Get the data referenced by the results of a data query. This typically +only applies to Simple Image Access service in which the result of a +query include a column of "\fIaccess references\fP" to the actual data +that must be resolved separately. Setting this flag will cause all data +references to be resolved by the task once all of the data queries have +been completed. + +Access references are appended to a master "access list" as each query +completes. In general the order in which these are retrieved cannot be +guaranteed. Data downloads can be done in parallel by setting the number +of concurrent max downloads using the \fI--maxdownloads=<N>\fP flag, the +default is to download one file at a time. If this flag is followed +with a comma-delimited list of numbers, only those rows in the result +table will be accessed. +.TP 8 +.B \-m, --meta +Print only the column metadata for the named services. The output will be +a list of the columns return by a data query to the service, but will not +save the actual data. A default position and search size will be used for +the query: In the case of Cone services a negative size is used, for SIAP +services the FORMAT=METADATA flag is used in the query, and for tabular +Vizier services the entire table is accessed. Compliant VO services will +respond quickly with only the column metadata, tabular services may respond +more slowly due to the need to transfer the data. Adding the \fI-v\fP or +\fI--verbose-<N>\fP options will increment the VERBOSE level of services +and may return more metadata if available, to access these extra columns +the same level of verbosity must be set during a data query. + +.TP 0 +The following flags specify data query options: +.TP 8 +.B \-b <bpass> or --bandpass=<bpass> +Constrain the query by bandpass. The argument following the flag must +be one of the allowed bandpass specification string. Setting the flag +will constrain any Directory search used to only those resources where +the spectral coverage matches the given bandpass. Aliases for bandpasses +are allowed, see below. +.TP 8 +.B \-i <file> +Specify a file containing the remaining positional command-line input. The +command line is thought of as having the following components: the options +beginning with a '-' character and their associated arguments, one or more +<\fIresource\fP> names giving the service to invoke, an object name or +position, and an optional query size. The '-i' flag allows everything +except the options to be specified from a file (or the standard input if +the '-' argument is used), creating in effect a means to interactively +specify the e.g. resource/object without restarting the task, or to take +these values from a file or input stream to create multiple independent +queries. If either the resource or object name/position has already been +specified they do not need to be specified again. + +The format for the command file is the same as for the <\fIresource\fP>, +<\fIobjname\fP>, <\fIra\fP> <\fIdec\fP>, <\fIurl\fP>, or <\fIsr\fP> described +below and as they would appear on the command line, all input lines are +terminated with a newline, the file or input stream is terminated with an +EOF. An example of how this +may be used would be the using a command file such as: + +.nf + 2mass-psc m31,m51 0.5 + chandra ngc4258,m51 0.25 +.fi + +The task will process this file as if the two lines had been invoked as +separate commands. The advantage is that this input can be created +dynamically by another task, and we can group resource and object lists +into independent queries. See the \fIExamples\fP below for other uses. +.TP 8 +.B \-o <obj|file> or --object=<obj|file> +Specify the object name to use in a query. Object names are resolved +automatically to J2000 equatorial coordinates. The argument to this +flag may be the name of a single object, a comma-delimited list of +object names, the name of a file containing object names, or the +reserved value '-'. + +The reserved value '-' tells the task to take this information from the +standard input, processing doesn't begin until the object list has been +fully read. +.TP 8 +.B \-p <pos|file> or --pos=<pos|file> +Specify the position to use in a query. Positions are composed of two +values assumed to be equatorial J2000 coordinates. +Values specified as a floating-point decimal are assumed to be in units +of degrees, sexagesimal values may also be used and are assumed to be +equatorial RA and Dec. If the \fI<pos>\fP arg is used only one set of +coordinates may be given on the command-line and must be delimited by a comma, +however the argument may also be the name of a file containing coordinates +to be processed, or the reserved value '-'. + +The reserved value '-' tells the task to take this information from the +standard input, processing doesn't begin until the position list has been +fully read. +.TP 8 +.B \-r <radius> or --sr=<radius>[<units>] +Set the search radius. The default search size is 0.1 degrees unless +specified on the command-line and argument are assumed to be +in degrees, setting the size in other units is permitted using the +\fI-sr\fP flag. To specify the <units> for the \fI--sr\fP flag, the +argument should be suffixed by an 's' to specify arcseconds, an 'm' to set +arcminutes, and 'd' to set the size in degrees. By default, all +queries will be done using the same search size. Variable search +sizes accomplished using the '-i' flag described above. +.TP 8 +.B \-s <service> or --svc=<service> +Specify the service or url to invoke. In most cases the service, i.e. +the <\fIresource\fP> argument will be taken from the commandline based on +it's position. The exception is when the user want to specify a service +URL directly (e.g. to test a local service) because it isn't known to the +Directory, or to use the reserved values '-' or 'any'. Use of '-' tells +the task to read the service list from the standard input; use of +the word 'any' is a means to telling the task to dynamically create the +resource list from other options (e.g. "any image service" by using the +'-t image' option). +.TP 8 +.B \-t <type> or --type=<type> +Constrain the query by service type. The list of allowed service types +is given below. The actual string used in a Directory resource record +may be used if known, otherwise common use is to specify 'image' to +access Simple Image Access (SIAP) services, 'catalog' for Cone searches, +or 'table' for Vizier tabular data. + +.TP 0 +The following flags are specific to the writing of HTML or KML files: +.TP 8 +.B \--webnoborder +Disable the shaded border drawn around an HTML table. +.TP 8 +.B \--webnocolor +Disable the coloring for an HTML table. +.TP 8 +.B \--webnoheader +Disable the HTML page header written to the output file. + +.TP 8 +.B \--kmlmax=<N> +Specify the max number of placemarks to write. The default is 50, ordering +is not guaranteed. Setting the sampling will automatically increase the +maximum number of results returned. +.TP 8 +.B \--kmlsample=<N> +Specify the sampling of the result to be every \fI<N>\fP rows. The +default is to write all rows to the output file. If set, this value will +be used as a multiplier for the max number of placemarks automatically. +.TP 8 +.B \--kmlgroup=object +Group the results of a multi-resource/multi-object query into a single +hierarchical KML file grouped by the object or position index (default); +.TP 8 +.B \--kmlgroup=service +Group the results of a multi-resource/multi-object query into a single +hierarchical KML file grouped by the service name. +.TP 8 +.B \--kmlgroup=both +Groups the results of a multi-resource/multi-object query into a single +hierarchical KML file. The two top-level folders will be 'By Source' and +'By Object' indicating the grouping of the results. +.TP 8 +.B \--kmlnolabel +Disable the labelling of placemarks. By default, the ID_MAIN ucd for +each point will be used as a label. +.TP 8 +.B \--kmlnoregion +Disable the drawing of the region bounding box in a KML file. +.TP 8 +.B \--kmlnoverbose +Disable the writing of verbose information to the KML placemarks. By +default, each placemark will contain all information from that result. + + +.TP 0 +Input Options: +.TP 8 +.B \--cols=\fIcol_str\fP +Use columns specified in \fIcol_str\fP to read the \fIra\fP, \fIdec\fP +and \fIid\fP values respectively. \fIcol_str\fP is a comma-delimited +list where the \fIid\fP column is optional and will not be used if not +present as the third element in the list. Other columns may be given as +a single integer or as a range of the form \fIstart\fP-\fIend\fP indicating +the values in the \fIstart\fP thru \fIend\fP columns should be combined +into a single value. +.TP 8 +.B \-d, --delim=\fIdelim\fP +Use the \fIdelim\fP as the input table delimiter. By default, a space, tab, +comma, vertical bar ('|'), or semicolon may be used as a delimiter for the +input table. If no explicit delimiter is specified, the first occurance of +any one of these will be used. The reserved words \fIcomma\fP, +\fIspace\fP, \fItab\fP, or \fIbar\fP may be used in place of a specific +character. + +<DT><B>--ecols=</B><I>col_str</I> +.TP 8 +.B \--ecols=\fIcol_str\fP +Use the explicit columns specified in \fIcol_str\fP in the input table. This +option should only be used with formatted text tables where the desired +values will always be in the same columns of the file. Note that 'column' in +this case refers to a specific character column in a text file. Columns +may be a single integer or a range, and is a comma-delimited list as with +the \fI--cols\fP option. +.TP 8 +.B \-f, --force +Force the input table to be used even in the number of columns varies on +each line. The assumption here is that any variation (e.g empty columns) +occurs after the \fIra\fP, \fIdec\fP and \fIid\fP columns in the table. +.TP 8 +.B \--hskip=\fI<N>\fP +Skip \fI<N>\fP header lines in the input file. This option is only needed +when the lines to be skipped do not begin with the normal '#' comment +character. +.TP 8 +.B \--nlines=\fI<N>\fP +Use only \fI<N>\fP lines of the input table. +.TP 8 +.B \--sample=\fI<N>\fP +Sample the table every \fI<N>\fP lines. Setting the sample will not affect +the \fInlines\fP used. + + +.TP 0 +Output Options: +.TP 8 +.B \-1,--one +Save the results into a single file regardless of format. This option will +be set automatically if the output is being written to the standard output. +If the output format is something other than KML or XML, all results will +be concatenated into individual files of the form +"<svc>_<pid>.<extn>" so that each file will contain the object results from +each service where the columns will be the same. +.TP 8 +.B \-A,--ascii +Save the results as a whitespace delimited ascii table. If an output file +is created it will have a ".txt" extension appended automatically. +.TP 8 +.B \-C,--csv +Save the results as a comma-separated-value (CSV) table. If an output file +is created it will have a ".csv" extension appended automatically. +.TP 8 +.B \-H,--html +Save the results as an HTML table. If an output file +is created it will have a ".html" extension appended automatically. See +above for the \fI--webnoheader\fP option that can be used to disable the HTML page +header. +.TP 8 +.B \-I,--inventory +Query the \fIInventory Service\fP rather than the data services directly. +This will return simply a count of the results found, but when presented +with a table of resources and sources can be used to do a simple +crossmatch of the sources found in the catalogs available through the +service. +.TP 8 +.B \-K,--KML +Save the results as a Google Earth/Sky KML placemark file. If an output file +is created it will have a ".kml" extension appended automatically. See +above for additional options that control the content of the file. +.TP 8 +.B \-R, \-V or --raw, --votable +Save the results as a raw VOTable. If an output file is created it will +have a ".vot" extension appended automatically. +.TP 8 +.B \-T,--tsv +Save the results as a tab-separated-value (TSV) table. If an output file +is created it will have a ".tsv" extension appended automatically. +.TP 8 +.B \-O <root> or --output=<root> +Set the root of the output name. The reserved value '-' tells the task to +write to the standard output. +.TP 8 +.B \-X,--xml +Save the results wrapped XML file of the raw VOTable results. If an output +file is created it will have a ".xml" extension appended automatically. The +XML document will gather all the individual VOTable result files to a single +XML document, where each entry is wrapped by the element \fI<VOTABLE_ENTRY>\fP. +There will be three attributes: \fIsvc\fP will be the data service name, +\fIobj\fP will be the object name (if supplied), and the \fIindex\fP attribute +giving an index into the results. This index list is created by looping over +each service, and for each service, looping over the object/position list. +.TP 8 +.B \-e,--extract +Extract positional and access information to extra output files. By +default both files will be written, using \fI--extract=pos\fP will write only the +positional information file, using \fI--extract=urls\fP will write the access URLs only. +Access URLs are written one-per-line to a file with the same root name as +the main output but with a ".urls" extension; Positional information is +written to a file with a ".pos" extension and will contain three columns +made up of the identifier (the column with the \fIID_MAIN\fP ucd), RA and +Dec (the \fIPOS_EQ_RA_MAIN\fP and \fIPOS_EQ_DEC_MAIN\fP ucd columns +respectively). If these ucds appear more than once in a table, the first +occurrance will be used. + +Additionally, the \fI--extract=headers\fP and \fI--extract=kml\fP flags may be to to specify the +HTML and KML output be written to files with ".html" and ".kml" +extensions respectively. The \fI--extract=KML\fP flag will cause multi-resource +and/or multi-object queries to be collected into a single KML file. +The format-specific \fI--kml<opt>\fP and \fI--web<opt>\fP flags will apply to these files. +A \fI--extract=xml\fP flag will force the output format to be raw VOTable and gather +the results to a single XML document (see the \fI-X\fP option). + +Note that the URLs file can be used to later access the data (perhaps +after sub-selecting from the table based on some criteria) by calling +the task again using the filename as the only argument. +.TP 8 +.B \-n,--nosave +If enabled, this flag tells the task not to save results to local disk. +Status and result information will continue to be printed to the screen, +but no data are saved to disk. +.TP 8 +.B \-q,--quiet +Quiet mode. Suppress any extraneous output and warning messages. +.TP 8 +.B \-u,--url +Force the specified URL to be downloaded. + + +.TP 0 +DAL2 Query Options: +.TP 8 +.B \--band=\fIband_string\fP +The spectral bandpass is given in range-list format. For a numerical +bandpass the units are wavelength in vacuum in units of meters. +The spectral rest frame may optionally be qualified as either +\fIsource\fP or \fPobserver\fP, specified as a range-list +qualifier. Bandpass names are often not useful for spectra (they are +probably more useful for image or time series data) but there are cases where +they are useful for spectra, for example for a velocity spectrum of a +specific emission line. +.TP 8 +.B \--time=\fItime_string\fP +The time coverage (epoch) specified in range-list form as defined in section +8.7.2, in ISO 8601 format. If the time system used is not specified UTC is +assumed. The value specified may be a single value or an open or closed +range. If a single value is specified it matches any spectrum for which the +time coverage includes the specified value. If a two valued range is given, a +dataset matches if any portion of it overlaps the given temporal region. + + + +.SH DESCRIPTION +The \fBvodata\fP task allows a user to query and access VO data for multiple +resources and objects from a desktop or scripting environment. By design, +the task interface is meant to provide the following features: + +.TP 4 +.B \- +Resources (i.e. data services) may be referred to using a more familiar +\fIShortName\fP designation, or an IVO identifier, either of which will be +resolved to a specific \fIServiceURL\fR internally using the Directory. +.TP 4 +.B \- +Object names may be used to specify the location of a data query, the +position will be resolved internally using the \fISesame\fR web service. +.TP 4 +.B \- +Output files may be created in a variety of common formats easily manipulated +with other desktop tools. +.TP 4 +.B \- +Multiple resources and objects shall be queried in parallel when possible to +optimize the task. +.TP 4 +.B \- +Data referenced in a query response should be accessible by the task +automatically. +.TP 4 +.B \- +The command-line interface should be as friendly and as flexible as possible +to allow the task to be used in multiple ways. + +.PP +The task should quickly become familiar to users and is meant operate in +concert with the \fBvodirectory\fP and \fBvosesame\fP tasks to allow novice +users to begin to explore for data resources to be used in the final query. +Some of the flexibility of the task is shown in the Examples section below. +Major concepts of the task are detailed below as well. + + +.SS Argument Parsing +.PP +The meaning of the various command-line arguments is detailed below: + +.TP 8 +.B \fI<resource>\fP +The \fIShortName\fP or \fIIdentifier\fP of a data resource to be queried, a +comma-delimited list of either, or the name of a file containing either. +These names will be resolved to a data service URL using the Directory. The +\fI-s\fP option may be used to specify a non-registered \fIServiceURL\fP that +the task may use, however the \fI-t\fP option is then also required to +specify the type of service. +.TP 8 +.B \fI<objname>\fP +The name of an object, a comma-delimited list of object names, or the name of +a file containing object names. The coordinates of each object will be +resolved to a position prior to processing using the \fISesame\fR name +resolver service. An error will be returned if an object name cannot be +resolved, and that object will be skipped. +.TP 8 +.B \fI<ra> <dec>\fP +The J2000 equatorial RA and Dec position to the searched. Values given as +floating point values are assumed to be in decimal degrees, sexagesimal +values are assumed to be equatorial RA/Dec positions. Sexagesimal values may +be of the form \fIhh:mm:ss.s\fP or \fIhh:mm.m\fP for RA, or +\fIdd:mm:ss.s\fP or \fIdd:mm.m\fP for Dec. Only one coordinate pair may be +specified on the commandline. +.TP 8 +.B \fI<sr>\fP +The search size for the data query specified in decimal degrees. The default +size of 0.1 degrees will be used if this is not specified on the command line. +The \fI-rs\fP and +\fI-rm\fP options may be used specify the size in arc seconds and minutes +respectively. The \fI-i\fP option may be used to specify command-line input +options, where each command-line can include a different value for the search +size, otherwise only one value is allowed. +.TP 8 +.B \fI<url>\fP +A single URL, or the name of a file containing URLs listed one per line. + + +.SS Multi-Thread and Multi-Process Data Querying +.PP +All data queries require at least one \fIresource\fP and one \fIsource\fP +to be successful. The \fIresource\fP defines a specific data service to be +queried, and the \fIsource\fR is either an explicit position on the sky or +the name of an object that can be resolved to a position. Additional +parameters to the query are used to specify other options, but in essence +each data query is translated to a single URL that must be accessed by the +client task. In a complex query, lists of resource and/or objects create a +potentially large matrix of queries that must be made (i.e. +\fIN-services\fP by \fIN-objects\fP in total). Because a large fraction of +the time spent in waiting for a query to finish is in waiting for the +server to respond, we are able to run multiple queries simultaneously +without saturating our network bandwidth in most cases. +.PP +The \fBvodata\fP task will parallelize the list of services to be queried +by running a separate processing \fIthread\fP (i.e. a lightweight process +running in parallel within the main application) for each of the services +to be called. This allows queries to different servers to be run in +parallel, and since these servers will often reside on multiple machines +the client won't impact any one data provider too badly. In addition, +the list of objects to be queried at each service will be broken up into +multiple child processes and called simultaneously. This allows, for +example, 10 objects to be queried from 3 services (a total of 30 queries) +simultaneously. +.PP +The \fI--maxthreads=<N>mt\fP option can be used to set the max number of threads to be +created for processing the resource list (the default is 20). If the +resource list is larger than this value, the list will be processed with no +more than the max number running at any one time until all resources have +been queried. Similarly, the \fI--maxprocs=<N>\fP option can be used to set the +number of child processes to be created to process the object list (the +default is 10). When setting these values it is important to remember that +the total number of \fIpotential\fP processes running on your machine will +be the product of these to values. The default values were empirically found +to work reasonably well on most modern machines. +.PP +Additionally, it is worth considering the potential strain that can be put +on data providers' machines before changing these settings. The large +majority of Cone services for example come from a single server at HEASARC +and overloading the server with hundreds of requests to multiple resources +it provides may result in a failed request and what would appear to be no +data. One should consider using the \fI-i\fP flag as a means to query a +large object list against a resource list such that only the object +processing is parallelized and the server load is minimized (See the example +below). + + +.SS Output Filename Generation +.PP +The \fI-O\fP option may be used to specify the root part of output files +created by a data query. However, to guarantee that a multi-service, and/or +multi-object query doesn't overwrite a single output file, the filename root +will also include the \fIpid\fP (process ID) of the task that created it. +For a single service and object query no \fIpid\fP will be used as part of +the filename. This scheme guarantees unique output files across the various +processing scenarios, with similar root names for multiple files associated +with a specific query. + +Output tables may be created in a number of formats and will likewise have +extensions indicating the table type. The \fI-e\fP/\fI--extract\fP option +may create additional files for each query, and the \fI-g\fP/\fI--get\fP +option to access data will similarly create additional files. The structure +of an output filename is: + +.nf + + \fI<root>[_<pid>].<extn>\fP + +.fi +The meaning of \fI<pid>\fP and \fI<extn>\fP have been discussed above. If +the \fI-O\fP option was set then the \fI<root>\fP part of the name will +simply be the argument given to set the root name. Otherwise, the \fI<root>\fP +element will be of the form: +.nf + + \fI<svc>_<type>_<objname>\fP + \fI<svc>_<type>_<index>\fP + +.fi +The \fI<svc>\fP is derived from the service name used, the \fI<type>\fP is +a single-character code to indicate the type of service used ('I' for image, +'C' for catalog, and 'T' for table). The final element will either be the +object name or the index in a list of positions of no object was specified. + +.SS Verbosity +.PP +The \fI-v\fP and \fI-vv\fP options serve a dual purpose: within the task they +set the level of output verbosity in terms of what is reported during +processing (Similarly, the \fI-q\fP option can be used to turn off output +reporting entirely). These flags will however also increase the value of the +\fBVERBOSE\fP parameter sent to services during a data query. The default +value is at least 1, with the highest level being 3. Using the \fI-v\fP flag +sets \fBVERBOSE=2\fP and \fI-vv\fP sets \fBVERBOSE=3\fP. +.PP +The VERBOSE level can be important in accessing result columns that may only +be returned at the highest level. When using the \fI-m\fR +(or \fI--meta\fR) flag to print +the column metadata, the verbose options will also affect the results and it +is important that the same verbosity be set when doing the actual data +query and access. + +.SS Bandpass and Service Type Aliases +.PP +The type constraint (\fI-t\fP or \fI--type\fP) accepts only the +following arguments: +.nf + + catalog Cone search services + image Simple Image Access services + spectra Simple Spectral Access services + table Vizier services + <literal> ResourceType from Directory record + +.fi +.PP +The bandpass constraint (\fI-b\fP or \fI--bandpass\fP) accepts only +the following arguments: +.nf + + Radio Millimeter Infrared (IR) + Optical Ultraviolet (UV) X-Ray (xray) + Gamma-Ray (GR) + +.fi +Values in parenthese are acceptable aliases. All matches are cases +insensitive. + + +.SS Range-List Parameters +.PP +Some parameters (for example BAND and TIME) may allow a parameter value to be +specified as a numeric range. Such range-valued parameters use the forward +slash (\fI/\fP) character as the separator between elements of the +range specification (as in the ISO 8601 date specification after which this +convention is patterned). For example, \fI5E-7/8E-7\fP would +specify a range consisting of all values from 5E-7 to 8E-7, inclusive. If a +third field is specified it is a step size for traversing the indicated +range. If a parameter permits a step size the semantics of the step size are +defined by the specific parameter. +.PP +An open range may be specified by omitting either range value. If the first +value is omitted the range is open toward lower values. If the second value +is omitted the range is open toward higher values. Omitting both values +indicates an infinite range which accepts all values. For example, +\fI/5\fP is an open range which accepts all values less than or +equal to 5. To specify all values less than 5, \fI/4\fP would be +used (for an integer valued range). Range values are limited to numeric +values or ISO dates. +.PP +A list may be qualified by appending the character \fI;\fP (semicolon) +followed by a qualifier string. For example \fI1E-7/3E-6;source\fP could +specify a spectral bandpass in the rest frame of the source. List and +range syntax may be combined, e.g., to indicate a list of scalar or +range-valued parameter values. Such a range list may be ordered or +unordered, and may contain either numeric or string data. An ordered +list is one which requires values to be processed in a specified order, +and to ensure this the range list is sorted or ordered by the service as +necessary before being used. It is the responsibility of the service to +sort an ordered range list, hence the client can input ranges or range +values in any order for an ordered range list and the result will be +the same. The sequence in which items in an unordered list occur on the +other hand is significant, as since there is no intrinsic ordering for +the list which can be enforced by the service, items will be processed +by the service in the order they are input by the client. +.PP +TIME and BAND are typical examples of ordered range lists. Since a dataset +matches the query if it contains data in any of the specified ranges, +logically it does not matter in what order the ranges are given, or whether +the first element of a range is less than the second, or whether ranges +overlap; the result should be the same in all cases. Hence the range list has +an intrinsic ordering irrespective of how ranges are input. Unless otherwise +specified in the definition of a given parameter, range lists are assumed to +be ordered. + + + +.SH VOCLIENT DAEMON PROCESSING +All VO-CLI tasks are built upon the VOClient interface an rely on a +separate \fIvoclientd\fP process to provide the VO functionality. The +voclientd task is distributed as part of VO-CLI and will be started +automatically by each task if it is not already running. If problems are +encountered, you may want to manually start the voclientd in a separate +window before running the task so you can monitor the output for error +messages. + +.SH RESOURCE CACHING +Directory resolution is a common activity of VO-CLI tasks and so results +will be cached in the $HOME/.voclient/cache/regResolver directory based on +the search term, service type and bandpass parameters. Defining the +\fIVOC_NO_CACHE\fP environment variable will cause the task to ignore the +cache. + +.SH EXAMPLES + +.TP 4 +1) +Query the GSC 2.3 catalog for stars a) within the 0.1 degree +default search size around NGC 1234: b) around all positions +contained in file 'pos.txt': c) for the list of objects given +on the command line: d) query a list of services for a list +of positions: e) print a count of results that would be returned +from 3 services for each position in a file: +.nf + + % vodata gsc2.3 ngc1234 (a) + % vodata gsc2.3 pos.txt (b) + % vodata gsc2.3 m31,m51,m93 (c) + % vodata svcs.txt pos.txt (d) + % vodata hst,chandra,gsc2.3 pos.txt (e) + +.fi + +.TP 4 +2) +Query all (142) image services having data of the subdwarf galaxy +IC 10, print a count of the results only: +.nf + + % vodata -c -t image any IC10 + % vodata --count --type=image any IC10 + +.fi +Note that we use the reserved word '\fIany\fP' for the service name and +constrain by image type. The task will automatically query the Directory to +create the list of services to be queried. + +.TP 4 +3) +Print a count of X-ray catalog data around Abell2712: +.nf + + % vodata -c -t catalog -b x-ray any abell2712 + % vodata --count --type=catalog --bandpass=x-ray any abell2712 + +.fi +In this case we constrain both the service type as well as the spectral +coverage published for the resource in the Directory. We use the reserved +'\fIany\fP' service name to query multiple services and use the '\fI-c\fP' +flag to print a count without saving results. The object name is resolved +to coordinates internally. (Note: this example may take a while to run). + + +.TP 4 +4) +Print the column metadata returned by the RC3 catalog service: +.nf + + % vodata --meta rc3 or vodata -m rc3 + +.fi +The output will print the result using the default VERBOSE level, adding +the \fI-v\fP will set the query parameter VERBOSE=2, adding \fI-vv\fP will +set VERBOSE=3 (to print all available columns). When accessing data the +same \fI-v\fP flags will be required to retrieve columns at that +\fIVERBOSE\fP level. + + +.TP 4 +5) +Use the Directory to query for resources using the search terms +"cooling flow". Upon examining the output the user notices a +Vizier paper titled "\fICooling Flows in 207 clusters of Galaxies\fP" +that looks interesting. Use the \fBvodata\fP task to download all +tables associated with this paper, save tables in the default +CSV format: +.nf + + % vodirectory cooling flow + % vodata -O white97 -all J/MNRAS/292/419/ + % vodata --output=white97 --all J/MNRAS/292/419/ + +.fi +All 7 tables will be written to the current directory to files +having a root name 'white97' (chosen based on the author and +publication date). + + +.TP 4 +6) +Find a suitable XMM image service, get a (brief) count of the +XMM images available for 3c273, and if there aren't too many, +download the images and save the extracted access URLs: +.nf + + % vodirectory -rv -t image xmm + ShortName ResourceType Title + ------------------------------------------------------.... + XMM-Newton SIAP/ARCHIVE XMM-Newton Archive .... + + % vodata -cq xmm-newton 3c273 + xmm-newton 27 I XMM-Newton Archive .... + + % vodata --count --quiet xmm-newton 3c273 + xmm-newton 27 I XMM-Newton Archive .... + + % vodata --get xmm-newton 3c273 + .... will query and download 27 images. + +.fi + +.TP 4 +7) +Query for the images available from 2MASS at a given position, +extract the positions and service URLs to separate files: +.nf + + % vodata -e -O 2mass -t image 2mass 12:34:56.7 -23:12:45.2 + % vodata -e --output=2mass --type=image 2mass 12:34:56.7 -23:12:45.2 + +.fi +The query produces files with the root name '2mass', and exten- sions of +"\fI.csv\fP" (the main response), "\fI.pos\fP" (the extracted pos- itions), +and "\fI.urls\fP" (the access references). The user inspects the files and +notices that the references return both FITS and HTML files, but she only +wants the FITS image date and uses \fBvodata\fP to download only those: +.nf + + % grep fits 2mass_I_001_15998.urls > images.txt + % vodata images.txt + or + % grep fits 2mass_I_001_15998.urls | vodata -i - + +.fi +In both cases we pass URLs to the task which bypasses the query and directly +access the images. However, in the first case we process the entire list +and are able to take advantage of the \fI-maxdownloads=<N>\fR option to +increase the number of simultaneous downloads. In the second case, the +\fI-i\fP flag causes the task to interpret each line of the input stream +as a separate command and so the data are always downloaded one at a time +(which is the default download behavior anyway). + +.TP 4 +8) +Use \fBvodata\fP as a test client for a locally-installed SIAP service: +.nf + + % vodata -t image -s http://localhost/siap.pl 180.0 0.0 + % vodata --type=image --svc=http://localhost/siap.pl 180.0 0.0 + +.fi +In this case we force the ServiceURL using the '\fI-s\fP' flag, but since we +can't do a Directory query to discover what type of service this is, we must +use the '\fI-t\fP' flag to indicate it is an image service. + + +.TP 4 +9) +Create a local table containing the Abell catalog. Begin with a +Directory query to find likely services using the \fBvodirectory\fP task, print +a verbose description of each resource and page the results with \fIless\fP: +.nf + + % vodirectory -v -v --type=catalog abell | less + +.fi +The verbose results indicate a number of services with differing +requirements for what is included. We decide to use the service from +HEASARC since it contains southern hemisphere data and constraints we are +interested in. Try an all-sky query to retrieve the entire catalog, use +the service identifier to be specific about where we want to go: +.nf + + % vodata -e ivo://nasa.heasarc/abell 0.0 0.0 180.0 + % vodata --extract ivo://nasa.heasarc/abell 0.0 0.0 180.0 + +.fi +We use the '\fI-e\fP' flag to extract the positions to a separate file with +a "\fI.pos\fP" extension so that we can use these in later queries. +However, the \fI.pos\fP file additionally contains the ID from the original +catalog in column 1. Strip this column so we're left with only RA and DEC +and query for Chandra observations at each position: +.nf + + % cut -c6- *.pos | vodata ivo://nasa.heasarc/chanmaster -p - + % cut -c6- *.pos | vodata ivo://nasa.heasarc/chanmaster --pos=- + +.fi +Here we used the unix '\fIcut\fP' utility to remove the first column and +pipe the resulting positions to the task, using the '\fI-p -\fP' option to +indicate positions should be read from stding, and the IVO identifier of +the Chandra observation master log we also discovered from the Directory. + + +.TP 4 +10) +Interactively query for a count of Chandra observations of Messier +objects: +.nf + + % vodata -cq chandra -i - + m31 + chandra 335 I Chandra X-Ray Observatory Data Archive + : : : : : + +.fi +Note that by using the '\fI-i\fP' flag we execute each query as if we'd put +the object name on the command line. Using the '\fI-o\fP' flag would +instead read all of the objects from the stdin but then execute the queries +in parallel following an EOF to terminate the input. + + +.TP 4 +11) +Use the \fISTILTS\fP task '\fItpipe\fP' to select rows from a VOTable +of QSOs (made with an earlier query) where the redshift is > 0.2. Output +only the positions and pipe this to \fBvodata\fP to generate a new query to +see whether HST has observed any of these objects: +.nf + + % stilts tpipe ifmt=votable qso_survey.vot \ + cmd='select "Z > 0.2"' \ + cmd='keepcols "RA DEC"' | vodata -p - hstpaec + +.fi +Note that we use the '\fI-p -\fP' flag to tell the task to take it's list of +positions from the piped input. The positions are used to query the HST +Planned and Archived Exposure Catalog (\fIhstpaec\fP) + + +.TP 4 +12) +The user has a task called '\fIwcsinfo\fP' that takes a list of images +and outputs a text table containing the plate center and size in degrees. +Use this task as part of a query for 2MASS point sources contained in each +image: +.nf + + % wcsinfo *.fits | vodata 2mass-psc -i - + +.fi +Here we specify the desired service (\fI2mass-psc\fP) on the commandline as +usual, and allow the remainder of the args (i.e. the position and search +size) to be read from the stdin. This allows for variable search sizes but +processes the positions serially. If the sizes are all the same (say 25 +arcmin), multiple queries can be done simultaneously using just a position +file, e.g. +.nf + + % wcsinfo -pos_only *.fits > centers.txt + % vodata --sr=25m 2mass-psc centers.txt + +.fi + + + +.TP 4 +13) +Query a large list of objects against a number of ASCA resources. +Because the ASCA catalogs are served by the same machine, use the \fI'-i'\fP +option and a command file to process only each resource sequentially while +still parallelizing the object lists: + +.nf + % cat cmds.txt + ASCA survey.tbl + ASCA\\ GIS survey.tbl + ASCA\\ GPS survey.tbl + : : + + % vodata -i cmds.txt +.fi +.PP +Note that we've needed to escape the space in the resource name in some +cases. To avoid this, use of the IVO identifier for each resource is usually +preferred. + +.TP 4 +14) +Query the VO for GALEX data of M51. Because the \fIShortName\fP GALEX +is not unique, we must either specify the IVO identifier of a +specific service to query, or if we're interested in results from all +supported data services with \fIgalex\fP in the name: +.nf + + % vodata -a galex M51 + % vodata --all galex M51 + +.fi +The results come from the Cone and SIAP services both called \fIGALEX\fP, +as well as an additional SIAP service called 'GALEX_Atlas'. Note that the +service names are case insensitive in either case. + + +.TP 4 +15) +Process a list of hundreds of positions against the GSC2.3 catalog: +.nf + + % vodata gsc2.3 positions.txt + +.fi + + +.TP 4 +16) +Process a list of hundreds of positions against the GSC2.3 catalog, assume +that the input table has a 5 line header of text and three columns +(id,ra,dec). Note that the \fIcols\fP option requires the optional id +be specified last: +.nf + + % vodata --cols=2,3,1 --hskip=5 gsc2.3 positions.txt + +.fi + +.SH BUGS +Some services don't repond properly to the metadata query and will print +a "no attributes found" message. + +.SH TODO +.PP +- Additional command-line options should be allowed to be specified in a +command file. +.PP +- Support for SSAP, SAMP, TAP + + +.SH Revision History +June 2007 - This task is new. +.SH Author +Michael Fitzpatrick (fitz@noao.edu), June 2007 +.SH "SEE ALSO" +voclient(1), voclientd(1), vosesame(1), vodirectory(1) |