diff options
Diffstat (limited to 'vendor/voclient/doc/OLD/vodata.html')
| -rw-r--r-- | vendor/voclient/doc/OLD/vodata.html | 1152 |
1 files changed, 1152 insertions, 0 deletions
diff --git a/vendor/voclient/doc/OLD/vodata.html b/vendor/voclient/doc/OLD/vodata.html new file mode 100644 index 00000000..7f8a3851 --- /dev/null +++ b/vendor/voclient/doc/OLD/vodata.html @@ -0,0 +1,1152 @@ +Content-type: text/html + +<HTML><HEAD><TITLE>Manpage of VODATA</TITLE> +</HEAD><BODY> +<H1>VODATA</H1> +Section: User Commands (1)<BR>Updated: July 2007<BR><A HREF="#index">Index</A> +<A HREF="http://localhost/cgi-bin/man/man2html">Return to Main Contents</A><HR> + +<A NAME="lbAB"> </A> +<H2>NAME</H2> + +vodata - query and access VO data services +<A NAME="lbAC"> </A> +<H2>SYNOPSIS</H2> + +<B>vodata</B> [<I><flags></I>] [ <<I>resource</I>> [[ <<I>objname</I>> [ <<I>sr</I>> ]]] ] +<P> +<B>vodata</B> [<I><flags></I>] [ <<I>resource</I>> [[ <<I>ra</I>> <<I>dec</I>> [ <<I>sr</I>> ]]] ] +<P> +<B>vodata</B> [<I><flags></I>] [ <<I>url</I>> ] +<P> +<A NAME="lbAD"> </A> +<H2>OPTIONS</H2> + +The <I>vodata</I> task accepts the following options: +<DL COMPACT> +<DT><B>-h,--help</B> + +<DD> +Print a help summary to the terminal and exit. No processing is done +following this flag. +<DT><B>-v,--verbose</B> + +<DD> +Verbose output. The output will be more verbose than normal but exactly +what is printed depends on whether other flags are enabled to changed the +basic task behavior. +<DT><B>--vverbose</B> + +<DD> +Very-verbose output. Even more output. +<P> +<DT>The following flags control the major behavior of the task, i.e. the type <DD> +of output to present. +<DT><B>-a, --all</B> + +<DD> +Perform an action based on all available data. When used as part of a +data query, this flag causes the <<I>resource</I>> argument to be used in +a substring match of Directory <I>ShortName</I> or <I>Identifier</I> fields +to create the actual list of resources to be queried. If the <I>ShortName</I> +of a <I>TABULARSKYSERVICE</I> from Vizier is given, the <<I>resource</I>> +will typically expand to include all tables associated with the paper, +and providing a means to access all of these tables from a single query. +<DT><B>-c, --count</B> + +<DD> +Print only a count of the matching records found and do not save any +results. The standard output for the task is to echo some of the input +parameters and print a table of results showing progress and the number +of matching records. If this flag is set, the output written to the +screen will be the same, however the data will not be saved locally. +<DT><B>-g, --get</B> + +<DD> +Get the data referenced by the results of a data query. This typically +only applies to Simple Image Access service in which the result of a +query include a column of "<I>access references</I>" to the actual data +that must be resolved separately. Setting this flag will cause all data +references to be resolved by the task once all of the data queries have +been completed. +<P> +Access references are appended to a master "access list" as each query +completes. In general the order in which these are retrieved cannot be +guaranteed. Data downloads can be done in parallel by setting the number +of concurrent max downloads using the <I>--maxdownloads=<N></I> flag, the +default is to download one file at a time. If this flag is followed +with a comma-delimited list of numbers, only those rows in the result +table will be accessed. +<DT><B>-m, --meta</B> + +<DD> +Print only the column metadata for the named services. The output will be +a list of the columns return by a data query to the service, but will not +save the actual data. A default position and search size will be used for +the query: In the case of Cone services a negative size is used, for SIAP +services the FORMAT=METADATA flag is used in the query, and for tabular +Vizier services the entire table is accessed. Compliant VO services will +respond quickly with only the column metadata, tabular services may respond +more slowly due to the need to transfer the data. Adding the <I>-v</I> or +<I>--verbose-<N></I> options will increment the VERBOSE level of services +and may return more metadata if available, to access these extra columns +the same level of verbosity must be set during a data query. +<P> +<DT>The following flags specify data query options:<DD> +<DT><B>-b <bpass> or --bandpass=<bpass></B> + +<DD> +Constrain the query by bandpass. The argument following the flag must +be one of the allowed bandpass specification string. Setting the flag +will constrain any Directory search used to only those resources where +the spectral coverage matches the given bandpass. Aliases for bandpasses +are allowed, see below. +<DT><B>-i <file></B> + +<DD> +Specify a file containing the remaining positional command-line input. The +command line is thought of as having the following components: the options +beginning with a '-' character and their associated arguments, one or more +<<I>resource</I>> names giving the service to invoke, an object name or +position, and an optional query size. The '-i' flag allows everything +except the options to be specified from a file (or the standard input if +the '-' argument is used), creating in effect a means to interactively +specify the e.g. resource/object without restarting the task, or to take +these values from a file or input stream to create multiple independent +queries. If either the resource or object name/position has already been +specified they do not need to be specified again. +<P> +The format for the command file is the same as for the <<I>resource</I>>, +<<I>objname</I>>, <<I>ra</I>> <<I>dec</I>>, <<I>url</I>>, or <<I>sr</I>> described +below and as they would appear on the command line, all input lines are +terminated with a newline, the file or input stream is terminated with an +EOF. An example of how this +may be used would be the using a command file such as: +<P> +<PRE> + 2mass-psc m31,m51 0.5 + chandra ngc4258,m51 0.25 +</PRE> + +<P> +The task will process this file as if the two lines had been invoked as +separate commands. The advantage is that this input can be created +dynamically by another task, and we can group resource and object lists +into independent queries. See the <I>Examples</I> below for other uses. +<DT><B>-o <obj|file> or --object=<obj|file></B> + +<DD> +Specify the object name to use in a query. Object names are resolved +automatically to J2000 equatorial coordinates. The argument to this +flag may be the name of a single object, a comma-delimited list of +object names, the name of a file containing object names, or the +reserved value '-'. +<P> +The reserved value '-' tells the task to take this information from the +standard input, processing doesn't begin until the object list has been +fully read. +<DT><B>-p <pos|file> or --pos=<pos|file></B> + +<DD> +Specify the position to use in a query. Positions are composed of two +values assumed to be equatorial J2000 coordinates. +Values specified as a floating-point decimal are assumed to be in units +of degrees, sexagesimal values may also be used and are assumed to be +equatorial RA and Dec. If the <I><pos></I> arg is used only one set of +coordinates may be given on the command-line and must be delimited by a comma, +however the argument may also be the name of a file containing coordinates +to be processed, or the reserved value '-'. +<P> +The reserved value '-' tells the task to take this information from the +standard input, processing doesn't begin until the position list has been +fully read. +<DT><B>-r <radius> or --sr=<radius>[<units>]</B> + +<DD> +Set the search radius. The default search size is 0.1 degrees unless +specified on the command-line and argument are assumed to be +in degrees, setting the size in other units is permitted using the +<I>-sr</I> flag. To specify the <units> for the <I>--sr</I> flag, the +argument should be suffixed by an 's' to specify arcseconds, an 'm' to set +arcminutes, and 'd' to set the size in degrees. By default, all +queries will be done using the same search size. Variable search +sizes accomplished using the '-i' flag described above. +<DT><B>-s <service> or --svc=<service></B> + +<DD> +Specify the service or url to invoke. In most cases the service, i.e. +the <<I>resource</I>> argument will be taken from the commandline based on +it's position. The exception is when the user want to specify a service +URL directly (e.g. to test a local service) because it isn't known to the +Directory, or to use the reserved values '-' or 'any'. Use of '-' tells +the task to read the service list from the standard input; use of +the word 'any' is a means to telling the task to dynamically create the +resource list from other options (e.g. "any image service" by using the + +<DT><B>-t <type> or --type=<type></B> + +<DD> +Constrain the query by service type. The list of allowed service types +is given below. The actual string used in a Directory resource record +may be used if known, otherwise common use is to specify 'image' to +access Simple Image Access (SIAP) services, 'catalog' for Cone searches, +or 'table' for Vizier tabular data. +<P> +<DT>The following flags are specific to the writing of HTML or KML files:<DD> +<DT><B>--webnoborder</B> + +<DD> +Disable the shaded border drawn around an HTML table. +<DT><B>--webnocolor</B> + +<DD> +Disable the coloring for an HTML table. +<DT><B>--webnoheader</B> + +<DD> +Disable the HTML page header written to the output file. +<P> +<DT><B>--kmlmax=<N></B> + +<DD> +Specify the max number of placemarks to write. The default is 50, ordering +is not guaranteed. Setting the sampling will automatically increase the +maximum number of results returned. +<DT><B>--kmlsample=<N></B> + +<DD> +Specify the sampling of the result to be every <I><N></I> rows. The +default is to write all rows to the output file. If set, this value will +be used as a multiplier for the max number of placemarks automatically. +<DT><B>--kmlgroup=object </B> + +<DD> +Group the results of a multi-resource/multi-object query into a single +hierarchical KML file grouped by the object or position index (default); +<DT><B>--kmlgroup=service </B> + +<DD> +Group the results of a multi-resource/multi-object query into a single +hierarchical KML file grouped by the service name. +<DT><B>--kmlgroup=both </B> + +<DD> +Groups the results of a multi-resource/multi-object query into a single +hierarchical KML file. The two top-level folders will be 'By Source' and + +<DT><B>--kmlnolabel</B> + +<DD> +Disable the labelling of placemarks. By default, the ID_MAIN ucd for +each point will be used as a label. +<DT><B>--kmlnoregion</B> + +<DD> +Disable the drawing of the region bounding box in a KML file. +<DT><B>--kmlnoverbose</B> + +<DD> +Disable the writing of verbose information to the KML placemarks. By +default, each placemark will contain all information from that result. +<P> +<P> +<DT>Input Options:<DD> +<DT><B>--cols=</B><I>col_str</I> + +<DD> +Use columns specified in <I>col_str</I> to read the <I>ra</I>, <I>dec</I> +and <I>id</I> values respectively. <I>col_str</I> is a comma-delimited +list where the <I>id</I> column is optional and will not be used if not +present as the third element in the list. Other columns may be given as +a single integer or as a range of the form <I>start</I>-<I>end</I> indicating +the values in the <I>start</I> thru <I>end</I> columns should be combined +into a single value. +<DT><B>-d, --delim=</B><I>delim</I> + +<DD> +Use the <I>delim</I> as the input table delimiter. By default, a space, tab, +comma, vertical bar ('|'), or semicolon may be used as a delimiter for the +input table. If no explicit delimiter is specified, the first occurance of +any one of these will be used. The reserved words <I>comma</I>, +<I>space</I>, <I>tab</I>, or <I>bar</I> may be used in place of a specific +character. +<P> +<DT><B>--ecols=</B><I>col_str</I> +<DT><B>--ecols=</B><I>col_str</I> + +<DD> +Use the explicit columns specified in <I>col_str</I> in the input table. This +option should only be used with formatted text tables where the desired +values will always be in the same columns of the file. Note that 'column' in +this case refers to a specific character column in a text file. Columns +may be a single integer or a range, and is a comma-delimited list as with +the <I>--cols</I> option. +<DT><B>-f, --force</B> + +<DD> +Force the input table to be used even in the number of columns varies on +each line. The assumption here is that any variation (e.g empty columns) +occurs after the <I>ra</I>, <I>dec</I> and <I>id</I> columns in the table. +<DT><B>--hskip=</B><I><N></I> + +<DD> +Skip <I><N></I> header lines in the input file. This option is only needed +when the lines to be skipped do not begin with the normal '#' comment +character. +<DT><B>--nlines=</B><I><N></I> + +<DD> +Use only <I><N></I> lines of the input table. +<DT><B>--sample=</B><I><N></I> + +<DD> +Sample the table every <I><N></I> lines. Setting the sample will not affect +the <I>nlines</I> used. +<P> +<P> +<DT>Output Options:<DD> +<DT><B>-1,--one</B> + +<DD> +Save the results into a single file regardless of format. This option will +be set automatically if the output is being written to the standard output. +If the output format is something other than KML or XML, all results will +be concatenated into individual files of the form +"<svc>_<pid>.<extn>" so that each file will contain the object results from +each service where the columns will be the same. +<DT><B>-A,--ascii</B> + +<DD> +Save the results as a whitespace delimited ascii table. If an output file +is created it will have a ".txt" extension appended automatically. +<DT><B>-C,--csv</B> + +<DD> +Save the results as a comma-separated-value (CSV) table. If an output file +is created it will have a ".csv" extension appended automatically. +<DT><B>-H,--html</B> + +<DD> +Save the results as an HTML table. If an output file +is created it will have a ".html" extension appended automatically. See +above for the <I>--webnoheader</I> option that can be used to disable the HTML page +header. +<DT><B>-I,--inventory</B> + +<DD> +Query the <I>Inventory Service</I> rather than the data services directly. +This will return simply a count of the results found, but when presented +with a table of resources and sources can be used to do a simple +crossmatch of the sources found in the catalogs available through the +service. +<DT><B>-K,--KML</B> + +<DD> +Save the results as a Google Earth/Sky KML placemark file. If an output file +is created it will have a ".kml" extension appended automatically. See +above for additional options that control the content of the file. +<DT><B>-R, -V or --raw, --votable</B> + +<DD> +Save the results as a raw VOTable. If an output file is created it will +have a ".vot" extension appended automatically. +<DT><B>-T,--tsv</B> + +<DD> +Save the results as a tab-separated-value (TSV) table. If an output file +is created it will have a ".tsv" extension appended automatically. +<DT><B>-O <root> or --output=<root></B> + +<DD> +Set the root of the output name. The reserved value '-' tells the task to +write to the standard output. +<DT><B>-X,--xml</B> + +<DD> +Save the results wrapped XML file of the raw VOTable results. If an output +file is created it will have a ".xml" extension appended automatically. The +XML document will gather all the individual VOTable result files to a single +XML document, where each entry is wrapped by the element <I><VOTABLE_ENTRY></I>. +There will be three attributes: <I>svc</I> will be the data service name, +<I>obj</I> will be the object name (if supplied), and the <I>index</I> attribute +giving an index into the results. This index list is created by looping over +each service, and for each service, looping over the object/position list. +<DT><B>-e,--extract</B> + +<DD> +Extract positional and access information to extra output files. By +default both files will be written, using <I>--extract=pos</I> will write only the +positional information file, using <I>--extract=urls</I> will write the access URLs only. +Access URLs are written one-per-line to a file with the same root name as +the main output but with a ".urls" extension; Positional information is +written to a file with a ".pos" extension and will contain three columns +made up of the identifier (the column with the <I>ID_MAIN</I> ucd), RA and +Dec (the <I>POS_EQ_RA_MAIN</I> and <I>POS_EQ_DEC_MAIN</I> ucd columns +respectively). If these ucds appear more than once in a table, the first +occurrance will be used. +<P> +Additionally, the <I>--extract=headers</I> and <I>--extract=kml</I> flags may be to to specify the +HTML and KML output be written to files with ".html" and ".kml" +extensions respectively. The <I>--extract=KML</I> flag will cause multi-resource +and/or multi-object queries to be collected into a single KML file. +The format-specific <I>--kml<opt></I> and <I>--web<opt></I> flags will apply to these files. +A <I>--extract=xml</I> flag will force the output format to be raw VOTable and gather +the results to a single XML document (see the <I>-X</I> option). +<P> +Note that the URLs file can be used to later access the data (perhaps +after sub-selecting from the table based on some criteria) by calling +the task again using the filename as the only argument. +<DT><B>-n,--nosave</B> + +<DD> +If enabled, this flag tells the task not to save results to local disk. +Status and result information will continue to be printed to the screen, +but no data are saved to disk. +<DT><B>-q,--quiet</B> + +<DD> +Quiet mode. Suppress any extraneous output and warning messages. +<DT><B>-u,--url</B> + +<DD> +Force the specified URL to be downloaded. +<P> +<P> +<DT>DAL2 Query Options:<DD> +<DT><B>--band=</B><I>band_string</I> + +<DD> +The spectral bandpass is given in range-list format. For a numerical +bandpass the units are wavelength in vacuum in units of meters. +The spectral rest frame may optionally be qualified as either +<I>source</I> or observer, specified as a range-list +qualifier. Bandpass names are often not useful for spectra (they are +probably more useful for image or time series data) but there are cases where +they are useful for spectra, for example for a velocity spectrum of a +specific emission line. +<DT><B>--time=</B><I>time_string</I> + +<DD> +The time coverage (epoch) specified in range-list form as defined in section +8.7.2, in ISO 8601 format. If the time system used is not specified UTC is +assumed. The value specified may be a single value or an open or closed +range. If a single value is specified it matches any spectrum for which the +time coverage includes the specified value. If a two valued range is given, a +dataset matches if any portion of it overlaps the given temporal region. +<P> +<P> +<P> +</DL> +<A NAME="lbAE"> </A> +<H2>DESCRIPTION</H2> + +The <B>vodata</B> task allows a user to query and access VO data for multiple +resources and objects from a desktop or scripting environment. By design, +the task interface is meant to provide the following features: +<P> +<DL COMPACT> +<DT><B>-</B> + +<DD> +Resources (i.e. data services) may be referred to using a more familiar +<I>ShortName</I> designation, or an IVO identifier, either of which will be +resolved to a specific <I>ServiceURL</I> internally using the Directory. +<DT><B>-</B> + +<DD> +Object names may be used to specify the location of a data query, the +position will be resolved internally using the <I>Sesame</I> web service. +<DT><B>-</B> + +<DD> +Output files may be created in a variety of common formats easily manipulated +with other desktop tools. +<DT><B>-</B> + +<DD> +Multiple resources and objects shall be queried in parallel when possible to +optimize the task. +<DT><B>-</B> + +<DD> +Data referenced in a query response should be accessible by the task +automatically. +<DT><B>-</B> + +<DD> +The command-line interface should be as friendly and as flexible as possible +to allow the task to be used in multiple ways. +<P> +</DL> +<P> + +The task should quickly become familiar to users and is meant operate in +concert with the <B>vodirectory</B> and <B>vosesame</B> tasks to allow novice +users to begin to explore for data resources to be used in the final query. +Some of the flexibility of the task is shown in the Examples section below. +Major concepts of the task are detailed below as well. +<P> +<BR> +<A NAME="lbAF"> </A> +<H3>Argument Parsing</H3> + +<P> + +The meaning of the various command-line arguments is detailed below: +<P> +<DL COMPACT> +<DT><B></B><I><resource></I> + +<DD> +The <I>ShortName</I> or <I>Identifier</I> of a data resource to be queried, a +comma-delimited list of either, or the name of a file containing either. +These names will be resolved to a data service URL using the Directory. The +<I>-s</I> option may be used to specify a non-registered <I>ServiceURL</I> that +the task may use, however the <I>-t</I> option is then also required to +specify the type of service. +<DT><B></B><I><objname></I> + +<DD> +The name of an object, a comma-delimited list of object names, or the name of +a file containing object names. The coordinates of each object will be +resolved to a position prior to processing using the <I>Sesame</I> name +resolver service. An error will be returned if an object name cannot be +resolved, and that object will be skipped. +<DT><B></B><I><ra> <dec></I> + +<DD> +The J2000 equatorial RA and Dec position to the searched. Values given as +floating point values are assumed to be in decimal degrees, sexagesimal +values are assumed to be equatorial RA/Dec positions. Sexagesimal values may +be of the form <I>hh:mm:ss.s</I> or <I>hh:mm.m</I> for RA, or +<I>dd:mm:ss.s</I> or <I>dd:mm.m</I> for Dec. Only one coordinate pair may be +specified on the commandline. +<DT><B></B><I><sr></I> + +<DD> +The search size for the data query specified in decimal degrees. The default +size of 0.1 degrees will be used if this is not specified on the command line. +The <I>-rs</I> and +<I>-rm</I> options may be used specify the size in arc seconds and minutes +respectively. The <I>-i</I> option may be used to specify command-line input +options, where each command-line can include a different value for the search +size, otherwise only one value is allowed. +<DT><B></B><I><url></I> + +<DD> +A single URL, or the name of a file containing URLs listed one per line. +<P> +<P> +</DL> +<A NAME="lbAG"> </A> +<H3>Multi-Thread and Multi-Process Data Querying</H3> + +<P> + +All data queries require at least one <I>resource</I> and one <I>source</I> +to be successful. The <I>resource</I> defines a specific data service to be +queried, and the <I>source</I> is either an explicit position on the sky or +the name of an object that can be resolved to a position. Additional +parameters to the query are used to specify other options, but in essence +each data query is translated to a single URL that must be accessed by the +client task. In a complex query, lists of resource and/or objects create a +potentially large matrix of queries that must be made (i.e. +<I>N-services</I> by <I>N-objects</I> in total). Because a large fraction of +the time spent in waiting for a query to finish is in waiting for the +server to respond, we are able to run multiple queries simultaneously +without saturating our network bandwidth in most cases. +<P> + +The <B>vodata</B> task will parallelize the list of services to be queried +by running a separate processing <I>thread</I> (i.e. a lightweight process +running in parallel within the main application) for each of the services +to be called. This allows queries to different servers to be run in +parallel, and since these servers will often reside on multiple machines +the client won't impact any one data provider too badly. In addition, +the list of objects to be queried at each service will be broken up into +multiple child processes and called simultaneously. This allows, for +example, 10 objects to be queried from 3 services (a total of 30 queries) +simultaneously. +<P> + +The <I>--maxthreads=<N>mt</I> option can be used to set the max number of threads to be +created for processing the resource list (the default is 20). If the +resource list is larger than this value, the list will be processed with no +more than the max number running at any one time until all resources have +been queried. Similarly, the <I>--maxprocs=<N></I> option can be used to set the +number of child processes to be created to process the object list (the +default is 10). When setting these values it is important to remember that +the total number of <I>potential</I> processes running on your machine will +be the product of these to values. The default values were empirically found +to work reasonably well on most modern machines. +<P> + +Additionally, it is worth considering the potential strain that can be put +on data providers' machines before changing these settings. The large +majority of Cone services for example come from a single server at HEASARC +and overloading the server with hundreds of requests to multiple resources +it provides may result in a failed request and what would appear to be no +data. One should consider using the <I>-i</I> flag as a means to query a +large object list against a resource list such that only the object +processing is parallelized and the server load is minimized (See the example +below). +<P> +<P> +<A NAME="lbAH"> </A> +<H3>Output Filename Generation</H3> + +<P> + +The <I>-O</I> option may be used to specify the root part of output files +created by a data query. However, to guarantee that a multi-service, and/or +multi-object query doesn't overwrite a single output file, the filename root +will also include the <I>pid</I> (process ID) of the task that created it. +For a single service and object query no <I>pid</I> will be used as part of +the filename. This scheme guarantees unique output files across the various +processing scenarios, with similar root names for multiple files associated +with a specific query. +<P> +Output tables may be created in a number of formats and will likewise have +extensions indicating the table type. The <I>-e</I>/<I>--extract</I> option +may create additional files for each query, and the <I>-g</I>/<I>--get</I> +option to access data will similarly create additional files. The structure +of an output filename is: +<P> +<PRE> + + <I><root>[_<pid>].<extn></I> + +</PRE> + +The meaning of <I><pid></I> and <I><extn></I> have been discussed above. If +the <I>-O</I> option was set then the <I><root></I> part of the name will +simply be the argument given to set the root name. Otherwise, the <I><root></I> +element will be of the form: +<PRE> + + <I><svc>_<type>_<objname></I> + <I><svc>_<type>_<index></I> + +</PRE> + +The <I><svc></I> is derived from the service name used, the <I><type></I> is +a single-character code to indicate the type of service used ('I' for image, + +object name or the index in a list of positions of no object was specified. +<P> +<A NAME="lbAI"> </A> +<H3>Verbosity</H3> + +<P> + +The <I>-v</I> and <I>-vv</I> options serve a dual purpose: within the task they +set the level of output verbosity in terms of what is reported during +processing (Similarly, the <I>-q</I> option can be used to turn off output +reporting entirely). These flags will however also increase the value of the +<B>VERBOSE</B> parameter sent to services during a data query. The default +value is at least 1, with the highest level being 3. Using the <I>-v</I> flag +sets <B>VERBOSE=2</B> and <I>-vv</I> sets <B>VERBOSE=3</B>. +<P> + +The VERBOSE level can be important in accessing result columns that may only +be returned at the highest level. When using the <I>-m</I> +(or <I>--meta</I>) flag to print +the column metadata, the verbose options will also affect the results and it +is important that the same verbosity be set when doing the actual data +query and access. +<P> +<A NAME="lbAJ"> </A> +<H3>Bandpass and Service Type Aliases</H3> + +<P> + +The type constraint (<I>-t</I> or <I>--type</I>) accepts only the +following arguments: +<PRE> + + catalog Cone search services + image Simple Image Access services + spectra Simple Spectral Access services + table Vizier services + <literal> ResourceType from Directory record + +</PRE> + +<P> + +The bandpass constraint (<I>-b</I> or <I>--bandpass</I>) accepts only +the following arguments: +<PRE> + + Radio Millimeter Infrared (IR) + Optical Ultraviolet (UV) X-Ray (xray) + Gamma-Ray (GR) + +</PRE> + +Values in parenthese are acceptable aliases. All matches are cases +insensitive. +<P> +<P> +<A NAME="lbAK"> </A> +<H3>Range-List Parameters</H3> + +<P> + +Some parameters (for example BAND and TIME) may allow a parameter value to be +specified as a numeric range. Such range-valued parameters use the forward +slash (<I>/</I>) character as the separator between elements of the +range specification (as in the ISO 8601 date specification after which this +convention is patterned). For example, <I>5E-7/8E-7</I> would +specify a range consisting of all values from 5E-7 to 8E-7, inclusive. If a +third field is specified it is a step size for traversing the indicated +range. If a parameter permits a step size the semantics of the step size are +defined by the specific parameter. +<P> + +An open range may be specified by omitting either range value. If the first +value is omitted the range is open toward lower values. If the second value +is omitted the range is open toward higher values. Omitting both values +indicates an infinite range which accepts all values. For example, +<I>/5</I> is an open range which accepts all values less than or +equal to 5. To specify all values less than 5, <I>/4</I> would be +used (for an integer valued range). Range values are limited to numeric +values or ISO dates. +<P> + +A list may be qualified by appending the character <I>;</I> (semicolon) +followed by a qualifier string. For example <I>1E-7/3E-6;source</I> could +specify a spectral bandpass in the rest frame of the source. List and +range syntax may be combined, e.g., to indicate a list of scalar or +range-valued parameter values. Such a range list may be ordered or +unordered, and may contain either numeric or string data. An ordered +list is one which requires values to be processed in a specified order, +and to ensure this the range list is sorted or ordered by the service as +necessary before being used. It is the responsibility of the service to +sort an ordered range list, hence the client can input ranges or range +values in any order for an ordered range list and the result will be +the same. The sequence in which items in an unordered list occur on the +other hand is significant, as since there is no intrinsic ordering for +the list which can be enforced by the service, items will be processed +by the service in the order they are input by the client. +<P> + +TIME and BAND are typical examples of ordered range lists. Since a dataset +matches the query if it contains data in any of the specified ranges, +logically it does not matter in what order the ranges are given, or whether +the first element of a range is less than the second, or whether ranges +overlap; the result should be the same in all cases. Hence the range list has +an intrinsic ordering irrespective of how ranges are input. Unless otherwise +specified in the definition of a given parameter, range lists are assumed to +be ordered. +<P> +<BR> +<P> +<A NAME="lbAL"> </A> +<H2>VOCLIENT DAEMON PROCESSING</H2> + +All VO-CLI tasks are built upon the VOClient interface an rely on a +separate <I>voclientd</I> process to provide the VO functionality. The +voclientd task is distributed as part of VO-CLI and will be started +automatically by each task if it is not already running. If problems are +encountered, you may want to manually start the voclientd in a separate +window before running the task so you can monitor the output for error +messages. +<P> +<A NAME="lbAM"> </A> +<H2>RESOURCE CACHING</H2> + +Directory resolution is a common activity of VO-CLI tasks and so results +will be cached in the $HOME/.voclient/cache/regResolver directory based on +the search term, service type and bandpass parameters. Defining the +<I>VOC_NO_CACHE</I> environment variable will cause the task to ignore the +cache. +<P> +<A NAME="lbAN"> </A> +<H2>EXAMPLES</H2> + +<P> +<DL COMPACT> +<DT>1)<DD> +Query the GSC 2.3 catalog for stars a) within the 0.1 degree +default search size around NGC 1234: b) around all positions +contained in file 'pos.txt': c) for the list of objects given +on the command line: d) query a list of services for a list +of positions: e) print a count of results that would be returned +from 3 services for each position in a file: +<PRE> + + % vodata gsc2.3 ngc1234 (a) + % vodata gsc2.3 pos.txt (b) + % vodata gsc2.3 m31,m51,m93 (c) + % vodata svcs.txt pos.txt (d) + % vodata hst,chandra,gsc2.3 pos.txt (e) + +</PRE> + +<P> +<DT>2)<DD> +Query all (142) image services having data of the subdwarf galaxy +IC 10, print a count of the results only: +<PRE> + + % vodata -c -t image any IC10 + % vodata --count --type=image any IC10 + +</PRE> + +Note that we use the reserved word '<I>any</I>' for the service name and +constrain by image type. The task will automatically query the Directory to +create the list of services to be queried. +<P> +<DT>3)<DD> +Print a count of X-ray catalog data around Abell2712: +<PRE> + + % vodata -c -t catalog -b x-ray any abell2712 + % vodata --count --type=catalog --bandpass=x-ray any abell2712 + +</PRE> + +In this case we constrain both the service type as well as the spectral +coverage published for the resource in the Directory. We use the reserved +<I> +any</I>' service name to query multiple services and use the '<I>-c</I>' +flag to print a count without saving results. The object name is resolved +to coordinates internally. (Note: this example may take a while to run). +<P> +<P> +<DT>4)<DD> +Print the column metadata returned by the RC3 catalog service: +<PRE> + + % vodata --meta rc3 or vodata -m rc3 + +</PRE> + +The output will print the result using the default VERBOSE level, adding +the <I>-v</I> will set the query parameter VERBOSE=2, adding <I>-vv</I> will +set VERBOSE=3 (to print all available columns). When accessing data the +same <I>-v</I> flags will be required to retrieve columns at that +<I>VERBOSE</I> level. +<P> +<P> +<DT>5)<DD> +Use the Directory to query for resources using the search terms +"cooling flow". Upon examining the output the user notices a +Vizier paper titled "<I>Cooling Flows in 207 clusters of Galaxies</I>" +that looks interesting. Use the <B>vodata</B> task to download all +tables associated with this paper, save tables in the default +CSV format: +<PRE> + + % vodirectory cooling flow + % vodata -O white97 -all J/MNRAS/292/419/ + % vodata --output=white97 --all J/MNRAS/292/419/ + +</PRE> + +All 7 tables will be written to the current directory to files +having a root name 'white97' (chosen based on the author and +publication date). +<P> +<P> +<DT>6)<DD> +Find a suitable XMM image service, get a (brief) count of the +XMM images available for 3c273, and if there aren't too many, +download the images and save the extracted access URLs: +<PRE> + + % vodirectory -rv -t image xmm + ShortName ResourceType Title + ------------------------------------------------------.... + XMM-Newton SIAP/ARCHIVE XMM-Newton Archive .... + + % vodata -cq xmm-newton 3c273 + xmm-newton 27 I XMM-Newton Archive .... + + % vodata --count --quiet xmm-newton 3c273 + xmm-newton 27 I XMM-Newton Archive .... + + % vodata --get xmm-newton 3c273 + .... will query and download 27 images. + +</PRE> + +<P> +<DT>7)<DD> +Query for the images available from 2MASS at a given position, +extract the positions and service URLs to separate files: +<PRE> + + % vodata -e -O 2mass -t image 2mass 12:34:56.7 -23:12:45.2 + % vodata -e --output=2mass --type=image 2mass 12:34:56.7 -23:12:45.2 + +</PRE> + +The query produces files with the root name '2mass', and exten- sions of +"<I>.csv</I>" (the main response), "<I>.pos</I>" (the extracted pos- itions), +and "<I>.urls</I>" (the access references). The user inspects the files and +notices that the references return both FITS and HTML files, but she only +wants the FITS image date and uses <B>vodata</B> to download only those: +<PRE> + + % grep fits 2mass_I_001_15998.urls > images.txt + % vodata images.txt + or + % grep fits 2mass_I_001_15998.urls | vodata -i - + +</PRE> + +In both cases we pass URLs to the task which bypasses the query and directly +access the images. However, in the first case we process the entire list +and are able to take advantage of the <I>-maxdownloads=<N></I> option to +increase the number of simultaneous downloads. In the second case, the +<I>-i</I> flag causes the task to interpret each line of the input stream +as a separate command and so the data are always downloaded one at a time +(which is the default download behavior anyway). +<P> +<DT>8)<DD> +Use <B>vodata</B> as a test client for a locally-installed SIAP service: +<PRE> + + % vodata -t image -s <A HREF="http://localhost/siap.pl">http://localhost/siap.pl</A> 180.0 0.0 + % vodata --type=image --svc=<A HREF="http://localhost/siap.pl">http://localhost/siap.pl</A> 180.0 0.0 + +</PRE> + +In this case we force the ServiceURL using the '<I>-s</I>' flag, but since we +can't do a Directory query to discover what type of service this is, we must +use the '<I>-t</I>' flag to indicate it is an image service. +<P> +<P> +<DT>9)<DD> +Create a local table containing the Abell catalog. Begin with a +Directory query to find likely services using the <B>vodirectory</B> task, print +a verbose description of each resource and page the results with <I>less</I>: +<PRE> + + % vodirectory -v -v --type=catalog abell | less + +</PRE> + +The verbose results indicate a number of services with differing +requirements for what is included. We decide to use the service from +HEASARC since it contains southern hemisphere data and constraints we are +interested in. Try an all-sky query to retrieve the entire catalog, use +the service identifier to be specific about where we want to go: +<PRE> + + % vodata -e <A HREF="ivo://nasa.heasarc/abell">ivo://nasa.heasarc/abell</A> 0.0 0.0 180.0 + % vodata --extract <A HREF="ivo://nasa.heasarc/abell">ivo://nasa.heasarc/abell</A> 0.0 0.0 180.0 + +</PRE> + +We use the '<I>-e</I>' flag to extract the positions to a separate file with +a "<I>.pos</I>" extension so that we can use these in later queries. +However, the <I>.pos</I> file additionally contains the ID from the original +catalog in column 1. Strip this column so we're left with only RA and DEC +and query for Chandra observations at each position: +<PRE> + + % cut -c6- *.pos | vodata <A HREF="ivo://nasa.heasarc/chanmaster">ivo://nasa.heasarc/chanmaster</A> -p - + % cut -c6- *.pos | vodata <A HREF="ivo://nasa.heasarc/chanmaster">ivo://nasa.heasarc/chanmaster</A> --pos=- + +</PRE> + +Here we used the unix '<I>cut</I>' utility to remove the first column and +pipe the resulting positions to the task, using the '<I>-p -</I>' option to +indicate positions should be read from stding, and the IVO identifier of +the Chandra observation master log we also discovered from the Directory. +<P> +<P> +<DT>10)<DD> +Interactively query for a count of Chandra observations of Messier +objects: +<PRE> + + % vodata -cq chandra -i - + m31 + chandra 335 I Chandra X-Ray Observatory Data Archive + : : : : : + +</PRE> + +Note that by using the '<I>-i</I>' flag we execute each query as if we'd put +the object name on the command line. Using the '<I>-o</I>' flag would +instead read all of the objects from the stdin but then execute the queries +in parallel following an EOF to terminate the input. +<P> +<P> +<DT>11)<DD> +Use the <I>STILTS</I> task '<I>tpipe</I>' to select rows from a VOTable +of QSOs (made with an earlier query) where the redshift is > 0.2. Output +only the positions and pipe this to <B>vodata</B> to generate a new query to +see whether HST has observed any of these objects: +<PRE> + + % stilts tpipe ifmt=votable qso_survey.vot + cmd='select "Z > 0.2"' + cmd='keepcols "RA DEC"' | vodata -p - hstpaec + +</PRE> + +Note that we use the '<I>-p -</I>' flag to tell the task to take it's list of +positions from the piped input. The positions are used to query the HST +Planned and Archived Exposure Catalog (<I>hstpaec</I>) +<P> +<P> +<DT>12)<DD> +The user has a task called '<I>wcsinfo</I>' that takes a list of images +and outputs a text table containing the plate center and size in degrees. +Use this task as part of a query for 2MASS point sources contained in each +image: +<PRE> + + % wcsinfo *.fits | vodata 2mass-psc -i - + +</PRE> + +Here we specify the desired service (<I>2mass-psc</I>) on the commandline as +usual, and allow the remainder of the args (i.e. the position and search +size) to be read from the stdin. This allows for variable search sizes but +processes the positions serially. If the sizes are all the same (say 25 +arcmin), multiple queries can be done simultaneously using just a position +file, e.g. +<PRE> + + % wcsinfo -pos_only *.fits > centers.txt + % vodata --sr=25m 2mass-psc centers.txt + +</PRE> + +<P> +<P> +<P> +<DT>13)<DD> +Query a large list of objects against a number of ASCA resources. +Because the ASCA catalogs are served by the same machine, use the <I>'-i'</I> +option and a command file to process only each resource sequentially while +still parallelizing the object lists: +<P> +<PRE> + % cat cmds.txt + ASCA survey.tbl + ASCA\ GIS survey.tbl + ASCA\ GPS survey.tbl + : : + + % vodata -i cmds.txt +</PRE> + +</DL> +<P> + +Note that we've needed to escape the space in the resource name in some +cases. To avoid this, use of the IVO identifier for each resource is usually +preferred. +<P> +<DL COMPACT> +<DT>14)<DD> +Query the VO for GALEX data of M51. Because the <I>ShortName</I> GALEX +is not unique, we must either specify the IVO identifier of a +specific service to query, or if we're interested in results from all +supported data services with <I>galex</I> in the name: +<PRE> + + % vodata -a galex M51 + % vodata --all galex M51 + +</PRE> + +The results come from the Cone and SIAP services both called <I>GALEX</I>, +as well as an additional SIAP service called 'GALEX_Atlas'. Note that the +service names are case insensitive in either case. +<P> +<P> +<DT>15)<DD> +Process a list of hundreds of positions against the GSC2.3 catalog: +<PRE> + + % vodata gsc2.3 positions.txt + +</PRE> + +<P> +<P> +<DT>16)<DD> +Process a list of hundreds of positions against the GSC2.3 catalog, assume +that the input table has a 5 line header of text and three columns +(id,ra,dec). Note that the <I>cols</I> option requires the optional id +be specified last: +<PRE> + + % vodata --cols=2,3,1 --hskip=5 gsc2.3 positions.txt + +</PRE> + +<P> +</DL> +<A NAME="lbAO"> </A> +<H2>BUGS</H2> + +Some services don't repond properly to the metadata query and will print +a "no attributes found" message. +<P> +<A NAME="lbAP"> </A> +<H2>TODO</H2> + +<P> + +- Additional command-line options should be allowed to be specified in a +command file. +<P> + +- Support for SSAP, SAMP, TAP +<P> +<P> +<A NAME="lbAQ"> </A> +<H2>Revision History</H2> + +June 2007 - This task is new. +<A NAME="lbAR"> </A> +<H2>Author</H2> + +Michael Fitzpatrick (<A HREF="mailto:fitz@noao.edu">fitz@noao.edu</A>), June 2007 +<A NAME="lbAS"> </A> +<H2>SEE ALSO</H2> + +<A HREF="http://localhost/cgi-bin/man/man2html?1+voclient">voclient</A>(1), <A HREF="http://localhost/cgi-bin/man/man2html?1+voclientd">voclientd</A>(1), <A HREF="http://localhost/cgi-bin/man/man2html?1+vosesame">vosesame</A>(1), <A HREF="http://localhost/cgi-bin/man/man2html?1+vodirectory">vodirectory</A>(1) +<P> + +<HR> +<A NAME="index"> </A><H2>Index</H2> +<DL> +<DT><A HREF="#lbAB">NAME</A><DD> +<DT><A HREF="#lbAC">SYNOPSIS</A><DD> +<DT><A HREF="#lbAD">OPTIONS</A><DD> +<DT><A HREF="#lbAE">DESCRIPTION</A><DD> +<DL> +<DT><A HREF="#lbAF">Argument Parsing</A><DD> +<DT><A HREF="#lbAG">Multi-Thread and Multi-Process Data Querying</A><DD> +<DT><A HREF="#lbAH">Output Filename Generation</A><DD> +<DT><A HREF="#lbAI">Verbosity</A><DD> +<DT><A HREF="#lbAJ">Bandpass and Service Type Aliases</A><DD> +<DT><A HREF="#lbAK">Range-List Parameters</A><DD> +</DL> +<DT><A HREF="#lbAL">VOCLIENT DAEMON PROCESSING</A><DD> +<DT><A HREF="#lbAM">RESOURCE CACHING</A><DD> +<DT><A HREF="#lbAN">EXAMPLES</A><DD> +<DT><A HREF="#lbAO">BUGS</A><DD> +<DT><A HREF="#lbAP">TODO</A><DD> +<DT><A HREF="#lbAQ">Revision History</A><DD> +<DT><A HREF="#lbAR">Author</A><DD> +<DT><A HREF="#lbAS">SEE ALSO</A><DD> +</DL> +<HR> +This document was created by +<A HREF="http://localhost/cgi-bin/man/man2html">man2html</A>, +using the manual pages.<BR> +Time: 20:46:31 GMT, May 21, 2009 +</BODY> +</HTML> |