============================= Last Modified: Aug 10, 2012 ============================= This directory contains the VOClient command-line applications. Key files in this directory include: voApps.c The generic unix main() linked to all tasks generic.c A template code for building VOClient apps voatlas.c The VOATLAS multi-wavelength image atlas task vodata.c The VODATA general data query retrieval task voregistry.c The VOREGISTRY task to query/resolve Registry resources vosesame.c The VOSESAME name resolver vosamp.c The VOSAMP command-line SAMP tool votcnv.c The VOTCNV votable conversion tool votget.a The VOTGET task to retrieve acrefs from a votable votinfo.c The VOTINFO task to print information about a votable votpos.c The VOTPOS task to extract positional cols from votables votsort.c The VOTSORT task to sort a votable based on a column votstat.c The VOTSTAT task to print colum statistics Planned tasks Not Yet Implemented: votcat.c // VOTable Resource concatenator votjoin.c // VOTable inner joins votselect.c // Select rows/cols by index/expr votsplit.c // Split multi-resource votables VOClientd // C-based minimal implementation of VOClient Daeomon hub // C implementation of SAMP Hub voxmatch // Cross-compare local table and VO data vosput // Put files to a VOSpace vosget // Get files from a VOSpace vosmove // Move files/nodes between VOSpaces voslist // List files/nodes in a VOSpace vosdelete // Delete files/nodes in a VOSpace voskybot // List known moving objects in an image The 'lib' subdirectory contains support code for the applications. Of particular interest is the 'lib/voTask.c' code that defines the tasking interface used by the applications. Task Interface -------------- All VOClient apps are written as C procedures sharing a common signature: (int argc, char **argv, size_t *len, void **result); where 'argc' and 'argv' have the usual meaning for the CLI argument vector, 'len' is the length of any returned object (or 0 if there is no return), and 'result' is a pointer to the returned object in memory. The task() procedure returns 0 for success, and a positive value to indicate an error return (this may include a signal number for a crashed child process as well as task-specific error return codes). Tasks share a common main() responsible only for passing in the parameter options, executing the task as a connected subprocess (or by calling the procedure directly if the VOAPP_CONNECTED environment variable is defined, this is to aid in debugging) and for retrieving any returned result object. This architecture ensures that any memory allocated or threads spawned by the VOClient app do not interfere with an app using the language API. Host Process App Sub-Process +----------+ +----------------------+ | C | parameters | | | main() |--------------->| VOClient | | or |<---------------| application code | | Lang API | result | | +----------+ +----------------------+ The application is spawned as a child process by the main() and the CLI parameters are sent over IPC. Likewise, any return value from the app is sent back via IPC, a host CLI tool would simply discard this result however a language binding or other API would then have a pointer to the result object it could pass up to the calling interface. The details of what can be returned by a task is task-specific and the caller must be aware of the options to be of use, for this discussion the returned 'result' is a memory object of size 'len' given the above calling signature. The 'voApps.c' code contains the common main() function which normally calls the vo_runTask() procedure to execute the task based on the task name derived from a lookup table. (See lib/voTask.c for this procedure). Parameter Interface ------------------- VOClient apps are written to accept CLI/parameter options in a number of formats suited for use as Unix tasks or from an API that may be better suited to pass in "=" strings. The following formats for option strings are supported by all tasks: -p param[+-] --param --param=value param= For CLI tasks the following would be equivalent: % task -p # to set the 'p' option % task --param # long-form of 'p' option % task --param+ # if 'param' is a boolean option % task --param=value # to set a specific value for param % task --param value # to set a specific value for param % task param=value # to set a specific value for param From an API one could imagine these same parameters being passed in to form the required argument/parameter 'argv' vector as e.g. from VOClient import tasks as voc status = voc.task ("param=value", ....) Accomodating a return value from the task would affect this sort of interface (e.g. if the task were to return an in-memory FITS file instead of a status code) but the idea is to support both standard CLI options as well as a p=v format more appropriate for a tasking API interface. Common Options/Parameters: All tasks(*) support a core set of options, namely -h,--help Print a task usage summary and examples -%,--test Run unit tests (some tasks require ) -r,--return Return optional result object All other options are task-specific however an effort has been made to make these consistent across tasks (e.g. a "--input" for input files, "--fmt" for output VOTable formats, "--samp" for SAMP functionality, etc). The "--help" (or "-h") option to each task should provide enough runtime documentation to get started with using a task, in particular the examples given in the help are meant to always be valid. The "--test" (or "-%") option is a way to run unit tests on a particular task, primarily based on the examples given in "--help". Depending on the task an input file/arguments of some form must be provided for the tests to work. For example, unit tests of VOTable tools can be run using votables returned from various data services to ensure proper behavior for all VO resources or to test operation against various (non)compliant files. This same mechanism can be used to build a regression test suite against a static list of VO Resources or standardized data files. The "--return" option is task-specific and its use will depend largely on the API implementing it. A detailed description of the return objects will be provided in the task/API documentation. (*) Exceptions as of this writing are the original VO-CLI tasks VODATA/VOSESAME/VODIRECTORY, these will be implemented before release. SAMP Interoperability --------------------- The VOSAMP task a CLI tool for sending messages to other SAMP-enabled applications, either as a one-off CLI execution or from within a scripted environment. To hide the SAMP details, the CLI interface is written to be more user-friendly, e.g. the options(*) on the command line are (in part): load load the named image/table file loadImage load the named image loadVOTable load the named VOTable loadFITS load the named FITS bintable showRow [] [tblId] highlight specified row selectRows [] [tblId] select specified rows bibcode load the named bibcode exec execute a client command pointAt point at given coords setenv set an environment value getenv get an environment value setparam set a parameter value getparam get a parameter value send [ ...] generalized message send where "" might be an actual HTTP reference, a local file path/name, or a 'file://' URI. The commands themselves are cases insensitive and the VOSAMP help page should be consulted for details on sending non-standard messages using the 'send' option for general messages (i.e. those messages that don't conform to an IVOA-supported 'mtype' but which might be used in a custom workflow). When VOSAMP is called using an API these commands are passed through the argument string an provide a trivial high-level method of sending SAMP messages without requiring the app to explicty connect to the Hub (receiving messages will still normally require the app to register itself in order to handle messages using the application's callbacks). A "normal" SAMP-enabled application will establish a connection to the local desktop apps once and then process subsequent messages for the life of the application. In a CLI tool, this model introduces the overhead of the initial connection (several seconds) on each command, an effect which compounds when a CLI tool is used from within a script that might be processing many tens-to-thousands of messages as it is used in some user-defined workflow. To avoid this overhead, the VOSAMP task will default to become a bacakground 'proxy' service that lingers after the initial command is executed, i.e. in the same way a 'sudo' command won't require a password for each command for some time after the initial call, VOSAMP will background itself after the first command to maintain its connection to SAMP Hub, subsequent calls to VOSAMP will simply forward the command from the CLI to this proxy to make use of the existing SAMP connection and execute as quickly as any other persistant SAMP task. (*) SSA message data are not yet implemented. Inter-Desktop Messaging: VOSAMP reads its command either from stdin, a named command file (e.g. as from a VOSAMP-shell interpreter), or from a socket created when the proxy process is created. This socket is created on an inet port that may be visible to the whole internet, or on a private port restricted by local system administrators to trusted clients (e.g. port 8080 if that is a general service provided by a site, or the default port 4000 for sites willing to open a firewall hole). An example sequence of SAMP commands would be something like % vosamp listClients % vosamp loadVOTable foo.xml where the first command establishes a SAMP connection, and the second would forward the CLI command to the (already running) proxy client without establishing a new-application context. In this model, a proxy VOSAMP app reading from an inet socket opens the possibility of using this proxy from a remote host that can send a command from a trusted machine. For example, On host A: % vosamp start # start proxy client on host A (140.252.1.86) On host B: % vosamp --proxy=140.252.1.86:4000 loadVOTable foo.xml where the '140.252.1.86' is the machine running the intial VOSAMP task, '4000' is the inet port that proxy is reading commands, and the remainder of the commandline are args to be passed thru as it they were issued from the local host. Local data (e.g. the 'foo.xml' file) is sent to the remote before executing the command, file:// URL's are rewritten so the file reference remains valid on the remote machine (http:// URLs are unchanged). SAMP Session Manager: (In development) By using the proxy client, we can send commands to a VOSAMP application from a remote machine, but this is not quite the same as federating two (or more) desktops in a shared session. We are also limited by the ability to bypass firewalls in order to connect to the proxy client. The solution then is to have a public "session manager" service that serves as an alternate input source to the proxy client and will acto to forward message from one machine to the others. For example, +-------------+ | Session Mgr | +-------------+ ^ / \ / \ +-------------------------+ +-------------------------+ | Topcat \ | | / Topcat | | Hub - VOSAMP | | VOSAMP - Hub | | Aladin / | | \ IRAF | +-------------------------+ +-------------------------+ Host A Host B Since the session manager is on a public host, the VOSAMP on each machine makes an outgoing client connection and sets up that socket to receive commands and subscribes to all message types so that SAMP messages received on the local machine can be forwarded back to the session manager and then on to other machines in the session. For example, the Aladin on Host A broadcasts an image.load.FITS message, the VOSAMP on Host forwards the message to the session manager than then passes it on to the VOSAMP on Host B for rebroadcast. In this way the message is seen on both desktops transparently. When local data is being used, this is uploaded to the session manager for storage in a web-accessible area, the forwarded message is then rewritten to use the URL and is accessed only when needed. Sessions are created/joined with no special setup required, e.g. % vosamp --session=foo e.g. commandline tool or voc.vosamp ("session=foo") e.g. from language API The VOSAMP in this case would contact the session manager to join the list of machines in session 'foo', this session would be created if it did not already exist and sessions quietly end when the VOSAMP proxies timeout due to inactivity or are shut down explicitly. There is no need for formal security since it is up to the parties in the session to agree on and share the session name of their choosing. Applications written using the language bindings can participate in sessions by simply calling the VOSAMP task to create the proxy regardless of whether it is used explicitly for messaging. If the VAO were to operate a public session manager service, it would trivial to log its use for reporting to the funding agencies, and would meet the mandate of VAO providing tools for community use. Details of the communication protocol could be written as an IVOA note to allow other implementations to make use of the service, or it could be formalized into a next version of the SAMP protocol itself. [Status: Proxy clients for use on the desktop and between trusted machines is implemented, development of the session manager and extensions to VOSAMP will be ready for demonstration at the Seattle meeting. (8/11/12)]