path: root/vendor/voclient/voapps/README

=============================
Last Modified:   Aug 10, 2012
=============================


	This directory contains the VOClient command-line applications.
Key files in this directory include:

    voApps.c		The generic unix main() linked to all tasks
    generic.c		A template code for building VOClient apps
    
    voatlas.c		The VOATLAS multi-wavelength image atlas task
    vodata.c		The VODATA general data query retrieval task

    voregistry.c	The VOREGISTRY task to query/resolve Registry resources

    vosesame.c		The VOSESAME name resolver

    vosamp.c		The VOSAMP command-line SAMP tool

    votcnv.c		The VOTCNV votable conversion tool
    votget.a		The VOTGET task to retrieve acrefs from a votable
    votinfo.c		The VOTINFO task to print information about a votable
    votpos.c		The VOTPOS task to extract positional cols from votables
    votsort.c		The VOTSORT task to sort a votable based on a column
    votstat.c		The VOTSTAT task to print colum statistics


    Planned tasks Not Yet Implemented:

    votcat.c		// VOTable Resource concatenator
    votjoin.c		// VOTable inner joins
    votselect.c		// Select rows/cols by index/expr
    votsplit.c		// Split multi-resource votables

    VOClientd		// C-based minimal implementation of VOClient Daeomon
    hub			// C implementation of SAMP Hub

    voxmatch		// Cross-compare local table and VO data

    vosput		// Put files to a VOSpace
    vosget		// Get files from a VOSpace
    vosmove		// Move files/nodes between VOSpaces
    voslist		// List files/nodes in a VOSpace
    vosdelete		// Delete files/nodes in a VOSpace

    voskybot		// List known moving objects in an image



The 'lib' subdirectory contains support code for the applications.  Of
particular interest is the 'lib/voTask.c' code that defines the tasking
interface used by the applications.


Task Interface
--------------

    All VOClient apps are written as C procedures sharing a common signature:

	<task> (int argc, char **argv, size_t *len, void **result);

where 'argc' and 'argv' have the usual meaning for the CLI argument vector,
'len' is the length of any returned object (or 0 if there is no return),
and 'result' is a pointer to the returned object in memory.  The task()
procedure returns 0 for success, and a positive value to indicate an error
return (this may include a signal number for a crashed child process as
well as task-specific error return codes).

    Tasks share a common main() responsible only for passing in the
parameter options, executing the task as a connected subprocess (or by
calling the procedure directly if the VOAPP_CONNECTED environment variable
is defined, this is to aid in debugging) and for retrieving any returned
result object.  This architecture ensures that any memory allocated or 
threads spawned by the VOClient app do not interfere with an app using
the language API.

      Host Process                    App Sub-Process
      +----------+                +----------------------+
      |    C     |   parameters   |                      |
      |  main()  |--------------->|      VOClient        |
      |    or    |<---------------|   application code   |
      | Lang API |     result     |                      |
      +----------+                +----------------------+

The application is spawned as a child process by the main() and the CLI
parameters are sent over IPC.  Likewise, any return value from the app is
sent back via IPC, a host CLI tool would simply discard this result however
a language binding or other API would then have a pointer to the result
object it could pass up to the calling interface. The details of what can
be returned by a task is task-specific and the caller must be aware of the
options to be of use, for this discussion the returned 'result' is a memory
object of size 'len' given the above calling signature.

    The 'voApps.c' code contains the common main() function which normally
calls the vo_runTask() procedure to execute the task based on the task name
derived from a lookup table.  (See lib/voTask.c for this procedure).



Parameter Interface
-------------------

    VOClient apps are written to accept CLI/parameter options in a number
of formats suited for use as Unix tasks or from an API that may be better
suited to pass in "<param>=<value>" strings.  The following formats for
option strings are supported by all tasks:

         -p   param[+-]   --param   --param=value   param=<value>

For CLI tasks the following would be equivalent:

	% task -p			# to set the 'p' option
	% task --param			# long-form of 'p' option
	% task --param+			# if 'param' is a boolean option
	% task --param=value		# to set a specific value for param
	% task --param value		# to set a specific value for param
	% task param=value		# to set a specific value for param

From an API one could imagine these same parameters being passed in to form
the required argument/parameter 'argv' vector as e.g.

	from VOClient import tasks as voc
	status = voc.task ("param=value", ....)

Accomodating a return value from the task would affect this sort of interface
(e.g. if the task were to return an in-memory FITS file instead of a status
code) but the idea is to support both standard CLI options as well as a p=v
format more appropriate for a tasking API interface.


Common Options/Parameters:

    All tasks(*) support a core set of options, namely

	-h,--help		Print a task usage summary and examples
	-%,--test <input>	Run unit tests (some tasks require <input>)
	-r,--return <opt>	Return optional result object

All other options are task-specific however an effort has been made to make
these consistent across tasks (e.g. a "--input" for input files, "--fmt" for
output VOTable formats, "--samp" for SAMP functionality, etc).

    The "--help" (or "-h") option to each task should provide enough runtime
documentation to get started with using a task, in particular the examples
given in the help are meant to always be valid.

    The "--test" (or "-%") option is a way to run unit tests on a particular
task, primarily based on the examples given in "--help".  Depending on the
task an input file/arguments of some form must be provided for the tests to
work.  For example, unit tests of VOTable tools can be run using votables
returned from various data services to ensure proper behavior for all VO 
resources or to test operation against various (non)compliant files.  This
same mechanism can be used to build a regression test suite against a static
list of VO Resources or standardized data files.

    The "--return" option is task-specific and its use will depend largely
on the API implementing it.  A detailed description of the return objects
will be provided in the task/API documentation.


    (*)  Exceptions as of this writing are the original VO-CLI tasks
	 VODATA/VOSESAME/VODIRECTORY, these will be implemented before
	 release.



SAMP Interoperability
---------------------

    The VOSAMP task a CLI tool for sending messages to other SAMP-enabled
applications, either as a one-off CLI execution or from within a scripted
environment.  To hide the SAMP details, the CLI interface is written to be
more user-friendly, e.g. the options(*) on the command line are (in part):


   load <url>                            load the named image/table file
   loadImage <url>                       load the named image
   loadVOTable <url>                     load the named VOTable
   loadFITS <url>                        load the named FITS bintable
   showRow [<url>] <row> [tblId]         highlight specified row
   selectRows [<url>] <rows> [tblId]     select specified rows
   bibcode <bibcode>                     load the named bibcode

   exec <cmd>                            execute a client command
   pointAt <ra> <dec>                    point at given coords
   setenv  <name> <value>                set an environment value
   getenv  <name>                        get an environment value
   setparam <name> <value>               set a parameter value
   getparam <name>                       get a parameter value

   send <mtype> [<args> ...]             generalized <mtype> message send


where "<url>" might be an actual HTTP reference, a local file path/name, or
a 'file://' URI.  The commands themselves are cases insensitive and the
VOSAMP help page should be consulted for details on sending non-standard
messages using the 'send' option for general messages (i.e. those messages
that don't conform to an IVOA-supported 'mtype' but which might be used in
a custom workflow).  When VOSAMP is called using an API these commands are
passed through the argument string an provide a trivial high-level method
of sending SAMP messages without requiring the app to explicty connect to
the Hub (receiving messages will still normally require the app to register
itself in order to handle messages using the application's callbacks).

    A "normal" SAMP-enabled application will establish a connection to the
local desktop apps once and then process subsequent messages for the life
of the application.  In a CLI tool, this model introduces the overhead of
the initial connection (several seconds) on each command, an effect which
compounds when a CLI tool is used from within a script that might be
processing many tens-to-thousands of messages as it is used in some
user-defined workflow.  To avoid this overhead, the VOSAMP task will
default to become a bacakground 'proxy' service that lingers after the
initial command is executed, i.e. in the same way a 'sudo' command won't
require a password for each command for some time after the initial call,
VOSAMP will background itself after the first command to maintain its
connection to SAMP Hub, subsequent calls to VOSAMP will simply forward the
command from the CLI to this proxy to make use of the existing SAMP
connection and execute as quickly as any other persistant SAMP task.

(*)  SSA message data are not yet implemented.


Inter-Desktop Messaging:

    VOSAMP reads its command either from stdin, a named command file (e.g.
as from a VOSAMP-shell interpreter), or from a socket created when the proxy
process is created.  This socket is created on an inet port that may be
visible to the whole internet, or on a private port restricted by local
system administrators to trusted clients (e.g. port 8080 if that is a
general service provided by a site, or the default port 4000 for sites
willing to open a firewall hole).

    An example sequence of SAMP commands would be something like

	% vosamp listClients
	% vosamp loadVOTable foo.xml

where the first command establishes a SAMP connection, and the second would
forward the CLI command to the (already running) proxy client without
establishing a new-application context.

    In this model, a proxy VOSAMP app reading from an inet socket opens
the possibility of using this proxy from a remote host that can send a
command from a trusted machine.  For example,

   On host A:
	% vosamp start		# start proxy client on host A (140.252.1.86)

   On host B:
	% vosamp --proxy=140.252.1.86:4000 loadVOTable foo.xml

where the '140.252.1.86' is the machine running the intial VOSAMP task,
'4000' is the inet port that proxy is reading commands, and the remainder
of the commandline are args to be passed thru as it they were issued from
the local host.  Local data (e.g. the 'foo.xml' file) is sent to the remote
before executing the command, file:// URL's are rewritten so the file
reference remains valid on the remote machine (http:// URLs are unchanged).


SAMP Session Manager:	(In development)

    By using the proxy client, we can send commands to a VOSAMP application
from a remote machine, but this is not quite the same as federating two (or
more) desktops in a shared session.  We are also limited by the ability to
bypass firewalls in order to connect to the proxy client.  The solution then
is to have a public "session manager" service that serves as an alternate
input source to the proxy client and will acto to forward message from one
machine to the others.  For example,

			    +-------------+
                            | Session Mgr |
			    +-------------+
                                   ^
                                 /   \
                                /     \
    +-------------------------+         +-------------------------+
    |  Topcat  \              |         |               /  Topcat |
    |            Hub - VOSAMP |         |  VOSAMP - Hub           |
    |  Aladin  /              |         |               \  IRAF   |
    +-------------------------+         +-------------------------+
               Host A                            Host B

Since the session manager is on a public host, the VOSAMP on each machine
makes an outgoing client connection and sets up that socket to receive
commands and subscribes to all message types so that SAMP messages received
on the local machine can be forwarded back to the session manager and then
on to other machines in the session.  For example, the Aladin on Host A
broadcasts an image.load.FITS message, the VOSAMP on Host forwards the
message to the session manager than then passes it on to the VOSAMP on
Host B for rebroadcast.  In this way the message is seen on both desktops
transparently.  When local data is being used, this is uploaded to the 
session manager for storage in a web-accessible area, the forwarded message
is then rewritten to use the URL and is accessed only when needed.

    Sessions are created/joined with no special setup required, e.g.

	% vosamp --session=foo			e.g. commandline tool
or	
	voc.vosamp ("session=foo")		e.g. from language API

The VOSAMP in this case would contact the session manager to join the list
of machines in session 'foo', this session would be created if it did not
already exist and sessions quietly end when the VOSAMP proxies timeout due
to inactivity or are shut down explicitly.  There is no need for formal
security since it is up to the parties in the session to agree on and share
the session name of their choosing.  Applications written using the language
bindings can participate in sessions by simply calling the VOSAMP task to
create the proxy regardless of whether it is used explicitly for messaging.

    If the VAO were to operate a public session manager service, it would
trivial to log its use for reporting to the funding agencies, and would 
meet the mandate of VAO providing tools for community use.  Details of
the communication protocol could be written as an IVOA note to allow other
implementations to make use of the service, or it could be formalized into
a next version of the SAMP protocol itself.

[Status:  Proxy clients for use on the desktop and between trusted machines
    is implemented, development of the session manager and extensions to 
    VOSAMP will be ready for demonstration at the Seattle meeting.  (8/11/12)]