diff options
Diffstat (limited to 'vo/Notes')
| -rw-r--r-- | vo/Notes | 435 |
1 files changed, 435 insertions, 0 deletions
diff --git a/vo/Notes b/vo/Notes new file mode 100644 index 00000000..2d98629d --- /dev/null +++ b/vo/Notes @@ -0,0 +1,435 @@ + + + NVO Package -- Current Status and Planned Work + ============================================== + Initial Draft: 10/11/06 + + + This document describes the current status (and possible future +work) of the NVO external package being developed as part of the NVO Summer +School Software (NVOSSS) distribution. This project represents the +client-side integration of IRAF with the NVO and is built on a VOClient +interface being developed in parallel (and in collaboration with Doug Tody). +The external package can be released indepedently and contains all the needed +components, however the installation process will require more steps than +most other packages. + + Development is constrained by a 12/1/06 deadline for a code freeze +of the NVOSSS. This affects primarily the VOClient interface since the +IRAF distribution installed by the NVOSSS remains on an outside server and +can be updated independently. What this means for the NVO package is that +only those VO services currently interfaced by VOClient are acccessible +for task development presently, however it is expected the NVOSSS will be +updated regularly and the subset of NVOSSS code needed for the NVO package +can be included in the distribution and developed outside the scope of the +NVOSSS (i.e. you won't need NVOSSS to run the package, we can update the +package (inc. a new VOClient) at will). + + +Current Status +============== + +The project has three main components described below: + + + VOClient -- Multi-Language client-side VO access + ------------------------------------------------ + + The VOClient code consists of the following: + + voclientd - The daemon process used as a server to the interface. + This is just a shell command for invoking the java code. + voclient.jar - Jar file of VOClient classes implementing the daemon + Client API - A C-based API that invokes the interface methods in the + voclient.jar file. Bindings for Fortran, SPP and Python + will be part of the initial release, work on automatic + builds for other languages (Perl, Ruby, Java, etc via + SWIG) are planned however these languages may be used + today after a platform-specific build process. + Example Code - The clientapi contains several dozen example tasks in + a variety of languages. + + The daemon and interface support a number of VO interfaces needed to + develop applications: + + Registry Query + Data Access (Cone and Siap) + SkyBoT Service interface + Name-to-Coordinate Resolution (Sesame web service) + + + + VO-CL -- Enhanced IRAF CL with new VO builtin functions + ------------------------------------------------------- + + + The VO-CL adds a number of new builtin functions for querying the + registry, data access, and VOClient control. Hi-level procedures are + summarized below: + + VO Client Library: + + stat = initVOClient (opts) + stat = closeVOClient (quit_flag) + stat = restartVOClient (quit_flag) + stat = validateObj (obj) + stat = vocReady () + + DAL Service Interface: + + qres = dalConeSvc (url, ra, dec, sr) + qres = dalSiapSvc (url, ra, dec, rsize[, dsize[, fmt]]) + count = dalRecordCount (qres) + stat = dalGetData (qres, recnum, fname) + val = dalGet[Str|Int|Dbl] (qres, attrname, recnum) + + fname = getData (url, [fname]) + + Registry Search Interface: + + str = regResolver (shortName,[svctype[,attr[,index]]]) + N = nresolved () + + resource = regSearch (term [, orValues]) + resource = regSearch (keywords, orValues) + resource = regSearch (sql, keywords, orValues) + resource = regSvcSearch (svcType, searchTerm, orValues) + + count = regResultCount (resource) + str = regValue (resource, attr_list, resIndex) + + + (The low-level interface is not shown here.) The interface described + is sufficient for developing most higher level science applications (and + indeed is what's used to develop most of the package tasks), providing + means to find data resources in the Registry and routines needed to query + a resource and step through the result in a CL script. In most cases + however, users will develop scripts using the toolbox tasks available in + the package. + + + + NVO Package -- Current Tasks + ---------------------------- + + The NVO package tasks are currently all at a single level with the + idea this would be restructured to provide user applications as well as a + sub-package of 'toolkit' tasks to be used in script development. The + majority of tasks are CL scripts built with the new VO-CL interface + above, however a few low-level compiled SPP tasks exist in the toolkit. + The VOClient jar (library) file as well as other Java components + (specifically the STILTS tools needed for VOTable processing) are + distributed with the package and appropriate foreign task declarations + are made in the NVO package-loading script. Similarly, the VOClient + sources as well as the VO-CL itself will be part of the package + distribution. + + To reduce the complexity of the installation, we see the package as + being a multi-architecture binary release rather than pure source (the + GUIAPPS package was likewise distributed with all supported binaries). + The only external dependency is that the user have Java 1.4+ available on + the machine. + + The contents of the current package are given below. For the purpose + of this document assume the package restructuring has been done as + described. A '*' indicates that a help page for the tasks currently + exists, note that a separate "NVO Script Programmer's Guide" (separate + from the VOClient programmer's doc) still needs to be written as well. + + + Toplevel apps: + + * datascope - Toy DataScope application + * registry - Query the NVO Registry + * skybot - Sky Bodies Tracker -- Minor planet locator service + * sloanspec - Browse SDSS spectra + getspec - Get one or more SDSS spectra to local FITS files + + + Toolbox sub-package: + + * conecaller - Call a Cone Search (catalog access) service + * rawcaller - Access a raw URL, optionally converting tabular output + * siapcaller - Call an SIAP (image access) service + * sesame - Sesame name resolver service + * vizier - Download one or more VizieR tables + + nedoverlay - Overlay NED objects in the image display + obslogoverlay - Overlay an observation catalog (HST, XMM, etc) + + tabclip - Clip a table to given boundaries + tabout - Output/convert a VOTable to various formats + + imwcsinfo - Summarize the WCS information of an image + + (Hidden Tasks) + prettystr - Pretty-print a string + regmetalist - List the Registry Metadata (column names) + sbquery - SkyBot query utility task + ssquery - Sloan Spectra query utility task + stilts - Starlink Tables Infrastructure Library Tool Set + + + + +Future Development +================== + + In the following table we summarize possible future work on the +package. This list probably also includes the listed toolbox tasks and maybe +one or more of the new toplevel apps (depending on identified priorities) and +may/should be supplemented by the scientific staff, Frank and/or Andrew would +be good people to implement them. I note that someting IR-specific is +missing, there may be a piece of the NEWFIRM pipeline that could use VO +resources and implemented here as well. + + Items marked in the 'User Need' column refer to work that must be +done for a "proper" user release of the package as a Christmas present. The +remaining items (incomplete as they are) would fill out proposed work for the +FY07 NVO grant. Deployment of simple web services is possible with a few +weeks effort, + + + Component Work Item Est Time User Need? + --------- --------- -------- ---------- + + VOClient Writing a detailed Programmer's Guide 1-2 weeks Yes + Improved error handling/trapping 2 days Yes + API calls to "free" objects in daemon + HashMap (could also be done using a + different data structure to limit size) 1 day Yes + + Restructure VOClient java code for future 3-4 days No + scalability + Switch to Registry 1.1. spec 3-4 days No + New DAL interfaces in API and daemon: + SSAP 2-3 days No + ADQL 1-2 days No + VOSpace 5 days No + Footprint Services 1-2 days No + Client-side i/f to Skynode crossmatch svc 5 days No + Auto-build of SWIG bindings 2 weeks No + VOClient logging and async messaging 3 days No + Additional multi-language example tasks 2 days No + + Registry integration in dalclient for + service discovery ??? No + Support for multiple services in dalclient + connection context ??? No + Query merge capability for DAL discovery + queries ??? No + + + VO-CL Bug fixes as they come up <1 day Yes + "NVO Script Programmer's Guide" 1-week Yes + + New DAL interfaces as they are implemented 1-day No + + + + NVO Pkg Help Pages for all tasks 4 days Yes + Minor structure changes to give all apps + a common i/f (local image, ObjName, Pos) 1 day Yes + Improved handling of NULL query returns 1 day Yes + + New toplevel apps (described below) 1-5 day ea. ??? + New toolbox tasks (described below) <1 day ea. ??? + + + + New Toplevel Science Apps: + -------------------------- + + ccdiagram -- Draw color-color diagram of sources in image/list + colselect -- Find the objects in a field w/ given color properties + finderchart -- Produce a finder chart for a region of the sky + hrdiagram -- Draw HR diagram of sources in image/list + multiview -- Multi-wavelength image/object display + sedbuilder -- Build an SED for a named object + + wcsfixer -- Compute or improve the WCS of an image + + Notes: + + CCDIAGRAM - Given an image footprint, sky position, or list of sources + computes a color-color diagram of objects. Uses SKYPORTAL + to get colors from selected catalogs and xmatch sources, + result table can be filtered with local xmatch against a + list of sources and an interactive loop lets the user + change the axes being displayed. If there's time, a grid + of multiple c-c diagrams could be displayed + + HRDIAGRAM - Special case of CCDIAGRAM, same/similar features. + COLSELECT - Given an image footprint or sky position use SKYPORTAL to + find objects with specified color properties (e.g. very + red, detected in one band but not another, etc). Marks + the resulting objects in the display and prints the table. + WCSFIXER - Desktop version of the webapp, able to process multiple + images in batch mode. Code needs only minor updating and + modification + FINDERCHART - Combines various overlay elements to produce and image + display marked with all sorts of interesting extra data + derived from VO sources. Can use an existing image or + DSS image of the field. + MULTIVIEW - Given an entire image or object/radius downloads cutouts + from various all-sky (or pointed observation) SIAP services + and loads them in the image display for blinking. The task + automatically registers the images, user selects image + services and passbands (e.g. 2MASS HJK, SDSS filters for + optical, NVSS or FIRST, etc). + SEDBUILDER - Uses SKYPORTAL to query skynodes for data points and + probably a new custom task to fit the data to get a + bolometric luminosity. + + Enhancements to Existing Tasks: + + DATASCOPE - Adding a multiple-connection context to the DAL interfaces + would mean we could query services much more quickly in + parallel. This would open the possibility of an interactive + mode for the task that could be used drill down into the + data visually, e.g. query observing log sources, overlay + the footprints and select image previews from the + associated SIAP service. + SLOANSPEC - The SKYPORTAL client would allow us the query the SDSS + tables for specific line information and could be used to + provide either advanced search capability in this task or + a new task altogether. What we get back would be a table + of spectra that could be downloaded as well as line + intensities (redshift, positions, etc). This would allow + a science app to be written that automatically plots e.g. + specific line ratios for a large sample of data (as when + looking for AGN, starforming galaxies, etc). + + + New Toolbox Tasks: (Est. less than a day to build each) + ------------------ + + imagecat - Create a catalog of objects in an image + skyportal - Get raw result of an ADQL query to OpenSkyQuery + + 2massoverlay - Overlay 2MASS PSC or XSC objects in the image display + nvssoverlay - Overlay NRAO VLA Sky Survey objects in the image display + radiocontour - Overlay radio contours (NVSS or FIRST) on the display + + tfootprint - Compute the bounding box (footprint) of a table + taboverlay - Overlay a generic table on the image display + + + Notes: + IMAGECAT - The tasks exists as part of the WCSFixer but would be + rewritten to use the VOClient interfaces. + SKYPORTAL - This is a Java foreign task that passes a generic + ADQL query on to OpenSkyQuery and returns the raw VOTable + result. The idea is that the calling script would have + to construct a meaning query based on its purpose, this + simply executes the query. + 2MASSOVERLAY + NVSSOVERLAY + RADIOCONTOUR - More of the same kind of catalog overlay utilities but + with named data sources for convenience. + + TFOOTPRINT - Like IMWCSINFO but computes the bounding box/circle that + covers all points in a table given the (x,y) column to be + used. Updates its parameters with center, corner, radius + TABOVERLAY - Computes the (x,y) display coords for named columns in a + table given an image wcs, overlays points, can optionally + display image as well. Tasks like NEDOVERLAY built w/ this + + + +Example Science Projects +======================== + +[NOTE: This section needs a lot more work by people who know what they're + talking about. Example science cases should be outlined here and + I can fill in gaps about how it might be done, I'm hoping that + the F2F will help complete this section.] + +1) Search for galaxies with suppressed star formation. This idea come from + a Gemini result announced at + + http://www.gemini.edu/index.php?option=content&task=view&id=201.html + + Basically, a sample of 20 galaxies with 2.0 < z < 2.7 and Kmag < 19.7 + were observed w/ GNIRS and 45% were found to not be forming stars (i.e. + no emission features). At that redshift, the Ha line is in the K band + but we can use the principles here to look at other redshift ranges using + a much larger sample. + + Using the NVO package toolbox a similar experiment could be done on + the larger SDSS spectra sample as follows: + + - Use FINDSPEC to create a table of SDSS spectra with the required + redshift range (gives us position of objects) + + - Query the 2MASS XSC at points around each SDSS object to select + those where the K magnitude is brighter than some desired cutoff + (gives us a sample of galaxies meeting the criteria). Likewise, + the SDSS catalog could similarly be queried and the ugriz magnitudes + used as the selector. + + - Use GETSPEC to download the spectra to local disk. + + - Use IDENTIFY to find spectra with no emission features + + With a SKYPORTAL client much of this could be done with a clever SQL + query. + +2) + +3) + + + +VO/IRAF AS A DPP ARCHITECTURE COMPONENT - Initial Thoughts +=========================================================== + + Current and planned DPP/E2E activities have various elements referred +to as the "VOI", however this is not in itself a concrete software component +and in an SOA environment this miscellaneous functionality introduces +complexity in other components or leads to redundant implementations. For +example, a VOSpace has both client and server-side uses in E2E, but managing +the VOSpace itself shouldn't be part of a Portal doing presentation nor does +it belong to the archive. Likewise, pipelines may need access to reference +catalogs in the VO and Portal may wish to provide access to these data as +well, but each side of the E2E would implement their own catalog clients. + + My limited (and no doubt outdated) understanding of the E2E project +is that the VOI has not yet been completely defined, and a definition of the +analysis functionality that may be needed is incomplete as well. Mosaic and +cutout services are just two examples of the types of processing that might +be called, however as the number of needed services increases the need to put +this functionality into a more centralized service becomes clear. As new +capabilities in Portal evolve (e.g. do XYZ to all images in my VOSpace) the +model of passing images one-at-a-time to other web-based analysis tasks +becomes unsustainable as well. + + The NVO-funded software development has produced both a demonstrated +framework for deploying legacy software as a web-service and a multi-language +VO client-side interface. Using this work, and an underlying IRAF system as +the processing engine, much of the functionality described as the VOI and +many of the analysis tasks needed could be implemented as a more centralized +service in the E2E system and the SOA architecture. As a multi-language API +the VOClient can provide a common interface and functionality to all parts of +E2E, realizing development savings by eliminating redunant implementations +and providing a single-point update/release as new capabilities are added. +Analysis tasks developed in the NVO package can be exposed as web-services to +the outside community as well as be used within the portal environment. +Likewise, new higher-level tasks can be developed in any of the many +languages used throughout E2E without needing to replicate much of the +infrastructure needed by VO (e.g. VOTable parsing, web-service calls, +protocol updates, etc). This also provides a clearer separation between what +is clearly VO from other components meant to provide storage, data transfer, +user-presentation, or standard reductions. + + From a user perspective, this also makes NSA data much more +attractive. Turning Portal into an full-blown analysis platform itself will +require an immense amount of work, however making the NSA data trivial to use +within scientist-written tasks and in a well-established analysis environment +will mean more people are likely to use the data products offered. The NVO +package has the potential to get many more scientists using VO data in their +everyday work. Tasks developed for the NVO package that can simultaneously +reach out to the VO for data and be called as a service itself, and which +are well-integrated with portal services, help make NOAO a destination for +VO data access (NSA or otherwise) because we would have capabilities not +found in other archive front-ends or web-applications. |