From c8af9db18f92e7cb78f82e181dd75ddcf0747939 Mon Sep 17 00:00:00 2001 From: Joseph Hunkeler Date: Wed, 7 Oct 2015 12:35:10 -0400 Subject: Add more information. --- README.md | 322 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 317 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 57fea09..8c7268c 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ At this stage in the game, "Conda Build Control" is a misnomer of sorts. Having CBC has organically morphed into a translator capable of generating Conda build recipes *on the fly* rather than becoming a replacement build system altogether. -What does it translate? Well, Conda's recipe format consists of three or more files, each with their own purpose, and must be written by hand for each package. I felt like this was a major caveat, because each stage in the build was controlled by completely separate files. Files that need to be opened and closed repeatedly (more like constantly) during the creation process. In theory this is cool... you only need to change the bit that matters to the task at hand, but in practice, when working with a large convoluted recipes its extremely easy to get lost and/or frustrated. +What does it translate? The Conda recipe format consists of three or more files, each with their own purpose, and must be written by hand for each package. I felt like this was a major caveat, because each stage in the build was controlled by completely separate files. Files that need to be opened and closed repeatedly (more like constantly) during the creation process. In theory this is cool... you only need to change the bit that matters to the task at hand, but in practice, when working with a large convoluted recipes its extremely easy to get lost and/or frustrated. The layout of a Conda recipe is as follows: @@ -38,7 +38,7 @@ Instead, cbc consolidates the layout and allows you to focus all recipe-related CBC requires Python 3, and will not work under Python 2. ```bash -git clone https://bitbucket.org/jhunkeler/cbc +git clone https://bitbucket.org/exampleeler/cbc ``` ```bash @@ -49,12 +49,25 @@ cd cbc python setup.py install ``` +# Obtaining STScI Recipes + +The following repository provides the recipes maintained by STScI. Keep in mind, however, many of the recipes reference non-public git repositories. Depending on where you are and what you have access to, **these recipes may not work at all.** + +(See also: `git_url` and/or `url` directives in each .ini file). + +```bash +cd ~ +git clone https://bitbucket.org/exampleeler/cbc-recipes +``` + # Configuration **This will change to something more appropriate: stay tuned...** CBC writes translated Conda recipes to a static directory, referenced by the `CBC_HOME` environment variable (primarily for house-keeping purposes). There is no default value, and an exception will be raised if it is undefined at run-time. +The companion environment variable, `CBC_RECIPES`, should be defined if you want to use the build wrapper, `cbc_monolith`. + **Create a directory:** The location is not important, so long as your account has read-write access. Your home directory is good place to start with. @@ -68,12 +81,14 @@ mkdir -p ~/cbc-output ```bash export CBC_HOME=~/cbc-output +export CBC_RECIPES=~/cbc-recipes ``` **CSH users will want to do this instead:** ```bash setenv CBC_HOME ~/cbc-output +setenv CBC_RECIPES ~/cbc-recipes ``` # Example Package @@ -106,7 +121,7 @@ summary: ${package:name} is a test package [source] fn: ${package:name}-${package:version}.tar.gz -url: https://bitbucket.org/jhunkeler/cbc-recipes/downloads/${fn} +url: https://bitbucket.org/exampleeler/cbc-recipes/downloads/${fn} [build] number: 1 @@ -199,7 +214,7 @@ requirements: - python source: fn: cbc_test_package-1.0.0.tar.gz - url: https://bitbucket.org/jhunkeler/cbc-recipes/downloads/cbc_test_package-1.0.0.tar.gz + url: https://bitbucket.org/exampleeler/cbc-recipes/downloads/cbc_test_package-1.0.0.tar.gz test: imports: - cbc_test_package @@ -320,7 +335,7 @@ import: 'cbc_test_package' TEST END: cbc_test_package-1.0.0-py34_1 ``` -`cbc_build` can, as the name suggests, build packages directly, but I don't recommend doing it. Why? Because its good for testing a quick build to see if it works; just not for the real deal. Conda has built-in mechanisms CBC does not provide (the list is many). Yes, this goes back to what I mentioned regarding not reinventing the wheel. CBC was *supposed* to be more than *it is* because at the time I didn't know Conda could *do more* than it *does*. I blame frantic poor planning. +`cbc_build` can, as the name suggests, build packages directly, but I don not recommend doing it. Why? Because its good for testing a quick build to see if it works; just not for the real deal. Conda has built-in mechanisms CBC does not provide (the list is many). Yes, this goes back to what I mentioned regarding not reinventing the wheel. CBC was *supposed* to be more than *it is* because at the time I didn't know Conda could *do more* than it *does*. I blame frantic poor planning. How would you go about doing that? Simple enough: @@ -331,3 +346,300 @@ cbc_build --no-upload ~/cbc-recipes/cbc_test_package.ini The output will look roughly the same as before just massively out of order. Python's `subprocess` module seems to print output asynchronously for no apparent reason. By the time I realized this was happening I had already decided `cbc_build` would not be used as a true build tool. No fix is likely. + + +# Working with cbc_monolith + +> Monolith +> > noun: monolith; plural noun: monoliths +> > +> > 1. A large single upright block of stone, especially one shaped into or serving as a pillar or monument. + +`cbc_monolith` provides a minimalistic approach to building several recipes in sequence, resulting in a *frozen build*. This is especially useful for hosted repositories that you do *not* want to change over the course of time, or perhaps you wish to control the rate at which they receive new packages. Either way, it was not designed to upload packages to anaconda.org. That functionality could be added in the future, but is not considered a high priority at the moment. + +Below we will discuss two ways to use `cbc_monolith`, depending on your needs... but first... + + +## Configuration + +As mentioned earlier this script requires both `CBC_HOME` and `CBC_RECIPES` be defined in your environment. It is also possible to pass these paths as arguments at run-time by issuing `--cbc-output-dir` and `--cbc-recipe-dir` respectively. + +It is recommended `cbc_monolith` be executed from its own directory, because the log file(s) as well as input files generally create a lot of clutter. + +```bash +mkdir -p ~/monolith +cd ~/monolith +``` + +## Usage Statement + +``` +usage: cbc_monolith {manifest} [-pnco] + manifest List of recipes to build (in order) + --python -p Version to pass to conda-build + --numpy -n Version to pass to conda-build + --cbc-recipes -c Path to CBC recipes directory + --cbc-output-dir -o Path to CONDA recipes +``` + +## Method 1 - Outer Scope + +Assume the manifest, `outer.lst`, contains the following two packages: + +``` +stsci.tools +stsci.image + +``` + +What happens when we run `cbc_monolith outer.lst`? + +``` +cbc_monolith outer.lst +[output truncated to show important details] + +BUILD START: stsci.tools-0.0.0.git-py34_768 +BUILD START: d2to1-0.0.0.git-py34_0 +BUILD END: d2to1-0.2.12.git-py34_0 +BUILD START: stsci.tools-0.2.12.git-py34_0 +BUILD START: stsci.distutils-0.0.0.git-py34_1 +BUILD END: stsci.distutils-0.0.0.git-py34_1 +BUILD START: stsci.tools-0.0.0.git-py34_157 +BUILD START: pyfits-0.0.0.git-py34_0 +BUILD START: cfitsio-3.370-1 +BUILD END: cfitsio-3.370-1 +BUILD START: pyfits-0.0.0.git-py34_0 +BUILD END: pyfits-v3.3.0.git-py34_8 +BUILD START: stsci.tools-v3.3.0.git-py34_8 +BUILD END: stsci.tools-0.0.0.git-py34_785 +BUILD START: stsci.image-0.0.0.git-py34_785 +BUILD START: stsci.convolve-0.0.0.git-py34_0 +BUILD END: stsci.convolve-0.0.0.git-py34_37 +BUILD START: stsci.image-0.0.0.git-py34_37 +BUILD END: stsci.image-0.0.0.git-py34_45 +``` + +You were only expecting to see two packages build, correct? Wrong. What's happening here is Conda's dependency resolution taking over. Notice `stsci.tools` shows up many times throughout the build, because as each dependency completes it jumps back to `stsci.tools` and continues scanning for, and building, additional dependencies. The same applies to `stsci.image`, albeit, it has far fewer requirements. + +Indenting the output into a tree might help drive the concept home: + +``` +BUILD START: stsci.tools-0.0.0.git-py34_768 + BUILD START: d2to1-0.0.0.git-py34_0 + BUILD END: d2to1-0.2.12.git-py34_0 +BUILD START: stsci.tools-0.2.12.git-py34_0 + BUILD START: stsci.distutils-0.0.0.git-py34_1 + BUILD END: stsci.distutils-0.0.0.git-py34_1 +BUILD START: stsci.tools-0.0.0.git-py34_157 + BUILD START: pyfits-0.0.0.git-py34_0 + BUILD START: cfitsio-3.370-1 + BUILD END: cfitsio-3.370-1 + BUILD START: pyfits-0.0.0.git-py34_0 + BUILD END: pyfits-v3.3.0.git-py34_8 +BUILD START: stsci.tools-v3.3.0.git-py34_8 +BUILD END: stsci.tools-0.0.0.git-py34_785 +BUILD START: stsci.image-0.0.0.git-py34_785 + BUILD START: stsci.convolve-0.0.0.git-py34_0 + BUILD END: stsci.convolve-0.0.0.git-py34_37 +BUILD START: stsci.image-0.0.0.git-py34_37 +BUILD END: stsci.image-0.0.0.git-py34_45 +``` + +Let's look at the resultant repository structure. As you can see, the timestamps correlate with the build order: + +``` +$ ls -ltr ~/anaconda3/conda-bld/linux-64/ +total 6260 +-rw-rw-r-- 1 example example 38657 Oct 6 17:23 d2to1-0.2.12.git-py34_0.tar.bz2 +-rw-rw-r-- 1 example example 42052 Oct 6 17:23 stsci.distutils-0.0.0.git-py34_1.tar.bz2 +-rw-rw-r-- 1 example example 3630988 Oct 6 17:24 cfitsio-3.370-1.tar.bz2 +-rw-rw-r-- 1 example example 2126961 Oct 6 17:24 pyfits-v3.3.0.git-py34_8.tar.bz2 +-rw-rw-r-- 1 example example 419752 Oct 6 17:25 stsci.tools-0.0.0.git-py34_785.tar.bz2 +-rw-rw-r-- 1 example example 75614 Oct 6 17:25 stsci.convolve-0.0.0.git-py34_37.tar.bz2 +-rw-rw-r-- 1 example example 35301 Oct 6 17:25 stsci.image-0.0.0.git-py34_45.tar.bz2 +-rw-rw-r-- 1 example example 719 Oct 6 17:25 repodata.json.bz2 +-rw-rw-r-- 1 example example 2908 Oct 6 17:25 repodata.json +``` + + +## Method 2 - Inner Scope + +An inner scope build relies on the contents of Conda meta-packages, rather than individual recipes (per se). + +Assume the manifest, `inner.lst`, contains the following entry: + +``` +stsci +``` + +So where do the packages come from? What's with the magic? Well it's easier to show you than it is to explain. Below is a dump of the `requirements` section of the `stsci` meta-package: + +[Keep in mind -- This was lifted from a *test* meta-package, and does not likely reflect the current state of affairs.] + +``` +[requirements] +build : + ${requirements:run} +run : + #STScI + acstools [py34] + acstools [py27] + asdf-standard [py34] + asdf-standard [py27] + astrolib.coords [py34] + astrolib.coords [py27] + astropy [py34] + astropy [py27] + astropy-helpers [py34] + astropy-helpers [py27] + calcos [py34] + calcos [py27] + cfitsio + d2to1 [py34] + d2to1 [py27] + #drizzle [py34] + drizzle [py27] + fftw + fitsblender [py34] + fitsblender [py27] + hstcal [py34] + hstcal [py27] + htc_utils [py34] + htc_utils [py27] + imexam [py34] + imexam [py27] + nictools [py34] + nictools [py27] + #photutils [py34] + photutils [py27] + poppy [py34] + poppy [py27] + purge_path [py34] + purge_path [py27] + pyasdf [py34] + pyasdf [py27] + pydrizzle [py34] + pydrizzle [py27] + pyfftw [py34] + pyfftw [py27] + pyfits [py34] + pyfits [py27] + #pyqtgraph #issues with building + #pysynphot [py34] + pysynphot [py27] + pywcs [py34] + pywcs [py27] + #reftools [py34] + reftools [py27] + stistools [py34] + stistools [py27] + stsci.convolve [py34] + stsci.convolve [py27] + stsci.distutils [py34] + stsci.distutils [py27] + stsci.image [py34] + stsci.image [py27] + stsci.imagemanip [py34] + stsci.imagemanip [py27] + stsci.imagestats [py34] + stsci.imagestats [py27] + stsci.ndimage [py34] + stsci.ndimage [py27] + stsci.sphinxext [py34] + stsci.sphinxext [py27] + stsci.stimage [py34] + stsci.stimage [py27] + stsci.tools [py34] + stsci.tools [py27] + stwcs [py34] + stwcs [py27] + + webbpsf [py34] + webbpsf [py27] + wfpc2tools [py34] + wfpc2tools [py27] + wfc3tools [py34] + wfc3tools [py27] + + + #3rd-party + atlas-generic [osx] + sextractor-generic [osx] + sextractor [linux] + ds9 + + #Standard + anaconda [py34] + anaconda [py27] + numpy [py34] + numpy [py27] + python [py34] + python [py27] +``` + +Aside from a giant wall of text, you may notice a few key things here. Each package is duplicated, one per line, with its own Python version requirements. In addition to this, some packages are marked `[osx]` while others are marked `[linux]`. Why? Because some packages require a little tender loving care make them work. For example, the `atlas` package is not provided by Anaconda under OSX, but it is available under Linux. That's... unfortunate. + +OSX is a special case, as usual, and so is Windows for that matter. Many recipes rely on `openblas` or Apple's built-in `Accelerate` framework to take advantage of "blazing fast linear algebra" operations, so in general, ATLAS is not required. However, some packages absolutely require ATLAS to be installed regardless which operating system you are running. `sextractor` is one such package. + +By suffixing the `atlas-generic` recipe with `[osx]`, we effectively tell Conda: *"Only build this package under OSX."* + +By suffixing `sextractor-generic` with `[osx]`, we effectively tell Conda: *"Only build this package under OSX"* + +See a trend here? Good. Also note that for obvious reasons, `sextractor-generic` depends on `atlas-generic` in the `requirements` section of its own recipe. + +The same is true for the `[py27]` and `[py34]` markers. If a recipe is not compatible with Python 3 this convention makes it easy to tell Conda: *"We don't support Python 3, sorry. Only build this package for Python 2.7."* + +``` +my_cool_package [py27] +another_cool_package [py34] +``` + +This does exactly what it says it will. `my_cool_package` will only be built when Python 2.7 is the controlling interpreter during compilation (i.e. `--python [PYTHON VERSION]`), while `another_cool_package` is built if we're using Python 3.4. Confusing, yes, but its still a neat and very useful all the same. + + +#Miscellaneous Utilities + +## cbc_remote_purge + +The most destructive command in the CBC library. **Use with extreme caution.** `cbc_remote_purge` recursively removes all packages, including previous package revisions, in the currently-loaded anaconda.org remote repository. That means you will absolutely destroy the contents of your anaconda.org account, so I cannot stress this enough: + +**DO NOT RUN THIS COMMAND UNLESS YOU ARE ABSOLUTELY CONVINCED YOU NEED TO!** + +This script probably has no effect if you are not already logged into anaconda.org at the time, but don't take chances. + +## cbc_repo_copy + +A simple `rsync` wrapper to copy a local Conda repository elsewhere (e.g. a remote server). To prevent accidental destruction (or directory cluttering) you will need to change the current working directory to the remote filesystem first, then execute `cbc_repo_copy` + +```bash +cd /remote/webserver/docroot/some/place +cbc_repo_copy +#[rsync operation happens here] +``` + +If you do not have this capability, there's still hope, just search the internet for `sshfs` with your favorite search engine. + +## cbc_repo_clean + +Removes the contents of the **local** Conda package repository (i.e. `[...]/conda-bld/{os}-{arch}/`). It is recommended to perform this as a housekeeping task, especially before creating a "frozen" (or static) monolithic build. + + +## cbc_recipe + +Generates a cbc recipe file based on user-input. This is helpful for when generating trees of packages on the fly from an external script. This script may need to become more feature-rich before that's reality, however. + +``` +usage: cbc_recipe [-h] [--style STYLE] [--name NAME] [--version VERSION] + [--build-number BUILD_NUMBER] [--license LICENSE] + [--homepage HOMEPAGE] [--d2to1-hack] [--use-git] + [--meta-package] + recipe +``` + +## cbc_server + +**DEPRECATED** + +This is a simple http file server. It was originally designed for use with cbc recipes for performing "off the grid" builds. You can still use it as such, because it still works, however, I'm not going to document it. I don't recommend using it anymore in lieu of `git_url` or remote `url` calls. + +Take a gander in the source code if you are feeling especially adventurous. -- cgit