****************** Contributing Guide ****************** .. attention:: A `GitHub `_ account is required to begin contributing to AstroConda Guidelines ========== .. attention:: The following packaging guidelines are subject to change at any time. - Please be respectful when commenting on pull-requests or issues. - If your contribution is not accepted into AstroConda, as a general courtesy, you will be given a clear and concise reason. - As a contributor you may not claim exclusive rights to a particular recipe. - You are free to maintain a recipe in AstroConda by issuing regular pull requests. - Everyone is welcome to improve upon recipes so long as the changes do not introduce packaging conflicts. - Abandoned recipes may be moved into the ``deprecated`` directory at any time without warning. (i.e. The package no longer compiles, has been obsoleted, or presents a conflict that cannot be resolved, etc). - Packages derived from ``deprecated`` recipes will remain available in AstroConda for historical purposes (i.e. to preserve backwards compatibility). Bugs, questions, and requests ============================= Please open a new issue or send us a pull request for bugs, feedback, questions, or enhancements. * For documentation issues use the `astroconda issue tracker `_ * For recipe issues, use the `astroconda-contrib issue tracker `_ Adding a recipe to astroconda-contrib ===================================== In this example we will be adding a new recipe to the AstroConda repository for `sympy `_, the symbolic mathematics library. Navigate to the `astroconda-contrib `_ repository on GitHub, login, and create a fork (or click `here `_ to have your fork created automatically). Now that you have a fork of astroconda-contrib, go ahead and clone it to your system: .. code-block:: sh git clone https://github.com/[Your_Account]/astroconda-contrib cd astroconda-contrib To get started adding our recipe, create a new branch and name it ``sympy-contrib``: .. code-block:: sh git checkout -t -b sympy-contrib Git will automatically switch your branch from ``master`` to ``sympy-contrib`` as denoted by the following output: .. code-block:: sh Branch sympy-contrib set up to track local branch master. Switched to a new branch 'sympy-contrib' If you have taken the liberty of looking around the astroconda-contrib directory, you will have noticed a bunch of directories are sitting in there all named by-package. So let's keep things simple and straight forward. Go ahead and create a directory and name it ``sympy``, and proceed inside: .. code-block:: sh mkdir sympy cd sympy .. note:: This is not an full Conda packaging tutorial. For more information about creating recipes from scratch, please refer to the `conda-build documentation `_. **Hint:** Investigate the contents of the recipes in astroconda-contrib. For most cases, copying an existing recipe and changing its values will suffice. Copy the contents of the ``astroconda-contrib/template`` recipe. Three files ``bld.bat``, ``build.sh``, and ``meta.yaml`` will be copied to your working directory: .. code-block:: sh cp ../template/* . Go ahead and open ``meta.yaml`` with your favorite plain-text editor: .. caution:: It is *highly recommended* that you enable "tabs to spaces" for your editor. Tab widths are unpredictable and may cause Conda's YAML parser to fail. .. code-block:: sh vim meta.yaml The general structure of the file is as follows: .. code-block:: yaml # These directives are commented here due to Pygments # failing to parse this section of code. # ... They are not commented in the real file. #{% set name = '' %} #{% set version = '' %} #{% set number = '0' %} about: # Package homepage home: # Package license license: # A brief description summary: package: # Define these values above name: {{ name }} version: {{ version }} build: # Define this value above number: {{ number }} source: fn: {{ name }}-{{ version }}.tar.gz url: http://example.com/example/{{ name }}-{{ version }}.tar.gz requirements: build: # Dependencies required at BUILD-TIME go here - setuptools - python x.x run: # Dependencies required at RUN-TIME go here # - ... #test: # imports: # # - (e.g. a_python_module) # # commands: # # - (e.g. program --help) First, let's fill in the blanks. Modify the JINJA2 template markers for ``name``, ``version``: .. code-block:: none {% set name = 'sympy' %} {% set version = '1.0' %} Fill in the ``about`` section with relevant information regarding the package: .. code-block:: yaml about: home: http://sympy.org license: GPL summary: Python library for symbolic mathematics Next, modify the ``source`` section's ``url`` so that it points to ``sympy``'s source archive (on the internet): .. code-block:: sh source: fn: {{ name }}-{{ version }}.tar.gz url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz What's with the never-ending stream of bracket encapsulated keywords, you ask? Conda uses JINJA2, a basic template system that provides basic string interpolation within recipes. This comes in handy if, let's say, you decide to build a more recent version of ``sympy``, you need only modify the first two variable definitions in this file (assuming the URL structure has not changed). The ``requirements`` section may be confusing to some, so let's clarify the distinction between ``build`` and ``run`` before diving in. The ``build`` section defines Conda packages required at compile-time (i.e. ``python setup.py install``), whereas the ``run`` section lists Conda packages required at install-time (i.e. ``conda install [package]``). As recipe maintainer the method used to dependency discover is that of trial and error. For many Python packages obtained via PyPi, it is easy enough to visually examine ``setup.py`` or ``requirements.txt`` to get a good idea of the recipes you need to depend on. Some package may require several iterations of executing ``conda build`` and returning to your recipe in the editor to append more packages. As we can see below, ``sympy`` requires ``mpmath``, ``setuptools`` and ``python`` to *build* the recipe, but only ``mpmath`` and ``python`` to *run it* successfully after installation: .. code-block:: yaml requirements: build: - mpmath - setuptools - python x.x run: - mpmath - python x.x What does the ``x.x`` imply exactly? This instructs ``conda build`` *not* to proceed unless ``python=[version]`` has been issued as an argument on the command-line. If ``x.x`` is omitted here, the recipe will choose the version of Python currently active in your environment. In most cases it is best to be explicit rather than implicit when it comes to defining version requirements in Conda. The ``test`` section defines few useful lists, ``imports``, ``commands``, and ``requires``. While these are not *required* to be used in any given recipe, we do use them in AstroConda. The ``imports`` section is a list of Python module imports, the ``commands`` are executed in a basic shell environment, and finally ``requires`` defines any extraneous packages to be installed into the environment before running the tests. .. code-block:: yaml test: imports: - sympy #commands: # - no shell commands to execute #requires: # - does not require any extra testing-related packages If ``sympy`` provided a command-line utility named ``sympy-show``, you would use the ``commands`` section to verify the utility's functionality. A simple example of this would be to output a usage statement. .. code-block:: sh test: # ... commands: - sympy-show --help If a program returns greater than zero ``conda build`` will fail as if it observed an error. Not all programs return zero after issuing ``--help`` (or an equivalent argument). Due to this, you may need to omit this style of test. It will not hurt to keep the ``commands`` section populated but disabled with a note describing why it doesn't work. Others will find this information useful. Given this scenario, the optimal approach would be to contact the developers and plead with them to normalize the exit value. Below is our ``sympy`` final recipe. Despite the overwhelming use of JINGA2 in our example, things are looking pretty streamlined. .. code-block:: none {% set name = 'sympy' %} {% set version = '1.0' %} {% set number = '0' %} about: home: http://sympy.org license: GPL summary: Python library for symbolic mathematics source: fn: {{ name }}-{{ version }}.tar.gz url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz requirements: build: - mpmath - setuptools - python x.x run: - mpmath - python x.x test: imports: - sympy The ``template`` directory copied earlier in this tutorial contains two basic python build scripts for both \*Nix (``build.sh``) and Windows (``bld.bat``), and is coincidentally compatible with the example we're using here. Not all Python packages (especially Makefile-based packages) will compile successfully using this "one-liner" template. Always refer to the ``INSTALL`` file or equivalent documentation for the program you are attempting to compile to learn more about what the package expects from the end-user at compile-time. Before we can issue a pull request on GitHub, we first ensure it builds, tests, and installs properly on our local system. To do this we need to change our directory to one level above the recipe. .. code-block:: sh cd .. # i.e. /path/to/astroconda-contrib Now run ``conda build`` to compile our ``sympy`` recipe into a Conda package. In the example below we are building against Python 3.5: .. code-block:: sh conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=3.5 sympy That's probably a bit more involved than you thought. Let's break it down. We issue ``-c [URL]`` which instructs the build to utilize the AstroConda channel while checking for package dependencies (i.e. the recipe's ``requirements`` section). Secondly, we issue ``--skip-existing`` to prevent ``conda build`` from rebuilding dependencies discovered in the local astroconda-contrib directory. That is to say, if a package defined as a requirement exists remotely, it will then download and install it, rather than rebuild it from scratch. ``--python=`` is self-explanatory, and the final argument is the name of the recipe(s) we intend to build. At this point, if the build was successful, our Conda package (a bzipped tarball) called ``sympy-1.0-py35_0.tar.bz2`` is emitted to ``/path/to/miniconda3/conda-bld/[os-arch]/``. This directory is a local Conda package repository. To install this new ``sympy`` package and interact with it ourselves you could run the following: .. code-block:: sh conda create -n sympy_test --use-local sympy source activate sympy_test Then manually verify the package is working: .. code-block:: sh python And checking it out for yourself: .. code-block:: python >>> import sympy >>> sympy.__file__ '/path/to/miniconda3/envs/sympy_test/lib/python3.5/site-packages/sympy/__init__.py' Now that you have verified the recipe is fully functional and are happy with the outcome, it's time to create a pull request against astroconda-contrib main repository. Push your ``sympy-contrib`` branch up to your fork on GitHub: .. code-block:: sh git push origin sympy-contrib It is recommended that you familiarize yourself with GitHub pull requests before proceeding (see: `tutorial `_). Using GitHub, you will want to browse to your ``astroconda-contrib`` fork, select the ``sympy-contrib`` branch from the drop-down menu (the default will read: "Branch: master", next to a black downward-pointing caret). Once selected, click the large green button labeled: "New pull request". From here, you may wish to edit the title of your pull request and add initial comments or notes regarding what you have done, or list any work that may still need to be done. After submitting your pull request, a member of the Science Software Branch at STScI, or fellow contributors will review the requested changes, ask questions, offer feedback or advice. At this point if everything appears to be in order your recipe will likely be merged, built, and incorporated into AstroConda! Updating a recipe in astroconda-contrib ======================================= Let's assume time has passed and our ``sympy`` package from the previous example is no longer up to date with what's generally available on GitHub. Updating recipes is a fairly straight forward process. At the top of the file you will remember we have a few variables defined encapsulated by ``{% %}``. These ``jinja2`` variables control the name, version, and build revision of the recipe. Using variable interpolation saves time, because it's much easier to edit a single variable that effects an entire recipe, than it is to search/replace specific fields. Typos are also much easier to spot. ``{{ name }}``, ``{{ version }}`` and ``{{ number }}`` expand to ``sympy``, ``1.0`` and ``0`` respectively: .. code-block:: none {% set name = 'sympy' %} {% set version = '1.0' %} {% set number = '0' %} [...] build: number: {{ number }} [...] source: fn: {{ name }}-{{ version }}.tar.gz url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz So to update ``sympy`` to version ``1.2``, for example, you would perform the following steps in your forked ``astroconda-contrib`` repository. Checkout a new branch --------------------- .. code-block:: sh git checkout -tb update-sympy master Doing this ensures your new branch is based on ``master`` rather than your current branch, if any. It also keeps your ``master`` branch pristine, avoiding merge conflicts in the future. Make your modifications ----------------------- .. code-block:: sh $EDITOR sympy/meta.yaml [...] {% set version = '1.2' %} # ^ Was '1.0', but not anymore. Now save and exit your editor. Review your modifications ------------------------- As stated earlier, the fastest way to find out whether your recipe works correctly is to try building it for yourself. .. code-block:: sh conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=2.7 sympy conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=3.5 sympy Did it work? If not, review the error message and make changes accordingly. Commit your modifications ------------------------- Assuming you are able to build the package locally, then you're ready to push your changes up to your fork. .. code-block:: sh git add sympy/meta.yaml git commit -m 'Update sympy 1.0 -> 1.2' git push origin update-sympy Open a pull request ------------------- See: `Using Pull Requests `_ 1. Using your browser, visit the ``update-sympy`` branch of your fork: https://github.com/YOUR_ACCOUNT/astroconda-contrib/tree/update-sympy 2. Click the gray "New pull request" button 3. Fill out the pull request form 4. Click the green "Create pull request" button That's all there is to it. One of our maintainers will review the pull request and get back to you.