1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
|
[](https://github.com/spacetelescope/ohmycal/actions/workflows/cmake-multi-platform.yml) [](https://oh-my-cal.readthedocs.io/en/latest/?badge=latest)
OMC aims to consolidate the steps required to build, test, and deploy calibration pipelines and other software suites maintained by engineers at the Space Telescope Science Institute.
# Requirements
- Linux, or MacOS (Darwin)
- cmake
- libcurl
- libxml2 (optional)
- rsync
# Installation
Download the source code
```shell
git clone https://github.com/jhunkeler/ohmycal.git
cd ohmycal
```
Create and enter the build directory
```shell
mkdir build
cd build
```
Run cmake
```shell
cmake .. -DCMAKE_INSTALL_PREFIX="$HOME/programs/ohmycal"
```
Compile and install ohmycal
```shell
make
make install
```
# Configuration
## Environment variables
| Name | Purpose |
|------------------------------|-------------------------------------------------------|
| TMPDIR | Change default path to store temporary data |
| OMC_SYSCONFDIR | Change default path to search for configuration files |
| OMC_JF_ARTIFACTORY_URL | Artifactory service URL (ending in `/artifactory`) |
| OMC_JF_ACCESS_TOKEN | Artifactory Access Token |
| OMC_JF_USER | Artifactory username |
| OMC_JF_PASSWORD | Artifactory password |
| OMC_JF_SSH_KEY_PATH | Path to SSH public key file |
| OMC_JF_SSH_PASSPHRASE | Password associated with SSH public key file |
| OMC_JF_CLIENT_CERT_CERT_PATH | Path to OpenSSL cert files |
| OMC_JF_CLIENT_CERT_KEY_PATH | OpenSSL key file (in cert path) |
| OMC_JF_REPO | Artifactory "generic" repository to write to |
# Variable expansion
## Template strings
Template strings can be accessed using the `{{ subject.key }}` notation in any Ohmycal configuration file.
| Name | Purpose |
|----------------------------|------------------------------------------------------------------------------------------------------------------------|
| meta.name | Delivery name |
| meta.version | Delivery version |
| meta.codename | Delivery codename |
| meta.mission | Delivery mission |
| meta.python | Python version (e.g. 3.11) |
| meta.python_compact | Python (e.g. 311) |
| info.time_str_epoch | UNIX Epoch timestamp |
| info.release_name | Rendered delivery release name |
| info.build_name | Rendered delivery build name |
| info.build_number | Rendered delivery build number |
| storage.tmpdir | Ohymcal temp directory |
| storage.delivery_dir | Ohmycal delivery output directory |
| storage.results_dir | Ohmycal test results directory |
| storage.conda_artifact_dir | Ohmycal conda package directory |
| storage.wheel_artifact_dir | Ohmycal wheel package directory |
| storage.build_sources_dir | Ohmycal sources directory |
| storage.build_docker_dir | Ohmycal docker directory |
| conda.installer_name | Conda distribution name |
| conda.installer_version | Conda distribution version |
| conda.installer_platform | Conda target platform |
| conda.installer_arch | Conda target architecture |
| conda.installer_baseurl | Conda installer URL |
| system.arch | System CPU Architecture |
| system.platform | System Platform (OS) |
| deploy.docker.registry | Docker registry |
| deploy.jfrog.repo | Artifactory destination repository |
| workaround.tox_posargs | Return populated `-c` and `--root` tox arguments.<br/>Force-enables positional arguments in tox's command line parser. |
The template engine also provides an interface to environment variables using the `{{ env:VARIABLE_NAME }}` notation.
```ini
[meta]
name = {{ env:MY_DYNAMIC_DELIVERY_NAME }}
version = {{ env:MY_DYNAMIC_DELIVERY_VERSION }}
python = {{ env:MY_DYNAMIC_PYTHON_VERSION }}
```
## Runtime strings
Alternatively, one may expand environment variables using the `${VARIABLE_NAME}` notation.
Note: Environment variable expansion is not rendered in `[test:*].script` keys.
```ini
[meta]
name = ${MY_DYNAMIC_DELIVERY_NAME}
version = ${MY_DYNAMIC_DELIVERY_VERSION}
python = ${MY_DYNAMIC_PYTHON_VERSION}
```
# Delivery files
## Sections
All configuration section names and keys are _case-sensitive_.
### meta
| Key | Type | Purpose | Required |
|----------|---------|-----------------------------------------------------------------|----------|
| mission | String | Mission rule directory to use | Y |
| name | String | Name of the delivery | Y |
| version | String | Version of delivery | Y |
| codename | String | Release code name (HST-only) | N |
| rc | Integer | Release candidate level (i.e. delivery iteration) | Y |
| final | Boolean | Toggle final delivery logic (possibly deprecated in the future) | N |
| based_on | String | A conda environment YAML file/URL to use as a baseline | N |
| python | String | Version of Python to base delivery on | Y |
### conda
| Key | Type | Purpose | Required |
|--------------------|--------|----------------------------|----------|
| installer_name | String | Conda distribution name | Y |
| installer_version | String | Conda distribution version | Y |
| installer_platform | String | Conda target platform | Y |
| installer_arch | String | Conda target architecture | Y |
| installer_baseurl | String | Conda installer URL | Y |
| conda_packages | List | Conda packages to install | N |
| pip_packages | List | Pypi packages to install | N |
### runtime
Environment variables exported are _global_ to all programs executed by ohmycal. There is no limit to the number of environment variables that can be set.
| Key | Type | Purpose | Required |
|------------------------|--------|------------------------------------------------------------|----------|
| _arbitrary name_ | String | Export environment variables to build and test environment | N |
| _artitrary_name ... N_ | - | - | - |
### test:_name_
Sections starting with `test:` will be used during the testing phase of the ohmycal pipeline. Where the value of `name` following the colon is an arbitrary value, and only used for reporting which test-run is executing. Section names must be unique.
| Key | Type | Purpose | Required |
|--------------|--------|-------------------------------------------------------|----------|
| build_recipe | String | Git repository path to package's conda recipe | N |
| repository | String | Git repository path or URL to clone | Y |
| version | String | Git commit or tag to check out | Y |
| runtime | List | Export environment variables specific to test context | Y |
| script | List | Body of a shell script that will execute the tests | Y |
### deploy:artifactory:_name_
Sections starting with `deploy:artifactory:` will define the upload behavior of build and test artifacts to Artifactory. Where the value of `name` is an arbitrary value, and only used for reporting. Section names must be unique.
| Key | Type | Purpose | Required |
|-------|--------|----------------------------------------|----------|
| files | List | `jf`-compatible wildcard path | Y |
| dest | String | Remote artifactory path to store files | Y |
### deploy:docker
The `deploy:docker` section controls how Docker images are created, when a `Dockerfile` is present in the `build_docker_dir`.
| Key | Type | Purpose | Required |
|-------------------|--------|----------------------------------------------|----------|
| registry | String | Docker registry to use | Y |
| image_compression | String | Compression program (with arguments) | N |
| build_args | List | Values passed to `docker build --build-args` | N |
| tags | List | Docker image tag(s) | Y |
# Mission files
Mission rules are defined in the `OMC_SYCONFDIR/mission` directory. Each mission configuration file shares the same name as the directory. To create a new mission, `example`, the directory structure will be as follows:
```text
OMC_SYSCONFDIR/
mission/
example/
example.ini
supporting_text.md.in
```
## Sections
### meta
| Key | Type | Purpose | Required |
|------------------|--------|-----------------------------------------------------------|----------|
| release_fmt | String | A string containing one or more release format specifiers | Y |
| build_name_fmt | String | - | Y |
| build_number_fmt | String | - | Y |
#### Release format specifiers
| Formatter | Purpose |
|-----------|----------------------------------|
| %n | Delivery name |
| %c | Delivery codename |
| %r | Delivery RC level |
| %R | Delivery RC level (final-aware) |
| %v | Delivery version |
| %m | Mission name |
| %P | Python version (i.e. 3.11) |
| %p | Compact Python version (i.e. 311 |
| %a | System CPU Architecture |
| %o | System Platform (OS) |
| %t | UNIX Epoch timestamp |
### template:filename
Sections starting with `template:` will make use of the built-in template engine, replacing any strings in `filename`, relative to the mission directory, with expanded data values. The output will be written to `destination`.
```ini
# example.ini
[template:supporting_text.md.in]
destination = {{ storage.delivery_dir }}/output_filename.md
```
|
