Project

General

Profile

User manual » History » Version 9

koszko, 04/28/2022 05:29 PM
document running Hydrilla builder

1 1 koszko
# User manual
2
3 6 koszko
This page documents basic usage of Hydrilla. The instructions assume a POSIX shell and a UNIX-like system are being used.
4
5 1 koszko
{{toc}}
6
7
## Installation
8 4 koszko
9 1 koszko
### Using Python wheel
10
11 4 koszko
You can install Hydrilla server and(or) builder using .whl files from the [[Releases]] page. Please consider [[hachette:Verifying signatures|verifying the downloads]] using provided cryptographic signatures.
12
13
#### Installing dependencies
14
15
##### Python3
16
17
Hydrilla requires Python interpreter in at least version 3.7. You'd typically use Python3 as provided by you operating system distribution. For example, on Debian-based systems (including Trisquel) you can install it with:
18
``` shell
19
sudo apt install python3
20
```
21
22
##### pip
23
24
Pip is the package manager for Python. While not a direct dependency of Hydrilla, it is needed to utilize .whl files. You most likely also want to install pip as provided by your distro, e.g. for APT-based ones:
25
``` shell
26
sudo apt install python3-pip
27
```
28
29
##### Python libraries
30
31
Hydrilla relies on the following Python packages:
32
33
* `jsonschema`
34
* `click`
35
* `flask` (needed for Hydrilla server only)
36
* `reuse` (optional, only needed for Hydrilla builder to generate SPDX report)
37
38
If you don't have those dependencies installed, pip will automatically pull them from [PyPI](https://pypi.org) (except for reuse which would need to be installed separately with a command like `python3 -m pip install reuse`).
39
40
Nevertheless, you are encouraged to instead install the respective packages from your operating system's official repositories because those usually have stricter policies on stability, security and free licensing. In case of APT-based distributions the packages to install would be `python3-jsonschema`, `python3-click`, `python3-flask` and `reuse`[^reuse].
41
42
[^reuse]: Reuse tool was first packaged for Debian Bookworm and is not yet available in Debian Bullseye nor in Trisquel Nabia.
43
44
#### Installing Hydrilla
45
46
Let's assume you want to install version 1.0 of Hydrilla server. First, download and verify both[^server_depends_on_builder] `hydrilla.builder-1.0-py3-none-any.whl` and `hydrilla-1.0-py3-none-any.whl`. Then, run:
47
``` shell
48
python3 -m pip install path/to/downloaded/hydrilla.builder-1.0-py3-none-any.whl path/to/downloaded/hydrilla-1.0-py3-none-any.whl
49
```
50
51
[^server_depends_on_builder]: Hydrilla server also depends on Hydrilla builder.
52
53
This will install Hydrilla **for the current user**. The commands `hydrilla` and `hydrilla-builder` will be made available in `~/.local/bin/`.
54
55
#### Installing in virtualenv
56
57
If for example you don't want pip to install things under `~/.local/`, you might choose to create a virtual Python environment. First, make sure you have the `virtualenv` tool installed[^virtualenv_tool] (for example from APT package `python3-virtualenv`). Then, choose the folder in which you'd like to install the environment and run:
58
``` shell
59
virtualenv -p python3 --system-site-packages path/to/chosen/folder
60
```
61
62
[^virtualenv_tool]: Since Python 3.3 a virtual environment can also be created without this tool.
63
64
The `--system-site-packages` flag is not strictly necessary for it to work but is needed if you want packages inside the virtual environment to be able to see globally-installed dependencies.
65
66
Once the environment is created, you need to enter it by sourcing a script created by the virtualenv command, e.g.:
67
``` shell
68
source path/to/chosen/folder/bin/activate
69
```
70
71
Afterwards, the `python3 -m pip` commands you enter in this shell will install packages inside this virtual environment. You can learn more about Python virtual environments from online tutorials and the [virtualenv documentation](https://virtualenv.pypa.io).
72 1 koszko
73
### Using APT
74
75 3 koszko
Hydrilla APT repository is hosted at https://hydrillarepos.koszko.org/apt/ and is signed with Wojtek's PGP key (fingerprint **E9727060E3C5637C8A4F4B424BC5221C5A79FD1A**). It is expected to work with modern releases of most APT-based distributions (including Debian bullseye and Trisquel nabia).
76
77
This APT repository can be used to install Hydrilla server and builder system-wide and to later update the installation. It has to be said that this also requires you to trust Wojtek's repository with your system's safety (a malicious APT repository could easily take over a system that uses it).
78
79
If you've decided you want to install the APT repository on your system, the easiest way to do so is by copy-pasting the following script into your POSIX shell (and then confirming with your password). You can of course modify it according to your needs.
80
81
``` shell
82
__install_hydrilla_apt_repo() {
83
    local TMP="$1"
84
    local LISTS="$(cat <<EOF
85
deb     https://hydrillarepos.koszko.org/apt/ koszko main
86
deb-src https://hydrillarepos.koszko.org/apt/ koszko main
87
EOF
88
)"
89
90
    if ! wget -O "$TMP/koszko-keyring.gpg" https://hydrillarepos.koszko.org/apt/koszko-keyring.gpg; then
91
	echo "Error! Failed to download keyring file!" >&2
92
	return 1
93
    elif ! gpg --no-default-keyring --keyring "$TMP/koszko-keyring.gpg" --list-key E9727060E3C5637C8A4F4B424BC5221C5A79FD1A; then
94
	echo "Error! Invalid keyring file! Someone might be doing something nasty!" >&2
95
	return 1
96
    elif ! sudo cp "$TMP/koszko-keyring.gpg" /etc/apt/trusted.gpg.d/; then
97
	echo "Error!" >&2
98
	return 1
99
    elif ! printf %s "$LISTS" | sudo tee /etc/apt/sources.list.d/hydrillarepos.list > /dev/null; then
100
	echo "Error!" >&2
101
	return 1
102
    fi
103
104
    sudo apt-get update
105
}
106
107
install_hydrilla_apt_repo() {
108
    local TMP="$(mktemp -d)"
109
    __install_hydrilla_apt_repo "$TMP"
110
    local RESULT="$?"
111
112
    rm -r "$TMP"
113
114
    return "$RESULT"
115
}
116
117
install_hydrilla_apt_repo
118
```
119
120
This snippet is idempotent (i.e. it can be run multiple times and the effect will be as if it was run once). In addition, it executes `apt-get update` command at the end so that your APT is immediately aware of the new repository and its contents.
121
122
After installing the repository you can install Hydrilla builder and server using the following commands:
123
``` shell
124
sudo apt install python3-hydrilla.builder
125
```
126
``` shell
127
sudo apt install python3-hydrilla # this alone will also pull the builder as a dependency
128
```
129
130
The packages install their modules under `/usr/lib/python3/dist-packages/` which is seen by Python3 interpreters installed from APT. The `hydrilla` and `hydrilla-builder` commands get placed in `/usr/bin/`.
131
132
In addition, the `python3-hydrilla` package also includes sample WSGI script and Apache2 config files for Hydrilla under `/usr/share/doc/python3-hydrilla/examples/`.
133 1 koszko
134
## Understanding the concepts
135
136 7 koszko
Hydrilla serves Haketilo packages through an HTTP interface, as described in [[Repository API]]. It takes the package files to serve from a specific directory in the system as configured by the administrator. The package files stored in that directory conform to [[Hydrilla on-disk data format]]. Since that format is inconvenient for humans to operate on, there also exists another one - the [[Hydrilla source package format]]. One would typically prepare a Haketilo site resource as a source package and then use the `hydrilla-builder` command to convert it to Hydrilla on-disk format.
137
138
Hydrilla builder takes a directory with a source package, processes it and (if no errors are encountered) writes the "built" package files into the requested directory. Somewhat counterintuitively, the "build" does not involve actual compilation of sources nor any similar task (in future versions of Hydrilla all these will be delegated to other software packaging systems like Guix). Rather, the purpose of this step is to save files under the desired names (which involve files' hash sums) and to generate complete JSON definitions of packages being processed.
139
140
The serveable directory can be populated by invoking Hydrilla builder multiple times to put the files of different Haketilo packages in it. However, it is also possible to "build" multiple source packages into separate directories and then combine them. It is up to you, as the administrator, to choose how you are going to manage built packages, Hydrilla doesn't impose anything in this regard. Just keep in mind that there is no facility to remove a package from a serveable directory and that brutally deleting one package's files could break the other ones.  
141 1 koszko
142
## Running
143
144 9 koszko
### Hydrilla builder
145 1 koszko
146 9 koszko
Hydrilla software includes a `hydrilla-builder` command that can be used to convert Hydrilla [[Hydrilla source package format|source package]] into its [[Hydrilla on-disk data format|serveable form]]. The command has an [associated manpage](https://git.koszko.org/hydrilla-builder/tree/doc/man/man1/hydrilla-builder.1) (also included in the APT package) as well as a `--help` option.
147
148
Assuming you have both Hydrilla builder and git installed, you can clone and "build" a sample Haketilo package with the following:
149
``` shell
150
git clone https://git.koszko.org/hydrilla-source-package-example
151
mkdir /tmp/tmprepo
152
hydrilla-builder -s ./hydrilla-source-package-example -d /tmp/tmprepo
153
```
154
155
If you then run:
156
``` shell
157
find /tmp/tmprepo/
158
```
159
160
you should see the following list of files written by the builder:
161
```
162
/tmp/tmprepo/
163
/tmp/tmprepo/mapping
164
/tmp/tmprepo/mapping/helloapple
165
/tmp/tmprepo/mapping/helloapple/2021.11.10
166
/tmp/tmprepo/resource
167
/tmp/tmprepo/resource/hello-message
168
/tmp/tmprepo/resource/hello-message/2021.11.10
169
/tmp/tmprepo/resource/helloapple
170
/tmp/tmprepo/resource/helloapple/2021.11.10
171
/tmp/tmprepo/file
172
/tmp/tmprepo/file/sha256
173
/tmp/tmprepo/file/sha256/cee8d88cf5e5346522fd9ee5e1f994b335fb68d2f460b27b2a2a05f0bcd2b55b
174
/tmp/tmprepo/file/sha256/a6b9425a80963373d90d8fa22fca07e8626291a4f4bf5f17a4487c16ea1b8dac
175
/tmp/tmprepo/file/sha256/18dd7d7ac9f74a5ba871791ac6aa4d55530a336ae1b373538b6dd5320b330414
176
/tmp/tmprepo/file/sha256/a2010f343487d3f7618affe54f789f5487602331c0a8d03f49e9a7c547cf0499
177
/tmp/tmprepo/file/sha256/fd3d332844be0923b29206b32b9111767d73dd995e3ead7b7f06f72020bce650
178
/tmp/tmprepo/source
179
/tmp/tmprepo/source/hello.json
180
/tmp/tmprepo/source/hello.zip
181
```
182
183
In some cases `hydrilla-builder` command may fail with a message about the REUSE tool being unavailable. This will only happen when a source package actually requires REUSE (for generation of an SPDX report). The error means that either you don't have it installed or it's somewhere where Hydrilla software cannot see its Python modules.
184
185
### Hydrilla development server
186
187 8 koszko
Hydrilla repository software includes a `hydrilla` command that can be used to quickly spawn a local repository server. This is unsuitable for deployment of a publicly visible Hydrilla instance but very suitable for testing of both Hydrilla itself and Haketilo packages being developed. The command has an [associated manpage](https://git.koszko.org/pydrilla/tree/doc/man/man1/hydrilla.1) (also included in the APT package) as well as a `--help` option.
188
189
For a sample run, you're going to need a directory with some Haketilo packages. You can clone the [source package example repository](https://git.koszko.org/hydrilla-source-package-example/) and perform something along the lines of:
190
``` shell
191
mkdir /tmp/tmprepo/
192
hydrilla-builder -s path/to/cloned/hydrilla-source-package-example/ -d /tmp/tmprepo/
193
```
194
195
Then comes a typical invocation of `hydrilla` command:
196
``` shell
197
hydrilla -m /tmp/tmprepo/ -p 0
198
```
199
200
It causes Haketilo packages from `/tmp/tmprepo/` directory to be served on a random free port on `localhost`. Sample run generated this output:
201
```
202
 * Serving Flask app "hydrilla.server" (lazy loading)
203
 * Environment: production
204
   WARNING: This is a development server. Do not use it in a production deployment.
205
   Use a production WSGI server instead.
206
 * Debug mode: off
207
 * Running on http://127.0.0.1:46485/ (Press CTRL+C to quit)
208
```
209
210
One could then (in another shell) try running some commands like the ones below to confirm that the just-spawned local server is responding:
211
``` shell
212
# The following assume that Hydrilla is loaded with the sample Haketilo package
213
curl http://127.0.0.1:46485/mapping/helloapple.json
214
curl http://127.0.0.1:46485/resource/hello-message/2021.11.10
215
curl http://127.0.0.1:46485/query?url=https://hydrillabugs.koszko.org/a/b/c
216 1 koszko
```
217
218 9 koszko
### Hydrilla as WSGI application under Apache2
219 1 koszko
220 5 koszko
This section describes how to configure an Apache2 virtual host to serve a Hydrilla repository. This guide is mostly meant to be useful to people running their own web servers.
221
222
You're going to need:
223
224
* root access on the machine[^rootless_hosting] (for writing to `/etc/apache2/sites-available/` directory)
225
* Apache2 with `mod_wsgi` installed and enabled
226
* Hydrilla installed
227
228
[^rootless_hosting]: If you want to run a Hydrilla server on shared hosting without root access, this might be achievable using a .htaccess file but is not documented right now.
229
230
First, choose a directory where you want to store your serveable Haketilo packages. The default is `/var/lib/hydrilla/malcontent`. You can override this by saving the following file as `/etc/hydrilla/config.json`:
231
``` javascript
232
{
233
    // Path to directory from which Hydrilla will load packages metadata and serve files.
234
    "malcontent_dir": "/your/chosen/dir"
235
}
236
```
237
238
Fill the directory with some package files. You might for example clone the [source package example repository](https://git.koszko.org/hydrilla-source-package-example/) and build it with something along the lines of:
239
``` shell
240
sudo hydrilla-builder -s path/to/cloned/hydrilla-source-package-example/ -d /var/lib/hydrilla/malcontent/
241
```
242
243
Once done, grab Hydrilla's [sample WSGI script](https://git.koszko.org/pydrilla/tree/doc/examples/hydrilla.wsgi) and save it in your chosen location (the suggested one is `/var/lib/hydrilla/wsgi/hydrilla.wsgi`). Follow the comments in this script to modify it according to your needs.
244
245
Now, get the [sample Apache2 configuration](https://git.koszko.org/pydrilla/tree/doc/examples/hydrilla.example.com.tls.conf) (there is also [one for TLS-less deployment](https://git.koszko.org/pydrilla/tree/doc/examples/hydrilla.example.com.conf)), also modify it according to your needs (in particular, you'll likely want to change `hydrilla.example.com` to some real domain of yours) and save under `/etc/apache2/sites-available/your.chosen.config.name.conf`.
246
247
You can now enable the configuration with:
248
``` shell
249
sudo a2ensite your.chosen.config.name
250
```
251
252
You also need to reload or restart the Apache daemon for the configuration to be picked up (the command to do that varies between init systems). Once you do so, you can verify that the server is running properly. Consider running something like the following (replacing `hydrilla.example.com` with the domain name you used):
253
``` shell
254
# The following assume that Hydrilla is loaded with the sample Haketilo package
255
curl http://hydrilla.example.com/mapping/helloapple.json
256
# -v flag will let us verify that the "Content-Type: application/json" header is present
257
curl -v http://hydrilla.example.com/resource/hello-message/2021.11.10
258
curl -v http://hydrilla.example.com/query?url=https://hydrillabugs.koszko.org/a/b/c
259
```
260
261
If everything is working as expected (i.e. JSON documents are properly served by Hydrilla&Apache2), you can start populating the "malcontent directory" with built packages of your choice.