Haketilo

The tool used to make Haketilo packages is Hydrilla builder. Right now Hydrilla builder does not handle program builds in the conventional sense. There's no compilation nor minification, only computation of files' SHA256 sums and (optionally) generation of an SPDX report.

We want to give Hydrilla builder the ability to install specified dependencies and perform necessary software builds. This will be done as Roadmap task 16 (which, btw, is important enough that we should prioritize it over some of the lower-numbered tasks).

There are many possible ways to solve this. If I just pick one and jump straight to implementing it, I will most likely miss something and will have to write it from scratch later as is usually the case. Hence I want to discuss the feature before implementing anything. I hope to see some comments and suggestions.

Requirements¶

1. We should be able to specify the build procedure for a package.
   • Hydrilla builder should be able to automatically perform package build according to that procedure.
2. We should be able to specify the tools needed to build a package.
   • Hydrilla builder should be able to automatically install those tools when performing a build (by default preferably to an isolated environment).
3. It should be possible to build all Haketilo packages from source.
4. We should leverage existing software repositories where possible without violating the requirements above (this applies to both the build tools and software to be packaged as Haketilo resources).

Considerations¶

Relying on a GNU/Linux distribution's repository for build tools¶

We could allow Haketilo packages to name their build dependencies from repositories of a chosen distro. We'd then implement an ability for Hydrilla builder to install those for the duration of a build. We'd preferably perform builds inside a chroot, PRoot or a Linux namespace-based container, although UML is also an option here¹.

Choosing a distro¶

At the beginning it's probably best to just tie Hydrilla builder to one chosen distro. With time we can add support for using build dependencies from other ones. This means right now we just need to choose which distro to start with. I'm considering the 3 described below and leaning towards Guix. I excluded Hyperbola due to it lacking some important packages (NodeJS) and Parabola because it's too unstable and in my view has suboptimal packaging hygiene.

Trisquel¶

Drawbacks:

• It doesn't emphasize reproducibility.
• It's an LTS distro. Move to a new stable release could cause problems. Pros:
• FSDG-compliance included.
• Affiliated with GNU which we also affiliate with.

Guix¶

Drawbacks:

• Doesn't have as many packages as Trisquel or NixOS. Pros:
• Highly reproducible.
• FSDG-compliance included.
• Affiliated with GNU which we also affiliate with.
• Easy installation of a specific version of a tool.

NixOS¶

Drawbacks:

• Not FSDG-compliant (although it filters out nonfree packages by default which should be sufficient for our pruposes). Pros:
• Highly reproducible.
• FSDG-compliance included.
• Affiliated with NLnet and NGI0 Discovery which we also affiliate with.
• Easy installation of a specific version of a tool.

Filling the missing parts¶

At some point we'll reach a situation where a certain build tool is not packaged for the distro or is present in a version that's inappropriate for building certain Haketilo package. There's no chance we can get a distro to package a tool just because we want so and to do it in a timely fashion. That's when we need to create our own, supplemental repository which would provide the required build tools. At that point we'll also need to give Hydrilla builder the ability to use custom repositories together with the main one.

Luckily, this is a problem we don't have to address now.

Naming build dependencies¶

The same tool might be named differently under different distros. If we want to enable building a certain Haketilo package under many distros, we have to specify how the build tools are named under each of them. We could do that inside the Haketilo source package definition, but that'd cause repetition when many Haketilo packages rely on the same tool. It'd be good to instead create meta-packages that specify how a build tool is named under each of supported distros. There'd be 1 meta-package for 1 build tool.

This is also something we need not be concerned about right now.

Relying on NPM¶

Many common tools for working with Web technologies are installable from Node Package Manager. This is what most people use for building and distributing JavaScript programs and integrating with it could make building Haketilo packages easier. Unfortunately, this is a bit more problematic than relying on distros. I explain below.

Software freedom¶

NPM repository is not FSDG-compliant. Even though it does require each project to declare a license from SPDX list, AFAIC it does not require it to be a free software license, not to mention fulfilling other requirements of FSDG. There's also no requirement for complete source code to be included in NPM package (although it is usually included by convention). This means relying on NPM could theoretically lead us to using running tools that have a free software license but are only distributed as obfuscated code². You might want to read Michael McMahon's article for some more insight into the problems of language-specific package managers.

Additionally, NPM allows anyone to register and upload a new package as long as its name is not yet used by another package. This by itself is a good thing because it fosters sharing. However, this also means NPM packages might have low hygiene.

In order to make NPM usable in freedom we need to spin up our own NPM repository with sanitized versions of packages from the official one. This might seem like an overkill but there're 2 important reasons to consider it:

1. A libre version of the NPM repo (and also of other language-specific package managers repos) is what the Free Software Community desperately needs. It would benefit more than just Haketilo.
2. We don't need to pull all free software packages from NPM. We should start with just the ones we need.

Missing tools¶

Not all tools related to web development are available through NPM. Some compiler toolchains, especially those targetting WebAssembly, are not in there and might never get included³. This means even with NPM integration Hydrilla builder would sometimes need to rely on some distro.

Work and experience needed¶

Regardless of the considerations above, I admit cloning NPM is not something we should do right now. That's especially true if I am considered, since I lack experience using NPM. Although tools to set up a private NPM registry are available, merely learning to use them and getting some packages served would cost too much precious time. Therefore, please consider all I wrote about NPM an idea we might want to implement somwehere in the future.

Relying on existing repositories for Haketilo packages as well¶

So far I discussed how we could satisfy Haketilo packages dependencies on build tools. However, many site fixes will not only require some build tools to be packaged but also have runtime dependencies on libraries. And it happens that the most common JS libraries are already packaged in distros like Trisquel and Guix. Instead of building these libs in terms of Hydrilla builder operation, we could arrange to use the artifacts provided by distros.

Let's consider JQuery as an example. Normally, we'd put JQuery sources in a Hydrilla source package. Hydrilla builder would perform any build steps necessary and would output the files of a JQuery Haketilo package ready to be served by Hydrilla.

Now, the alternative I am suggesting. Under Debian-based distros running apt install libjs-jquery places a non-minified JQuery script under /usr/share/javascript/jquery/jquery.js. Thus, to make a Haketilo JQuery package, we wouldn't even need to put JQuery sources inside our source package. We could instead make the source package use a "dummy" build procedure that simply copies over (or symlinks?) the JQuery script. The libjs-jquery APT package would of course be specified as a build dependency in the Hydrilla source package.

The same idea could be applied to NPM as well - facilitate making NPM packages into Haketilo libraries. Given the number of libraries in NPM this probably explains why I mentioned it in the first place...

benefits:

• It's easy to quickly make some popular JS libs into Haketilo packages and unblock some other tasks.
• It prevents us from duplicating the work distro packagers already did.
• We could cooperate with distro maintainers helping each other. drawbacks:
• It will be way harder or even impossible to make the build distro-agnostic.
• Packages already present in distros are not always built the way we expect. For example, the JavaScript file from libjs-jquery mentioned above also includes a library called Sizzle.js⁴. In Haketilo, we'd rather have JQuery and Sizzle as separate packages with a dependency of one on the other. This means we'd still want to package these on our own (but we can temporarily accept what distros provide).
• We will end up with some Haketilo packages being true packages and some (mostly libraries) being dummy packages. This looks like a severe inconsistency which I really dislike. We can either:
  • live with it or
  • require all Haketilo packages to be built as distro packages (this idea quite interestingly affects the other things⁵).
• What if some JS library is licensed under GPLv2-only? When distributing a built version we're also obligated to distribute the sources. But the true sources are not in the dummy Haketilo source package but rather in the distro's source package. This and similar nuances would be fairly easy to overlook. Also, having to set up a mirror of (parts of) distro's repo just to comply with a license seems like an overkill. Hopefully, this won't be a real issue in practice.
• How should we supply license information corresponding to the file(s) taken from distro package we're piggybacking on?
  • For JQuery, utilizing a Debian-based distro, we could just have the Haketilo package include everything under /usr/share/doc/libjs-jquery/. However, in case of other libs some of the files under /usr/share/common-licenses/ would need to be included as well.
  • I am yet to learn how this would look when utilizing Guix.

Keeping the source package format reasonable¶

How should we define build procedure (if at all⁵)? Debian packages use Makefiles. Arch packages use shell scripts.

Once we decide on strategies we want to adopt we could also talk at more length about how we want to change the source package format.

Request for comments¶

I started writing this with the goal of getting some feedback. Turns out I spent on it way more time than I planned but it also helped me clear my mind. Now I kind of see what I personally prefer.

Nevertheless, I'd still find any suggestions extremely helpful. Please, take voice :)