Project

General

Profile

Revamped javascript build system

Added by koszko about 2 years ago

Colby suggested making the build system cross-platform by rewriting it in javascript contained in a standalone html file.

Currently, the README's build instructions assume that the user is not
working on an adversarial system.  One thing I'd like to see with
Hachette, in order to maximize user freedom, is to not make this
assumption.  Instead, revamp the build system.  Consider someone
trying to work in a public library or from a lab where they do not
have control over the operating system of the machine that they're
running. They're unlikely to be able to use ./build.sh.  (And even
if the person trying to make modifications is using, say, a POSIX
system, subtle differences across ostensibly POSIX-compatible
implementations can cause problems.)

This can be ameliorated by redoing the build system so that a
description of the build process is detailed in a document (say
build.html) that lives in the repo root.  This document would be
responsible for both telling the user what steps are necessary, and to
function as the build script itself.  Under these circumstances, the
requirements to be able to modify/build Hachette would be exactly the
same as the requirements to use it--i.e., that they have access to a
Web browser.  This is a fairly safe assumption in general, but given
the nature of the project, it is known that this requirement will be
satisfied.

I am personally not convinced, but it seems like a nice hack nevertheless.


Replies (9)

Reply

RE: Revamped javascript build system - Added by jahoti about 2 years ago

It's definitely a nice hack. However, to actually implement it risks creating a real inconvenience for the sake of purely hypothetical potential contributors as far as I can see.

We could definitely add notes on how to edit the built packages- which are very readable, and will only be put in a zip archive- and manually incorporate patches made this way if the need arises. Until there is a concrete example of a potential contributor needing it, however, completely rewriting the build system doesn't seem a priority.

RE: Revamped javascript build system - Added by koszko about 2 years ago

On Mon, 5 Jul 2021 15:51:36 -0500
Colby Russell js-extensions-discussion@x.colbyrussell.com wrote:

On 7/5/21 8:54 AM, W. Kosior wrote:

What Jahoti already posted there is more or less what I would
respond if I was to keep arguing (although sometimes it's perhaps
better to stop and accept someone else's idea, so as not to waste
resources on useless wars).

I didn't realize we were in a war.


Ok, discussion can be considered moved to:
https://hachettebugs.koszko.org/boards/1/topics/1

As I understand it, what this means is that while neither you nor
Jahoti are enthusiastic enough to invest in personally implementing
this yourselves, you would be willing to accept patches if someone
else were willing to make the investment?

https://www.colbyrussell.com/2013/08/06/patches-wanted.html

If that were the case, it would appear that we're past the main hurdle
of the patch paradox.

https://www.colbyrussell.com/2017/08/06/contributors-dilemma.html

I.e., I can be confident that I wouldn't be at risk of wasting my own
resources to make this happen only to be turned away after the fact
due to lack of willingness interest--again, if that's actually the
case.

Can I get a clarification on this?

PS:

discussion can be considered moved

I just tried to reset my password, but it doesn't seem to be working,
so I can't log in.

PPS: Since my account is completely unused at this point, I would just
create a one, but IIRC, those require manual intervention, too. Maybe
consider allowing open registration for now--not just for me, but
again: other potential contributors--and if it becomes evident that
it's a problem, shutting it off at that point.  (But keyword being
"evident"!)

Yeah, my Redmine configuration is not really complete: https://hachettebugs.koszko.org/boards/2/topics/3
I will also enable open registrations once I sit at it.

As to war - we were not at it. Yet ;)

My stance regarding your idea is exactly as you said it: I shall not devote my personal time to such build system, but I shall accept your patches

On Mon, 5 Jul 2021 23:03:00 +0000
jahoti jahoti@envs.net wrote:

On 7/5/21 8:51 PM, Colby Russell wrote:

As I understand it, what this means is that while neither you nor
Jahoti are enthusiastic enough to invest in personally implementing
this yourselves, you would be willing to accept patches if someone
else were willing to make the investment?

https://www.colbyrussell.com/2013/08/06/patches-wanted.html

If that were the case, it would appear that we're past the main
hurdle of the patch paradox.

https://www.colbyrussell.com/2017/08/06/contributors-dilemma.html

I.e., I can be confident that I wouldn't be at risk of wasting my
own resources to make this happen only to be turned away after the
fact due to lack of willingness interest--again, if that's actually
the case.

Can I get a clarification on this?

It's of course not my place to accept or reject patches;
nevertheless, for what it's worth your interpretation corresponds
accurately to my opinion. While there are some additional
reservations I didn't write in the referenced comment (e.g. drawing
in masses of unnecessary external libraries), you already seem to
have brought those up in your journal and quite frankly seem like one
of the best people around to implement this idea appropriately.

RE: Revamped javascript build system - Added by koszko about 2 years ago

Colby, how is your progress?

You know, what? I thought it might be difficult to create the build system contained inside an HTML page because we have to read and write multiple files when performing the build. I then though you might actually choose to load zipped sources, unzip them in js, build, zip the result and prompt the user for a location to save it. Chances are you actually came up with some other solution (our ways of thinking seem to differ a lot). But in case you somehow came up with the same one, I'd like to ask you whether you used jszip[1] and how you plan to pull it (it seems Debian has it packaged, so it's not that bad).

I was interested in this because I was considering replacing JSON-based import/export system with a zip-based one, which would involve pulling some zip library into the extension. After a minute of thinking I decided this would incur too much bloat but I decided to ask you anyway.

[1] https://github.com/Stuk/jszip

RE: Revamped javascript build system - Added by koszko about 2 years ago

I now realized there is a web API for accessing entire directories:
https://developer.mozilla.org/en-US/docs/Web/API/File_and_Directory_Entries_API

No need for zip library then :)

RE: Revamped javascript build system - Added by colby over 1 year ago

Sorry, I never received these messages. (Not CCed, I guess; still not sure how to use Redmine to update the CC list myself.)

Colby, how is your progress?

I may have been a little bit coy in my earlier messages. When I brought up the subject, I wasn't considering the idea for the first time. I already have a build system that works on this principle and is in use for other projects. A walkthrough at https://triplescripts.org/example explains the relevant concepts. I was trying to sell you on one aspect (in-browser builds), but what I've been doing with triple scripts encompasses even more than that. For example, since project maintainers and regular contributors are unlikely to favor the overhead of having to open "build.html" in their browser versus a more ergonomic CLI-based workflow, the file format is devised to permit the same build script to optionally be run in the terminal.

Here's a 2 minute video demo where I show how these concepts can be applied even to systems programming—at least for embedded/small systems:

https://youtu.be/TUpd70Mu0Ek

Mirror: https://diode.zone/w/sUfRZgdxYe2yLPgH3h6Dqv

RE: Revamped javascript build system - Added by colby over 1 year ago

koszko wrote:

it might be difficult to create the build system contained inside an HTML page because we have to read and write multiple files

In practice, this isn't really a problem, because in practice you don't actually want the build system to be writing multiple files. A well-behaved build system outputs exactly one artifact (the thing you're trying to build) and doesn't take any liberties in assuming that it can write stuff wherever it wants. This is another strength of the browser sandbox, and not a shortcoming. In a world where this principle is more widespread, you as a casual user stumbling across a project on some repo hosting site somewhere can (a) know that you're not going to have any trouble performing the build, because there is no "implicit step 0" that you have to work out (e.g. "[install such-and-such runtime]" or "[get hold of this specific distro because that's what the maintainers are using, and they haven't tested it elsewhere]"), and (b) you can confidently run the build script and know that when you're prompted to save e.g. "haketilo-0.2-pre.xpi" that this is the extent of the build script's influence—that there are no other side effects of running the build script. The build process becomes a pure function that takes the project source as its input and produces the build artifact as its output.

I was interested in this because I was considering replacing JSON-based import/export system

(Side note: not sure what you're referring to here, but I noticed when I looked at the state of the extension ~2 months ago that it's using magic comments with IMPORT(/EXPORT?) directives and some runtime trickery. Any reason why you didn't use ECMAScript's standardized import declarations?)

a zip-based one[...] would involve pulling some zip library into the extension. After a minute of thinking I decided this would incur too much bloat

I've looked into this for my own purposes but not had a problem, for the reasons explained above. Of course, for Haketilo, you will want a ZIP writer, since XPI is just ZIP with a non-standard file extension. However, the ZIP container format is fairly straightforward, cf https://codeberg.org/triplescripts/trplkt/src/branch/release/v0.12/src/ZodLib.src#L26

Despite this, I have put serious effort into researching the ZIP implementations from the NodeJS world in the past. The state of art there doesn't inspire confidence. If I needed compression/decompression, then I'd either go with something like https://github.com/greggman/uzip-module—which is not widely used but better than the other options I've come across—or I'd port Project Nayuki's reference code.

RE: Revamped javascript build system - Added by koszko over 1 year ago

colby wrote:

Sorry, I never received these messages. (Not CCed, I guess; still not sure how to use Redmine to update the CC list myself.)

Lol! I didn't think we'd ever see you post here again - I thought you lost your interest in this project :p

You shall receive email notifications once you check the "Watch" star that appears at the top right on issue pages. Also, it's possible that I set up and enabled email notifications only after the messages here were posted. Nvm.

Colby, how is your progress?

I may have been a little bit coy in my earlier messages. When I brought up the subject, I wasn't considering the idea for the first time. I already have a build system that works on this principle and is in use for other projects. A walkthrough at https://triplescripts.org/example explains the relevant concepts. [...]

Quite an interesting invention! And I suppose it should be easy to run triple scripts under standalone ECMAScript runtimes other that V8 as well... However, I am a bit confused. Sure, your triple scripts would be more than sufficient for building a browser extension. But you seem to be talking about build systems in general. Are you going to be able to for example build C programs this way?

I was interested in this because I was considering replacing JSON-based import/export system

(Side note: not sure what you're referring to here,

The "Import" button that was on the extension's settings page and allowed user to select a JSON file describing scripts to inject on sites. I have not yet restored that functionality after remaking the guts of Haketilo (oh, and in case you haven't noticed - we changed the extension's name in the meantime).

but I noticed when I looked at the state of the extension ~2 months ago that it's using magic comments with IMPORT(/EXPORT?) directives and some runtime trickery. Any reason why you didn't use ECMAScript's standardized import declarations?)

I did, at the very beginning of prototyping (~1 year ago). Only to find out JS modules are not fully supported in WebExtensions and to then go through the pain of making it normal JS. I ended up adding those magic comments to help with code modularization.
Since then I've replaced the sh build script using those comments with a simple preprocessor written in awk (which handles IMPORT/EXPORT directives but also some more stuff). All with the goal of avoiding hard build-time dependency on bundlers and Node.

a zip-based one[...] would involve pulling some zip library into the extension. After a minute of thinking I decided this would incur too much bloat

[...] the ZIP container format is fairly straightforward, cf https://codeberg.org/triplescripts/trplkt/src/branch/release/v0.12/src/ZodLib.src#L26

Despite this, I have put serious effort into researching the ZIP implementations from the NodeJS world in the past. The state of art there doesn't inspire confidence. If I needed compression/decompression, then I'd either go with something like https://github.com/greggman/uzip-module—which is not widely used but better than the other options I've come across—or I'd port Project Nayuki's reference code.

I can't seem to find Project Nayuki's ZIP code, am I looking in a wrong place1? Anyway, thank you for mentioning all those. A simple ZIP library that doesn't pull 100 dependencies is exactly what I wanted.

RE: Revamped javascript build system - Added by colby over 1 year ago

koszko wrote:

I am a bit confused. Sure, your triple scripts would be more than sufficient for building a browser extension. But you seem to be talking about build systems in general. Are you going to be able to for example build C programs this way?

It's outside the scope of this issue, but yeah, that's viable. Check out the screencast linked earlier which does a 3-stage build, including cross-compiler for the foreign RSC object format and the Oberon language. This produces a runnable Oberon disk image. There's no reason the same couldn't be done for C programs.

JS modules are not fully supported in WebExtensions and to then go through the pain of making it normal JS. I ended up adding those magic comments to help with code modularization

Why invent a new syntax and linking model, though? You could continue to use standard import declarations.

I can't seem to find Project Nayuki's ZIP code, am I looking in a wrong place[nayuki]?

Here:

RE: Revamped javascript build system - Added by koszko over 1 year ago

JS modules are not fully supported in WebExtensions and to then go through the pain of making it normal JS. I ended up adding those magic comments to help with code modularization

Why invent a new syntax and linking model, though? You could continue to use standard import declarations.

A very limited subset of them? Perhaps. But that just didn't occur to me at that point. And the way I initially did it only required the build script to add some boilerplate around the source from each file. The actual code could remain unmodified thanks to the import/export directives appearing inside comments.

Since then I did move IMPORT/EXPORT out of comments and I also included some handy directives like:

#IF MV3
do_sth();
#ENDIF

and:

const some_variable = 
#INCLUDE_VERBATIM some.json
;

Right now those serve their purpose well enough that we can focus on working on the actual extension

    (1-9/9)

    Reply