Haketilo

The second sounds perfect to me!

The first is also perfectly good, and I fully support it; while we could for logical completeness adopt a recursive structure analogous to the new bags/files system for pages/pattern->payload mappings, that would still preserve the basic system you suggest (if it isn't identical to this "modification") and could easily be added on later.

The remaining issue is the situation with conflicts like for Mr Cooldev and Miss Greatskill, for which I can see three options:

1. Check for conflicts and then "group" colliding patterns, forcing the user to select just one. This could be difficult to implement in a UI, as mappings in different pages will have to somehow be symbolically linked for the user to make a choice between them. Also, it might lead to superposition: if Mr Cooldev only makes critical fixes and so (say) doesn't create a payload for https://fansofbloat.com/flashybuttons, while Miss Greatskill creates one unified script that fully cleans up and restores every page of the form https://fansofbloat.com/*, then anyone who installs both and selects Mr Cooldev's work is not going to get the intended effect on the flashybuttons page.
2. Execute every payload mapped from a compatible pattern. It fixes the issues above; of course, anybody who doesn't realize they have conflicting extensions installed might get a very big surprise. There's also no chance of streamlining the settings query with this approach, and in fact it could grow more complicated.
3. We could mark packages as conflicting in the same way Debian does. The overwhelming difficulty here might be the fact that it prevents the "superposition" descrbed above: as bad as it can be in some cases, the user could actually want it if (say) Mr Cooldev hasn't fixed https://fansofbloat.com/donate yet.

Perhaps the best approach is some combination of 1 and 3.

As a response to https://hachettebugs.koszko.org/issues/38#change-111, I imagined a bag would contain a list of css files analogous to a list of scripts. This is, however, not urgent (for now CSS can also be embedded inside script) and should be easy to add later, so I concluded it should be left for now.

As to your other ideas, I was not commenting, because I have nothing to add

EDIT: Another IMHO cool idea would be to actually facilitate storing 2 versions of a script: a minified and non-minified one (and the same for minified CSS files, too!). User could then choose which version (minified or not) to inject. Btw, I noticed Debian puts javascript libraries under /usr/share/javascript/. Each script there is under both a minified and a non-minified version. Hydrilla will (by default) look for scripts under /var/lib/hydrilla/content. We can thus make symlinks there to actual Debian-supplied scripts, dropping some of the burden and getting some cleanly packaged js libraries to start with :)

As a response to https://hachettebugs.koszko.org/issues/38#change-111, I imagined a bag would contain a list of css files analogous to a list of scripts. This is, however, not urgent (for now CSS can also be embedded inside script) and should be easy to add later, so I concluded it should be left for now.

That makes sense- and indeed, it is not at all a priority.

EDIT: Another IMHO cool idea would be to actually facilitate storing 2 versions of a script: a minified and non-minified one (and the same for minified CSS files, too!). User could then choose which version (minified or not) to inject. Btw, I noticed Debian puts javascript libraries under /usr/share/javascript/. Each script there is under both a minified and a non-minified version. Hydrilla will (by default) look for scripts under /var/lib/hydrilla/content. We can thus make symlinks there to actual Debian-supplied scripts, dropping some of the burden and getting some cleanly packaged js libraries to start with :)

The idea of symlinking to Debian-supplied scripts would be really useful- I love it!

Storing minified and non-minified versions of files client-side (I assume that's what you meant?) could be interesting too; however, could you clarify where it might be used? I can understand maybe wanting a copy of the source as well as the "compiled" code, yet would have thought source maps or whatever they're called serve that purpose better. Nevertheless, if there's a reason I've missed, then that would definitely be a cool idea to implement!

Storing minified and non-minified versions of files client-side (I assume that's what you meant?) could be interesting too; however, could you clarify where it might be used? I can understand maybe wanting a copy of the source as well as the "compiled" code, yet would have thought source maps or whatever they're called serve that purpose better.

I know something like source maps exists but haven't yet learned how they work. Debian packages all three: non-minified js, source maps and minified js. If source maps indeed render original code unneeded, we can instead make it possible to store the maps in a bag. Or make all 3 possible to store simultaneously.

Anyway, the point is indeed to make the user able to easily (i.e. without downloading Debian-format source package) access the original code for reference/modifications but also have the minified version for when fast loading of page is preferred.

I know something like source maps exists but haven't yet learned how they work. Debian packages all three: non-minified js, source maps and minified js. If source maps indeed render original code unneeded, we can instead make it possible to store the maps in a bag. Or make all 3 possible to store simultaneously.

Anyway, the point is indeed to make the user able to easily (i.e. without downloading Debian-format source package) access the original code for reference/modifications but also have the minified version for when fast loading of page is preferred.

They might be a better fit here then. Source maps somehow contain the original files from which a bundled and minified script is built, along with mappings from locations in the compiled script to locations in the originals. That helps makes debugging easier with supportive devtools, and IIRC the data is stored in JSON file from which we could then recover the originals for modification.

That said, it will make things more difficult for us and mean even bigger files doing it this way.

That said, it will make things more difficult

Let's consider this a low-priority feature for adding later on.

and mean even bigger files doing it this way.

Well, maps don't need to be always downloaded. Only when the user requests them

we could for logical completeness adopt a recursive structure analogous to the new bags/files system for pages/pattern->payload mappings

With time I came to think this is indeed a good idea.

As to the resursive structure for pages and pattern mappings, let me give an argument in favor of it in case some passersby happen to read this thread. We could have 3 fixes for each of Google Docs, Google Sheets and Google Drive. These ought to be separate - the user should be able to enable one and not the others. At the same time, it would be nice if user who actually wishes to install the whole set of related fixes could do that conveniently. If we allow these 3 fixes to be implemented in separate packages and a 4th one to depend on those, everybody's happy.

Actually, now I would opt for an even more ambitious change to the schema I proposed. Merging bags and pages into one kind of entity that has a recursive relationship. Why?

1. Because managing 1 kind of entity is easier than managing 2 kinds.
2. Because many js files or sets thereof are going to be site-specific, and so, many fixes would unnecessarily require creation of 2 entities.

Despite proposing it I am not 100% sure it's a good idea, so I am waiting for comments now

That's even better! If we do merge everything, however, and maybe even if we don't, the "script/bag/etc." editor might need to be shown in a separate view to the list of items; otherwise the settings will become far too unwieldy to use.

That's even better!

I now got troubled as to how we're going to define re-usable payloads :/

the "script/bag/etc." editor might need to be shown in a separate view to the list of items; otherwise the settings will become far too unwieldy to use.

A pop-up dialog like the ones used when importing settings or selecting payloads?

EDIT: Perhaps with some good styling the separate view would not be necessary. Not that it would be bad, though.

I now got troubled as to how we're going to define re-usable payloads :/

We could just define them as packages that don't contain any pattern->payload mappings couldn't we? Admittedly, that would make settings_query somewhat more difficult.

the "script/bag/etc." editor might need to be shown in a separate view to the list of items; otherwise the settings will become far too unwieldy to use.
A pop-up dialog like the ones used when importing settings or selecting payloads?
EDIT: Perhaps with some good styling the separate view would not be necessary. Not that it would be bad, though.

Either of those could work; now that you mention it, however, avoiding a separate view might make it easier to apply whatever ends up being developed to the popup as well, which already has the issue of insufficient space.

avoiding a separate view might make it easier to apply whatever ends up being developed to the popup as well, which already has the issue of insufficient space.

Actually, the popup window can be stretched a bit. I just happened to give it some pretty low temporary dimensions.
But anyway, I am going to do something about the popup. Maybe add tabs to it? Actually, I already slightly modified the order of popup contents as part of my (not yet committed) developments

I had some new thought and updated https://hydrillabugs.koszko.org/projects/hydrilla/wiki/Repository_API

The new version looks great! Apart from signatures and related information, it looks to have everything we might need too.

The only aspect I don't quite understand is the UUID. Isn't the string identifier "+" dot-separated version sufficient for that purpose, or is there some particular use I'm missing that makes having a pre-defined/arbitrary UUID more suitable?

jahoti wrote:

The new version looks great! Apart from signatures and related information, it looks to have everything we might need too.

Really? I am constantly noticing shortcomings in what I prepare (just look at the number of edits...)

The only aspect I don't quite understand is the UUID. Isn't the string identifier "+" dot-separated version sufficient for that purpose, or is there some particular use I'm missing that makes having a pre-defined/arbitrary UUID more suitable?

This is purely to avoid having someone accidently create a resource with the same name as another already-existing resource. Maybe there is a better way to prevent this that I haven't realized?

EDIT
My current thoughts regrding the schema I came up with so far:

1. We're indicating licenses of files that are served. It doesn't work well in case of resources that need to be built. We should instead indicate licenses of unbuilt sources.
2. The above means we'd be best always conveying license information regarding the entire source of a Haketilo package, even when only one out of many of package's built files is downloaded.
3. My current way of indicating licenses is "reinventing the wheel".
   • I looked at REUSE today and I believe we could use it to generate SPDX documents for packages and use that. It seems waaaaay clearer than both my JSON definitions and Debian-format copyright file.
   • To make it easier to integrate existing code, we could optionally allow other copyright formats to be used, just as Debian allows non-standard copyright file to be used.
4. Python Hydrilla is currently meant to compute some values (auto license detection, etc.) and put them in JSON definitions it serves. Once we come to cryptographically sign those definitions, we'll need to have the computation happen before, on developer's machine.

Really? I am constantly noticing shortcomings in what I prepare (just look at the number of edits...)

Admittedly, I am not very good at thinking through all the possibilities it may have to deal with :). Perhaps it would be more accurate to say the general principle looks great.

The only aspect I don't quite understand is the UUID. Isn't the string identifier "+" dot-separated version sufficient for that purpose, or is there some particular use I'm missing that makes having a pre-defined/arbitrary UUID more suitable?

This is purely to avoid having someone accidently create a resource with the same name as another already-existing resource. Maybe there is a better way to prevent this that I haven't realized?

You were right I think; the suggestion essentially amounts to computing the UUID live from other parameters, which adds another layer of difficulty.

We're indicating licenses of files that are served. It doesn't work well in case of resources that need to be built. We should instead indicate licenses of unbuilt sources.

Indeed, in the eventual case of source packages we may need both sets of licenses indicated. Does that need to be accounted for in the current protocol?

I looked at REUSE today and I believe we could use it to generate SPDX documents for packages and use that. It seems waaaaay clearer than both my JSON definitions and Debian-format copyright file.

That's a good idea actually! The REUSE tool might also be usable then once we start accepting uploads from the general public, as a form of automated checking.

The only aspect I don't quite understand is the UUID. Isn't the string identifier "+" dot-separated version sufficient for that purpose, or is there some particular use I'm missing that makes having a pre-defined/arbitrary UUID more suitable?

This is purely to avoid having someone accidently create a resource with the same name as another already-existing resource. Maybe there is a better way to prevent this that I haven't realized?

You were right I think; the suggestion essentially amounts to computing the UUID live from other parameters, which adds another layer of difficulty.

Other parameters? What do you mean? The only thing that's as unique as UUID is the identifier.

Anyway, it seems we so far agree that UUIDs are OK and may stay in the schema.

We're indicating licenses of files that are served. It doesn't work well in case of resources that need to be built. We should instead indicate licenses of unbuilt sources.

Indeed, in the eventual case of source packages we may need both sets of licenses indicated.

Both sets? I'd rather indicate the licensing of sources in all cases, regardless of whether those are then built or not. Btw, that's what Debian does :) It uses a (already known to you) copyright file to describe the licenses of source files and installs it under /usr/share/doc/<package-name>.

Does that need to be accounted for in the current protocol?

I initially thought we could make the changes later. But now as I see so many shortcomings in the current protocol and actually have an idea how it should look like, I think it'd be better to make it good from the beginning. We'll be adding things to it in the future, of course, but many of the things we plan could be added without breaking backward compatibility if we make the initial protocol sane...

I looked at REUSE today and I believe we could use it to generate SPDX documents for packages and use that. It seems waaaaay clearer than both my JSON definitions and Debian-format copyright file.

That's a good idea actually! The REUSE tool might also be usable then once we start accepting uploads from the general public, as a form of automated checking.

Indeed. Does the REUSE-generated spdx report contain the actual license texts? If not, we'd take from each package a REUSE-style LICENSES/ directory together with the spdx report (and maybe also the DEP5 file if it proves needed and is present) and distribute that as the copyright info of each package. We'd then modify Haketilo to store files by their sha256 hashes so that even when many packages use GPLv3, it only gets downloaded and stored once

Does the REUSE-generated spdx report contain the actual license texts?

Only for licenses not in the official list; however, it shouldn't be too hard to patch the relevant code (https://git.fsfe.org/reuse/tool/src/branch/master/src/reuse/report.py#L166) for that purpose.

As for everything else, it appears there are many things I am either confused about or didn't know. I will have to look further into this!

Only for licenses not in the official list; however, it shouldn't be too hard to patch the relevant code (https://git.fsfe.org/reuse/tool/src/branch/master/src/reuse/report.py#L166) for that purpose.

Let's not... After the policy smuggling methods we tried, I think we've already used up our limit for ugly hacks...
This will do. In most cases the licenses will belong to those from the list hence we won't be wasting space for them anyway :)

EDIT

As for everything else, it appears there are many things I am either confused about or didn't know. I will have to look further into this!

Not that I knew about everything from the beginning. I first looked into REUSE specification a few days ago. I also first read an IndexedDB API guide this week

0gitnick wrote:

I wondered the same as Jahoti about having a UUID and I read your
response but it still seems redundant. Can you explain what situation
you imagine the identifier being useful where the UUID alone wouldn't
suffice?

I'm surprised to see this question because I thought it'd rather be the UUID and not identifier that we could decide is useless.

Identifier is good for specifying dependencies. In resource_a we can have:

"dependencies": [{
    "identifier": "resource_b"
}, {
    "identifier": "resource_c"
}]

while with just UUIDs we'd need to do sth like:

"dependencies": [{
    "uuid": "78e7d26b-74d3-48f0-bb91-a28af8e656af"
}, {
    "uuid": "7af51bc2-2054-40a4-9b8b-8f63374dc8f8"
}]

Same when specifying payloads. This would work but would not be usable (unless we automate index.json writing with some tool).

Going back to the opposite question - why we'd want the UUID in the first place - I'd like to add that it not only prevents us from making a mistake and adding independent packages with identifier clash to the repository. It can also save users when using third-party repositories that happened to use the same identifier for their custom packages without knowing about each other.
I admit the UUID is not clearly necessary. Most package managers function without such thing. What are the possible approaches?

1. Take example from java/Maven and put packages in namespaces (like com.someorganization.somepackage).
2. Rely on the platform (which we don't yet have) to restrict users from creating packages with identifiers already used by other users' packages (we're going to do that anyway; doesn't solve the issue with 3rd party repos)
3. Have Haketilo distinguish packages with different names but coming from different repositories. Allow having them installed side-by-side and have Haketilo consider package "X" from repo A unsuitable for satisfying the dependency of package "Y" from repo B on package "X". In cases where cross-repo dependencies are actually needed, we could make them explicit.
4. Just try to be careful enough to avoid identifier clashes and ask others to be careful too ¯\_(ツ)_/¯
5. Keep using UUIDs along identifiers

Regardless whether we choose 1-3, we can keep the UUIDs for now and remove them at some later point. Alternatively, we could make them optional from the beginning (and for example have Hydrilla builder show a warning but not an error when one is absent in index.json).

Any more thoughts on this?

0gitnick wrote:

I found the revision number definition confusing and I'm still not
exactly sure what it's for:
"description": "Which revision of a packaging of given version of an
upstream resource is this"

Let's say we have packaged version 1.3 of some third-party library. In the package we initially specified 1 as the revision. Full version string is now "1.3-1". Now let's say our package has a bug. It gets fixed but the upstream version remains the same. In this case the next version of our package will be 1.3-2, then 1.3-3 and so on. Once the upstream version increases, we reset the revision number back to 1 (so, for example, 1.3-3 could be "followed" by 1.4-1).

This strategy is commonly employed by distros for packaging. I meant my revision numbers to be the equivalent of Debian version in Debian packages¹.

Can you suggest how we could reword the definition?

Btw, I meant Hydrilla_source_package_format page to be a more gentle introduction to ideas like version and revision. Do you thing the explanation there also requires rewording? How about the Repository_API page? It doesn't carry such explanations and I don't want to duplicate them there but perhaps we should add there some references to the source package format page?

koszko wrote:

This (UUIDs) would work but would not be usable (unless we automate index.json writing with some tool).

I see. If I understand correctly, the problem is UUIDs aren't human-readable.

1. Take example from java/Maven and put packages in namespaces (like com.someorganization.somepackage).

I support this. If we don't do this, people might start doing it on their own by embedding the namespace in the identifier itself. If I make a package I'll probably set the identifier to "nicksphere-pkgname" or "ch.nicksphere.pkgname" for uniqueness since I don't know what others will name their packages.

3. Have Haketilo distinguish packages with different names but coming from different repositories. Allow having them installed side-by-side and have Haketilo consider package "X" from repo A unsuitable for satisfying the dependency of package "Y" from repo B on package "X". In cases where cross-repo dependencies are actually needed, we could make them explicit.

*the same names but coming from different repositories
If we go with namespaces, packages sharing the same namespace + package name + version should be unique and cross-repo dependencies will be fine. Not allowing cross-repo dependencies would mean the same dependency might be downloaded several times.

Regardless whether we choose 1-3, we can keep the UUIDs for now and remove them at some later point. Alternatively, we could make them optional from the beginning (and for example have Hydrilla builder show a warning but not an error when one is absent in index.json).

If we're going to keep UUIDs for now then I vote to make them optional.

4. Just try to be careful enough to avoid identifier clashes and ask others to be careful too ¯(ツ)/¯

If Haketilo is used by lay people, which I think is what we eventually want, then they won't know how to resolve identifier clashes. So it's best that we make sure they never happen rather than hoping.

Unless there's a downside I'm not considering, I think we should use namespaces and ditch UUIDs. It keeps packages human-readable and unique.

1. Take example from java/Maven and put packages in namespaces (like com.someorganization.somepackage).

I support this. If we don't do this, people might start doing it on their own by embedding the namespace in the identifier itself.

And that's what I've been suggesting. Where else are we going to put the namespace? Do you suggest we should give resources and mappings an additional "namespace" property?

If I make a package I'll probably set the identifier to "nicksphere-pkgname" or "ch.nicksphere.pkgname" for uniqueness since I don't know what others will name their packages.

This creates another issue - often we'd want to have targeted website's domain (or its part) in the identifier. Would org-koszko-drive-google-com be a sensible identifier? If not, what should be changed?

*the same names but coming from different repositories
If we go with namespaces, packages sharing the same namespace + package name + version should be unique and cross-repo dependencies will be fine.

You're probably right.

Not allowing cross-repo dependencies would mean the same dependency might be downloaded several times.

True. I rather meant requiring sth like {"identifier": "resource-x", "repository": "https://hydrilla.koszko.org"} to create dependency on a package from specific repo.

Regardless whether we choose 1-3, we can keep the UUIDs for now and remove them at some later point. Alternatively, we could make them optional from the beginning (and for example have Hydrilla builder show a warning but not an error when one is absent in index.json).

If we're going to keep UUIDs for now then I vote to make them optional.

Agreed. I'll make the necessary changes in the code.

Unless there's a downside I'm not considering, I think we should use namespaces and ditch UUIDs. It keeps packages human-readable and unique.

Since we don't yet have a clear idea of how the namespaces should look, I'll continue preparing the 1.0 release without any specific support for them. In the meantime we can keep discussing it

koszko wrote:

1. Take example from java/Maven and put packages in namespaces (like com.someorganization.somepackage).

I support this. If we don't do this, people might start doing it on their own by embedding the namespace in the identifier itself.

And that's what I've been suggesting. Where else are we going to put the namespace? Do you suggest we should give resources and mappings an additional "namespace" property?

If I make a package I'll probably set the identifier to "nicksphere-pkgname" or "ch.nicksphere.pkgname" for uniqueness since I don't know what others will name their packages.

This creates another issue - often we'd want to have targeted website's domain (or its part) in the identifier. Would org-koszko-drive-google-com be a sensible identifier? If not, what should be changed?

*the same names but coming from different repositories
If we go with namespaces, packages sharing the same namespace + package name + version should be unique and cross-repo dependencies will be fine.

You're probably right.

Not allowing cross-repo dependencies would mean the same dependency might be downloaded several times.

True. I rather meant requiring sth like {"identifier": "resource-x", "repository": "https://hydrilla.koszko.org"} to create dependency on a package from specific repo.

Regardless whether we choose 1-3, we can keep the UUIDs for now and remove them at some later point. Alternatively, we could make them optional from the beginning (and for example have Hydrilla builder show a warning but not an error when one is absent in index.json).

If we're going to keep UUIDs for now then I vote to make them optional.

Agreed. I'll make the necessary changes in the code.

Unless there's a downside I'm not considering, I think we should use namespaces and ditch UUIDs. It keeps packages human-readable and unique.

Since we don't yet have a clear idea of how the namespaces should look, I'll continue preparing the 1.0 release without any specific support for them. In the meantime we can keep discussing it

EDIT: Made UUIDs optional in both Hydrilla and Haketilo

EDIT2: Here's a repository of fixes translated to the new format. You can see what identifiers I chose (no namespacing by code vendor/author rn). You might notice that in case of most fixes we just needed one mapping and 1 resource - that makes me think about adding some shorthand for such scenario. But that's not something I want to do for Hydrilla 1.0.

koszko wrote:

If I make a package I'll probably set the identifier to "nicksphere-pkgname" or "ch.nicksphere.pkgname" for uniqueness since I don't know what others will name their packages.

This creates another issue - often we'd want to have targeted website's domain (or its part) in the identifier. Would org-koszko-drive-google-com be a sensible identifier? If not, what should be changed?

That's a bad idea. It's not clear where the namespace stops and where the identifier starts. Consider ch-nicksphere-www-google-com. Both www.nicksphere.ch and www.google.com are valid domains. The namespace and identifier are ambiguous.

A separate namespace property would be an improvement.

{
  namespace:  'ch-nicksphere-www',
  identifier: 'www-youtube-com'
}

Alternatively the identifier property could be changed. ([-0-9a-z]+\.[-0-9a-z]+)

{
  identifier: 'ch-nicksphere-www.www-youtube-com'
}

I prefer the separate namespace property because I find it easier on the eyes. Adding another field also doesn't break compatibility.

Adding another field also doesn't break compatibility.

In some way it does. Depends what we mean by compatibility. Sure, Hydrilla builder would tolerate a "namespace" property in index.json. But it wouldn't use it. In fact, the property would get stripped and namespace information lost. The resulting package would not then behave in the expected way.

This seems to be an issue in case of most small schema changes we might want to make :/ (with exception of cryptographic signatures support). Seems we'll bumping the major schema version numbers a lot. I even considered making Hydrilla builder keep unknown properties. But then we'd just hit the same problem in Hydrilla server or in Haketilo itself.

I'll quickly implement the namespaces now

EDIT: Seems like an significant amount of work. Changing my mind. We'll do it somewhere in the future, after 1.0 release