Haketilo: Development - archivehttp://hydrillabugs.koszko.org/http://hydrillabugs.koszko.org/favicon.ico?15861920342021-08-06T00:52:29ZHydrilla issue tracker
Redmine Development - archive: RE: Injecting scripts without downloading themhttp://hydrillabugs.koszko.org/boards/5/topics/66?r=81#message-812021-08-06T00:52:29Zjahoti
<blockquote>
<p>No. The way I wrote Hydrilla it assumes [...]</p>
</blockquote>
<p>That makes sense! All your other points follow then, I think, except that most computer users and IT students have no idea what certificates are- which unfortunately I don't disagree with you about :/.</p>
<blockquote>
<p>Plus it was simply easier for me to program this way because I could leave the work of serving script files to Apache :p</p>
</blockquote>
<p>LOL- don't change that!</p>
Development - archive: RE: Injecting scripts without downloading themhttp://hydrillabugs.koszko.org/boards/5/topics/66?r=79#message-792021-08-05T10:10:16Zkoszkokoszko@koszko.org
<blockquote>
<p>In any case, if Microsoft ships the tools to generate self-signed certs, I expect that would make enabling HTTPS reasonably doable.</p>
</blockquote>
<p>We don't need M$' tools for this. I've been able to use Wine and MinGW to develop Windows programs in the past using only libre software. And even linked with OpenSSL <sup>^</sup><br>
I am also looking forward to be able to do the same for Crapple macOS using Darling.</p>
<blockquote>
<p>Would there be any significant group of people who want to set up a local proxy yet couldn't (with some searching) determine how to deal with the certificates, regardless of OS?</p>
</blockquote>
<p>If we want to make it a viable solution to the issues mentioned, we need to make the biggest possible subset of our users want to set it up.<br>
Most computer users and even many IT students are completely unaware of what certificates are. While learning about it is possible, it is not possible in a short time (~3 mins a user expects to spend installing stuff). If this is to be a viable solution, we need to automate everything. Also, if we don't, we might look unprofessional :/</p>
<blockquote>
<blockquote>
<blockquote>
<p>That said, I'm somewhat confused as to how this would/could put burden on Hydrilla.<br>
I silently assumed browser's own caching mechanisms would result in more re-downloads than if a proper Hydrilla-aware caching of scripts was in place. I might be wrong :)</p>
</blockquote>
</blockquote>
<p>You're definitely right about that! The point of confusion, which I now realize I never actually stated, was to do with it affecting Hydrilla; aren't scripts served by Hydrilla copied locally anyway, leaving only scripts from external sources specified by hash and url?</p>
</blockquote>
<p>No. The way I wrote Hydrilla it assumes (unlike Hachette) that the scripts are (1) never referenced from external sources (except approved mirrors) and yet are (2) always served by hash and url. Why so? (1) because we don't want to rely on third-party servers but rather server the same thing ourselves and (2) because this is more flexible. We'll add a "pull source" button in the settings which will enable the user to get a copy anyway. And once we come to serve various versions of the same script (minified, non-minified, map) we won't be wasting bandwidth by sending the source together with the actual settings. Plus it was simply easier for me to program this way because I could leave the work of serving script files to Apache :p</p>
<p>I actually assumed Hydrilla would only inform the clients about a path to script resource (e.g. "somedir/somefile.js") and that path would then work when appended to any of the mirrors of that repository (e.g. "<a href="https://hydrilla-repo1.org/content/somedir/somefile.js">https://hydrilla-repo1.org/content/somedir/somefile.js</a>" and "<a href="https://hydrilla-repo-service.org/mirror-of-koszko_org/content/somedir/somefile.js">https://hydrilla-repo-service.org/mirror-of-koszko_org/content/somedir/somefile.js</a>". We could then even separate the API server from actual mirrors <sup>^</sup></p>
Development - archive: RE: Injecting scripts without downloading themhttp://hydrillabugs.koszko.org/boards/5/topics/66?r=75#message-752021-08-05T09:30:12Zjahoti
<blockquote>
<blockquote>
<p>it wouldn't be too hard to enable mirrors over HTTPS in any case<br>
Depends for whom. Developing this should be easy, but if a user decides to set up a mirror on localhost, at least a self-signed cert would need to be created and imported into the browser. This can probably be automated when using a package manager, though (sorry, Losedows useds).</p>
</blockquote>
</blockquote>
<p>Somehow that fact managed to slip my mind despite the fact that I've been thinking about certificates quite regularly in the past few days (as part of a potential testing system).</p>
<p>In any case, if Microsoft ships the tools to generate self-signed certs, I expect that would make enabling HTTPS reasonably doable. Would there be any significant group of people who want to set up a local proxy yet couldn't (with some searching) determine how to deal with the certificates, regardless of OS?</p>
<blockquote>
<blockquote>
<p>That said, I'm somewhat confused as to how this would/could put burden on Hydrilla.<br>
I silently assumed browser's own caching mechanisms would result in more re-downloads than if a proper Hydrilla-aware caching of scripts was in place. I might be wrong :)</p>
</blockquote>
</blockquote>
<p>You're definitely right about that! The point of confusion, which I now realize I never actually stated, was to do with it affecting Hydrilla; aren't scripts served by Hydrilla copied locally anyway, leaving only scripts from external sources specified by hash and url?</p>
Development - archive: RE: Injecting scripts without downloading themhttp://hydrillabugs.koszko.org/boards/5/topics/66?r=72#message-722021-08-04T08:30:02Zkoszkokoszko@koszko.org
<blockquote>
<p>it wouldn't be too hard to enable mirrors over HTTPS in any case</p>
</blockquote>
<p>Depends for whom. Developing this should be easy, but if a user decides to set up a mirror on localhost, at least a self-signed cert would need to be created and imported into the browser. This can probably be automated when using a package manager, though (sorry, Losedows useds).</p>
<blockquote>
<p>That said, I'm somewhat confused as to how this would/could put burden on Hydrilla.</p>
</blockquote>
<p>I silently assumed browser's own caching mechanisms would result in more re-downloads than if a proper Hydrilla-aware caching of scripts was in place. I might be wrong :)</p>
Development - archive: RE: Injecting scripts without downloading themhttp://hydrillabugs.koszko.org/boards/5/topics/66?r=69#message-692021-08-03T21:54:01Zjahoti
<p>I think it would be a great idea, especially given the mirror suggestion you provide (it wouldn't be too hard to enable mirrors over HTTPS in any case).</p>
<p>That said, I'm somewhat confused as to how this would/could put burden on Hydrilla. Aren't scripts served by Hydrilla downloaded using the text key?</p>
Development - archive: Injecting scripts without downloading themhttp://hydrillabugs.koszko.org/boards/5/topics/662021-08-02T23:37:09Zkoszkokoszko@koszko.org
<p>For scripts with URL and SHA256 sum stored in the settings, would it make sense to inject them as <code><script></code> with <code>src</code> attribute instead of downloading with AJAX?</p>
<p>Cons:</p>
<ul>
<li>more burden on Hydrilla as users' browsers will re-download scripts from time to time (can be mitigated if people set up mirrors)</li>
<li>users' higher reliance on centralized servers providing scripts</li>
</ul>
<p>Pros:</p>
<ul>
<li>lower complexity of Hachette - we won't need to care about [1] and [2] as browser will do it for us</li>
</ul>
<p>What do you think?</p>
<p>[1] <a href="https://hachettebugs.koszko.org/issues/1">https://hachettebugs.koszko.org/issues/1</a> - "parallelize fetching of remote scripts"<br>
[2] <a href="https://hachettebugs.koszko.org/issues/4">https://hachettebugs.koszko.org/issues/4</a> - "make it possible to cache remote scripts"</p>
<p>EDIT: Perhaps all cons can be mitigated if user sets up a private mirror on localhost (we can develop Hydrilla to be able to function as a mirror or caching proxy for another Hydrilla instance). We then only need some way to allow scripts from <a href="http://localhost:*port*">http://localhost:*port*</a> to execute on HTTPS sites</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=38#message-382021-07-09T01:14:16Zjahoti
<p>Really only Python, JavaScript, and shell scripting (if that can really be called a language). I've studied theoretrically and very much love the idea of functional programming; however, I've never actually taken the time to learn such a language.</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=37#message-372021-07-08T19:39:09Zkoszkokoszko@koszko.org
<p><a class="user active" href="http://hydrillabugs.koszko.org/users/6">jahoti</a> wrote:</p>
<blockquote>
<p>somebody who has almost no experience with <em>any</em> of those languages...</p>
</blockquote>
<p>Out of curiosity: what languages do you have experience with?</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=36#message-362021-07-08T09:13:59Zjahoti
<p>My comment about Go was probably too emphatic; it was only intended as an alternative explanation for why Go might be relevant, rather than as advice which would be very inappropriate from somebody who has almost no experience with <em>any</em> of those languages...</p>
<p>Perhaps I should learn them, given they aren't OOP-based and I am a proceduralist/functionalist :).</p>
<p>My choice remains the same as originally: do whatever is easiest for you, and worry about technical merit only once we have a proof of concept.</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=35#message-352021-07-08T08:50:52Zkoszkokoszko@koszko.org
<blockquote>
<blockquote>
<blockquote>
<p>Go is a language with build system that fetches dependencies from places like github repos, right? That kind of falls in line with your other preferences :P</p>
</blockquote>
</blockquote>
<p>It can do that; however, it is also strongly oriented towards creating servers and server software, which does objectively fit this application. </p>
</blockquote>
<p>Btw, Go, C and Erlang all share one uncommon quality - they have not fallen for the OOP fashion.<br>
Erlang is also suited for creating servers and C... well, Apache and Nginx are themselves written in C. I can agree for Erlang but not for Go (which I dont know) make a choice ;)</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=34#message-342021-07-07T23:49:15Zjahoti
<blockquote>
<blockquote>
<p>If you are committed to your existing vision, how would you feel about writing such a "server" that is intended to be installed on a multi-user system (like any tildeverse host)<br>
Could you elaborate? How does a tildeverse host work and how you're going to utilize it (unfortunately, looking up things like this costs me too much time, so I decided to start asking people who bring the topic instead; I hope you understand).</p>
</blockquote>
</blockquote>
<p>Tildeverse hosts are more-or-less standard shared hosting, except with the goal of incubating a "maker community" rather than selling a service. They therefore offer free accounts to interested individuals, offer on-host IRC and bulletin boards, install all sorts of compilers and scripts, and so on. The idea of writing for a multi-user system would then be, I assume, that such hosts can install the program and all the hackers who use them can then publish their scripts.</p>
<blockquote>
<blockquote>
<p>Go is a language with build system that fetches dependencies from places like github repos, right? That kind of falls in line with your other preferences :P</p>
</blockquote>
</blockquote>
<p>It can do that; however, it is also strongly oriented towards creating servers and server software, which does objectively fit this application. </p>
<p>As to the actual topic at hand, there are really two separate points to make here.</p>
<p>Firstly, while I am sympathetic to the importance of decentralization, the point <a class="user active" href="http://hydrillabugs.koszko.org/users/5">koszko</a> makes in regard to maintaining primacy of the project-maintained repository is also critical to ensuring users do not have to choose between manually vetting most scripts they install and missing out on a significant proportion of available fixes. It is nonetheless an entirely social problem we have not assessed other possible solutions for; do you have any suggestions, <a class="user active" href="http://hydrillabugs.koszko.org/users/7">colby</a> ?</p>
<p>Secondly, I fully agree with the idea of not re-inventing the wheel and can definitely see the potential in using an already existing setup like Hypothes.is, provided we first confirm it could easily serve our needs now and into the future. The specific case of Hypothes.is nevertheless appears problematic; the official implementation is (as noted) overcomplicated, and <a href="https://github.com/judell/MinimalHypothesisServer">https://github.com/judell/MinimalHypothesisServer</a> is not released under a free license.</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=31#message-312021-07-07T17:18:20Zkoszkokoszko@koszko.org
<blockquote>
<p>We get decentralization "for free" this<br>
way--anyone with access to a free/cheap static web host can dump their script<br>
there and then point others to it with an annotation on the original site.</p>
</blockquote>
<p>Only script hosting would be decentralized this way and not annotation hosting.</p>
<blockquote>
<p>The fact that these are just ordinary annotations that one might stumble upon<br>
"in the wild" outside the context of Hachette is something that also has the<br>
potential to help with project awareness.</p>
</blockquote>
<p>It does... although it's a factor small enough we don't have to consider it.</p>
<blockquote>
<p>Any design for decentralization that depends on people downloading the server<br>
software (or writing their own compatible server) and running an instance<br>
themselves as a precondition to hosting off-main repostory scripts is pretty<br>
much doomed. It's a nice idea, but what happens is that by and large no one<br>
really does that, and of the few who do, it tends to be a short-lived hobby;<br>
people are attracted to the <em>idea</em> of doing so, but eventually get bored, no<br>
longer have time to admin the instance, and then move on.</p>
</blockquote>
<p>It seems you care about decentralization a lot. The idea of "downloading the server software and running an instance" is used in case of Debian's repos and Wikipedia - 2 projects I take inspiration from. Making off-main repository hosting that dumb simple and effortless does not yet - in my opinion - justify a bizarre design choice. We could (and should) instead provide a sandbox for users within the platform we develop. Consider it an equivalent of Ubuntu's PPA service or Wikipedia pages where beginner edits have to be approved by some more credited wikipedian but can be viewed by those who explicitly click to see them.</p>
<blockquote>
<blockquote>
<p>even if we're to use the existing infrastructure, we're not going to be<br>
doing that forever, are we?</p>
</blockquote>
<p>I dunno. Why not?</p>
</blockquote>
<p>I see our expeciations of the final outcome of this project are quite different. I imagine a strong-standing platform like Wikipedia or Debian which is <em>the</em> one meant to be primarily used with the tool(s) we're developing. Such platform would be forkable (which is a must if we're not to violate software freedom principles), example of that in case of Debian is Ubuntu. However, the platform itself should be centralized enough to allow us to impose some rules on scripts served, most notably FSDG compliance.</p>
<p>If you're much into decentralization, you might dislike that idea. However, the everybody-can-easily-host idea, in case of software, leads to the problem we have with AUR, Dockerhub, as well as most language-specific package repositories (there are exceptions, though) - they mostly allow nonfree packages and also make no requirements or even recommendation towards good practices. This leads to packages with dependencies on multiple versions of the same library, unclearly indicated licensing, packages built a "dirty" way (e.g. build utilizes some bundled binary programs or libs instead of using 1st party versions of these), lack of standardized way of building from source and probably other issues that didn't come to my mind right now. Playground for users is not a bad thing, I just don't want the main repo to become a playground.</p>
<p>The level of control I expect is not impossible to achieve with hypothes.is annotations, once we start utilizing PGP signatures (which we should anyway at some point - that's how serious software repos work; mere TLS is not trustworthy enough). However, annotations don't help (with the issues I described), either.</p>
<p>Btw, I do realize js for sites is somewhat different from usual software packages, because sites change often and we need to be elastic. However, I still want to maintain some level of stability and security (probably justified given that we might at some point start writing custom js for online banking sites). So there is obviously not going to be a 2-year schedule, but some policy is to be enforced in the "official" repo.</p>
<p>Related topic: <a href="https://hachettebugs.koszko.org/boards/1/topics/28">https://hachettebugs.koszko.org/boards/1/topics/28</a></p>
<blockquote>
<p>If you are committed to your existing vision, how would you feel about<br>
writing such a "server" that is intended to be installed on a multi-user<br>
system (like any tildeverse host)</p>
</blockquote>
<p>Could you elaborate? How does a tildeverse host work and how you're going to utilize it (unfortunately, looking up things like this costs me too much time, so I decided to start asking people who bring the topic instead; I hope you understand).</p>
<blockquote>
<p>I will say that I think going with C is a bad choice for a server, for<br>
both technical and social reasons (except perhaps for the microcosmic<br>
use case above as an exception, where it might be considered a strength,<br>
i.e., attractive to the types of people who operate and sign up for<br>
tilde-style systems). Have you considered Go?</p>
</blockquote>
<p>I haven't because I don't know it. Wait, Go is a language with build system that fetches dependencies from places like github repos, right? That kind of falls in line with your other preferences :P</p>
<p>As to simplicity, C is simple in some senses. It requires the minimal amount of runtime dependencies. Another plus is that C libraries are most often available from distros' repos, making it quite effortless to avoid FSDG-incompliant dependency sources. In that it beats both Erlang and Go.</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=30#message-302021-07-07T14:59:57Zcolbyhachettebugs@x.colbyrussell.com
<blockquote>
<p>Or do you mean we would then finally develop our own server software,<br>
as I want to do right now?</p>
</blockquote>
<p>No one can demand that you spend your time in a certain way to keep you<br>
from doing what you want to do. I'd just recommend, as I always do, to<br>
first do the simplest possible thing that works. From my perspective,<br>
that entails not having to develop and manage our own infrastructure--<br>
especially since that's a serial process, and the development part is a<br>
blocker.</p>
<p>If you are committed to your existing vision, how would you feel about<br>
writing such a "server" that is intended to be installed on a multi-user<br>
system (like any tildeverse host) for tracking the scripts contributed<br>
by the community there--as a microcosm (and proving ground) for what you<br>
have in mind? Globally, it would still make things public for discovery<br>
by Hachette under the principles described before, but this server could<br>
be e.g. tailored towards assisting that system's users get their scripts<br>
into their Web space on that system convieniently, provide an additional<br>
frontend for showcasing (to the public and/or internally) the work that<br>
that community is doing to create free alternatives of non-free scripts,<br>
etc.</p>
<p>I will say that I think going with C is a bad choice for a server, for<br>
both technical and social reasons (except perhaps for the microcosmic<br>
use case above as an exception, where it might be considered a strength,<br>
i.e., attractive to the types of people who operate and sign up for<br>
tilde-style systems). Have you considered Go?</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=29#message-292021-07-07T14:51:27Zcolbyhachettebugs@x.colbyrussell.com
<blockquote>
<p>Annotations seem suitable for advertising facts about websites and getting<br>
feedback. But are they also for sharing the actual scripts?</p>
</blockquote>
<p>Annotations <em>could</em> be used for very short scripts. But what I had in mind is<br>
hypothes.is being used purely as the side channel for annotating the original<br>
site with notices that there are replacements available for the non-free<br>
scripts used there--but the replacement script itself would be hosted<br>
elsewhere (wherever the script author chooses to put it), and the annotation<br>
would contain a link to it. We get decentralization "for free" this<br>
way--anyone with access to a free/cheap static web host can dump their script<br>
there and then point others to it with an annotation on the original site.</p>
<p>Note also that when I say use Hypothesis here, I'm not necessarily talking<br>
about instructing users to install the official Hypothesis client and create<br>
these annotations by hand (and worry about their well-formedness). I mean<br>
using hypothes.is as a carrier; Hachette itself, in addition to necessarily<br>
being able to understand this "protocol" so it can discover available<br>
replacement scripts from well-formed (appropriately tagged) annotations, would<br>
also be able to create and post them.</p>
<p>The fact that these are just ordinary annotations that one might stumble upon<br>
"in the wild" outside the context of Hachette is something that also has the<br>
potential to help with project awareness.</p>
<p>Any design for decentralization that depends on people downloading the server<br>
software (or writing their own compatible server) and running an instance<br>
themselves as a precondition to hosting off-main repostory scripts is pretty<br>
much doomed. It's a nice idea, but what happens is that by and large no one<br>
really does that, and of the few who do, it tends to be a short-lived hobby;<br>
people are attracted to the <em>idea</em> of doing so, but eventually get bored, no<br>
longer have time to admin the instance, and then move on. This will of course<br>
happen even under my proposed design (people abandoning, say, a tildeverse<br>
account tied to the namespace where their contributed scripts are hosted), but<br>
the fact that contributed scripts just live as ordinary resources on the Web<br>
means that we can pretty easily make sure they're backed up (e.g. using the<br>
Wayback Machine; we would also get versioning "for free" this way).</p>
<blockquote>
<p>Doesn't server rely on nonfree Elasticsearch?</p>
</blockquote>
<p>I'm not intimately familiar; I don't know, maybe at some point since the<br>
Elasticsearch license changes in January there has been some statement from<br>
Hypothesis aligning themselves with Elasticsearch and committing to it for the<br>
future, instead of either sticking with an older, free version or moving to<br>
the community-spawned fork. Even if that's true, though, it has low impact;<br>
there should be no reason that anyone looking to deploy their own<br>
infrastructure would be advised to use ElasticSearch themselves instead of<br>
OpenSearch, just because hypothes.is is committed to the former--and again, I<br>
don't even know that that is the case.</p>
<blockquote>
<p>even if we're to use the existing infrastructure, we're not going to be<br>
doing that forever, are we?</p>
</blockquote>
<p>I dunno. Why not?</p>
<blockquote>
<p>Once we reach the point we depart from Hypothesis service, are we going to<br>
have to deploy all their big software system with many dependencies and<br>
features we don't need?</p>
</blockquote>
<p>See <a href="https://github.com/judell/MinimalHypothesisServer">https://github.com/judell/MinimalHypothesisServer</a>. (I myself am not a<br>
fan of how complex and complicated the official client and server are.)</p>
Development - archive: RE: server technology considerationshttp://hydrillabugs.koszko.org/boards/5/topics/17?r=26#message-262021-07-07T10:53:49Zkoszkokoszko@koszko.org
<p>colby wrote:</p>
<blockquote>
<p>Just to be clear: the Hypothesis server and client are free software</p>
</blockquote>
<p>Doesn't server rely on nonfree Elasticsearch? Or am I missing something?</p>
<p>Anyway, what you suggest doesn't seem to be a bad idea. I have a personal issue, though - I already pretty much imagined how a repo should function and I know what I would need to program. With Hypothesis, I would need to first bite into the topic.</p>
<p>Annotations seem suitable for advertising facts about websites and getting feedback. But are they also for sharing the actual scripts?</p>
<p>Also, even if we're to use the existing infrastructure, we're not going to be doing that forever, are we? Once we reach the point we depart from Hypothesis service, are we going to have to deploy all their big software system with many dependencies and features we don't need? Or do you mean we would then finally develop our own server software, as I want to do right now?</p>