Project

General

Profile

Roadmap » History » Version 19

koszko, 03/25/2022 11:03 AM
consistent spaces

1 1 koszko
*Note: this is derived from the project plan prepared in relation to NLnet grant received by Haketilo/Hydrilla*
2
3
# Project plan Haketilo/Hydrilla
4
At the time of this writing the browser extension Haketilo and its repository, Hydrilla, have already
5
seen their initial 0.1 demo releases. Below is a plan for their further development that will use the
6
experience gained during initial prototyping to increase stability and supply functionalities that have
7
been missing or provisional only, as well as make the tools available to a wider audience, more
8
secure and easier to use.
9
10
{{toc}}
11
12 13 koszko
## 1. ~~Haketilo and Hydrilla 1.0 pre-release~~ (#103)
13 1 koszko
Some big code changes to land in Haketilo and Hydrilla 1.0 will be available in a pre-release. The
14
pre-release will be made before delivery of several other side artifacts planned for 1.0.
15
16
##### Milestones
17
* ~~project plan~~[^1_prplan]
18
* ~~tentative software bill of materials~~[^1_bom_haketilo][^1_bom_hydrilla]
19
* ~~use of registerContentScript API in Firefox Haketilo port~~ (#92)[^1_haketilo_buildable_again]
20
* ~~move to the new Hydrilla JSON API prototyped at [https://hydrillabugs.koszko.org/projects/hydrilla/wiki/Repository_API
21
](/projects/hydrilla/wiki/Repository_API)~~[^1_haketilo_buildable_again]
22
* ~~most WebExtension storage.local uses replaced with IndexedDB~~ (#98)[^1_haketilo_buildable_again]
23
* ~~Python implementation of Hydrilla~~[^1_python_hydrilla]
24
25
[^1_prplan]: [[Roadmap|this very document]]
26
[^1_bom_haketilo]: [[Haketilo Software Bill of Materials]]
27
[^1_bom_hydrilla]: [[hydrilla:Hydrilla Software Bill of Materials]]
28
[^1_haketilo_buildable_again]: [commit 4c6a2323d90e9321ec2b78e226167b3013ea69ab](/projects/haketilo/repository/haketilo/revisions/4c6a2323d90e9321ec2b78e226167b3013ea69ab)
29
[^1_python_hydrilla]: [Hydrilla](/projects/hydrilla/repository/hydrilla) and [Hydrilla builder](/projects/hydrilla/repository/hydrilla-builder) repositories
30
31 2 koszko
## 2. Haketilo and Hydrilla 1.0 release (#104)
32 1 koszko
This will be the first release since receiving the NLnet grant and the first non-demo
33
release, hence it includes many improvements in various fields.
34
35
##### Milestones
36
* ~~basic automated Haketilo tests using Selenium and a Firefox-based web browser~~ (#66)
37 18 koszko
* ~~JSON schemas describing Hydrilla on-disk resource format, Hydrilla HTTP API and other JSON interfaces in use~~[^2_schemas_repo]
38
* ~~validation of all external JSON data in Haketilo and Hydrilla using included JSON schemas~~ (#105)[^2_schemas_used_haketilo]
39 1 koszko
* sample Apache2 configuration file for use with Hydrilla (#55)
40
* detailed documentation for installation and running of Hydrilla (#55)
41 19 koszko
* manpage for Hydrilla (#55)[^2_manpages]
42 1 koszko
43
[^2_schemas_repo]: [JSON schemas](/projects/hydrilla/repository/hydrilla-json-schemas) repository
44 18 koszko
[^2_schemas_used_haketilo]: [commit 57ce414ca81682a71288018a4d9001604002ec23 ](/projects/haketilo/repository/haketilo/revisions/57ce414ca81682a71288018a4d9001604002ec23)
45 15 koszko
[^2_manpages]: [commit 1cb6aaae2055283d04aa0aa581e82addb8049ce4](/projects/hydrilla/repository/hydrilla/revisions/1cb6aaae2055283d04aa0aa581e82addb8049ce4) and [commit 363cbbb6a9fac49a377d8fa13ffede1483feabd5](/projects/hydrilla/repository/hydrilla-builder/revisions/363cbbb6a9fac49a377d8fa13ffede1483feabd5)
46 1 koszko
47
##### Estimated time
48 4 koszko
1.5 weeks
49
50 5 koszko
## 3. Distribution of Hydrilla and (when applicable) Haketilo in package managers (#106)
51 4 koszko
52
It is beneficial to have tools available in a format specific to various operating system distributions.
53
While the process of inclusion in official repositories is often a complex and lengthy one, preparing
54
the actual packages, as is the goal of this task, is a good first step to making that happen.
55
56
##### Milestones
57
58
* .deb packaging of Haketilo and Hydrilla
59
* Nix packaging of Hydrilla
60
* Pacman PKGBUILDs for Haketilo and Hydrilla
61
* Guix packaging of Haketilo and Hydrilla
62
* RPM packaging of Haketilo and Hydrilla
63
64
##### Estimated time
65
66
2 weeks
67
68 5 koszko
## 4. Development of Hydrilla website part (#35)
69 4 koszko
70
A project's website makes its first impression, and therefore deserves special care. In our case the
71
website will be part of our software Hydrilla.
72
73
##### Milestones
74
75
* planning a site structure
76
* designing a landing page
77
* cross-reference with Hydrilla to ensure uniformity of design and compatibility with the on-disk format
78
* crafting of text, graphics, and any other media
79
* assembly of website
80
81
##### Estimated time
82
83
2 weeks
84
85 5 koszko
## 5. Development of a user-controlled captcha client (#107)
86 4 koszko
87
Haketilo's goal is to give internet users control over their browsing. Replacing proprietary,
88
privacy-hostile client-side programs is part of that. A tool similar to the librecaptcha Python program
89 16 koszko
is needed, but in the form of a JavaScript library.
90 4 koszko
91
##### Milestones
92
93
* facility for Haketilo-supplied scripts to bypass CORS
94 16 koszko
* free/libre JavaScript library for solving reCAPTCHA challenges
95 4 koszko
* sample Haketilo resource making use of the library on a chosen website
96
97
##### Estimated time
98
99
3 weeks
100
101 5 koszko
## 6. Permissions system for Haketilo-supplied resources (#73)
102 4 koszko
103
Custom, user-supplied resources Haketilo may deploy on viewed pages might require looser
104
restrictions than those normally employed on pages. Or, they might allow for tighter security
105
mechanisms to be employed.
106
107
##### Milestones
108
109
* specification of a new revision of Hydrilla API and on-disk format with permissions support
110 8 koszko
* facility to limit domains for which a Haketilo-supplied script is allowed to bypass CORS
111
* facility to specify what custom Content Security Policy should be used on a given pages (#88)
112 4 koszko
113
##### Estimated time
114
115
2 weeks
116
117 5 koszko
## 7. Further means of user-controlled customization of sites (#108)
118 4 koszko
119 16 koszko
Besides the initial function of replacing sites' JavaScript it is also desired to facilitate supplying
120 4 koszko
additional data (e.g. images) and replacing other site components.
121
122
##### Milestones
123
124 6 koszko
* facility to make arbitrary bundled data files accessible to Haketilo-supplied scripts (#69)
125
* facility to replace the entire interface of a web page with user-supplied HTML (#70)
126 4 koszko
* facility to add user-supplied CSS to a web page
127
* facility to add user-supplied fonts to a web page
128
129
##### Estimated time
130
131
3 weeks
132
133 5 koszko
## 8. 50 sample site resources for Haketilo (#109)
134 4 koszko
135
To build the community its purpose depends on, Hydrilla must be clearly ready for use. This
136
requires a representative, well-stocked library of packages.
137
138
##### Milestones
139
140
* guide describing how to make and contribute custom site resources to Hydrilla
141
* at least 5 alternative site interfaces
142 16 koszko
* JavaScript of at least 10 free/libre web tools (like Etherpad, Ethercalc) repackaged to be run in a user-controlled way from Haketilo
143 4 koszko
* at least 50 different custom site resources in total
144
145
##### Estimated time
146
147
2 weeks
148
149 17 koszko
## 9. ~~Haketilo LibrePlanet presentation~~ (#110)
150 4 koszko
151
LibrePlanet is a conference organized by the Free Software Foundation (FSF). It is "an opportunity
152
to meet and interact with other people with both a technical and non technical background" and to
153
share experience.
154
155
##### Milestones
156 1 koszko
157 17 koszko
* ~~applied to LibrePlanet 2022~~
158
* ~~prepared presentation about giving users back the control over web browsing~~
159
* ~~made the presentation at LibrePlanet 2022 (if accepted there) or posted a video presentation on Haketilo website (as a fallback case)~~[^lp2022]
160
161
[^lp2022]: https://libreplanet.org/2022/speakers/#5790
162 4 koszko
163
## 10. Localization of Haketilo and Hydrilla
164
165
To truly empower to web users all over the world, Haketilo, Hydrilla, and all associated materials
166
must be able to support languages from across the world.
167
168
##### Milestones
169
170
* automatic content language negotiation on Hydrilla pages and the website
171
* language selection option on Hydrilla pages and the website
172 10 koszko
* internationalization of Haketilo (#51)
173 4 koszko
* language selection option in Haketilo
174
* Polish translation
175
176
##### Estimated time
177
178
3 weeks
179
180
## 11. Security vetting of Haketilo and Hydrilla
181
182
As NLNet-funded projects, Haketilo and Hydrilla have the privilege of a security review from
183
Radically Open Security. To make use of this opportunity, we will ensure any findings provided are
184
properly addressed.
185
186
##### Milestones
187
188
* action on any recommendations or other findings
189
* report of how each finding from the vetting was addressed, and why
190
* note of any key issues in the developer documentation, in order to avoid repetition in the future
191
192
##### Estimated time
193
194
2 weeks
195
196
## 12. Accessibility vetting of Haketilo and Hydrilla
197
198
To empower every web user, Haketilo and Hydrilla must support the interfaces they need.
199
200
##### Milestones
201
202
* action on any recommendations or other findings
203
* report of how each finding from the vetting was addressed, and why
204
* note of any key issues in the developer documentation, in order to avoid repetition in the future
205
* certified WCAG accessible
206
207
##### Estimated time
208
209
2 weeks
210
211
## 13. Manifest V3 Haketilo port
212
213
Although highly controversial, the Manifest V3 extension format seems unavoidable.
214
215
##### Milestones
216
217
* background page replaced with Service Workers
218
* blocking webRequest operations replaced with declarativeNetRequest
219
* Haketilo working under a Chromium-based browser as a Manifest V3 extension
220
221
##### Estimated time
222
223
4 weeks
224
225
## 14. Tighter testing of Haketilo
226
227
Testing in multiple browser environments is important to ensure stability of the extension.
228
229
##### Milestones
230
231
* automated tests under each supported extension platform with at least 1 Firefox-based and Chromium-based platform
232
* integration tests of communication between Haketilo and a Hydrilla instance
233
234
##### Estimated time
235
236
2 weeks
237
238
## 15. More thorough documentation of Haketilo and Hydrilla internals
239
240
With codebase refactored and stabilized, a worthy thing is to have it properly described for others
241
to hack on.
242
243
##### Milestones
244
245
* graphical diagram(s) describing execution contexts in Haketilo and the way scripts running in various context communicate
246
* graphical diagram(s) describing the algorithm for querying by Haketilo URL patterns
247
* comprehensive description of strategies employed and APIs used for replacing scripts and CSP in Haketilo
248
* graphical diagram describing how entities (resources, mappings, licenses) depend on each another
249
* docstring documentation of every Python function
250
* HTML documentation generated from Python source code
251 16 koszko
* JSDoc description of every Haketilo JavaScript function exported from file
252
* HTML documentation generated from JavaScript source code
253 4 koszko
254
##### Estimated time
255
256
2 weeks
257
258
## 16. Tooling for building of site resources
259
260
Simple scripts don't require building before distribution. Wasm modules and bigger libraries do. For
261
users to control the resources they use in Haketilo, there needs to be some well-defined way of
262
accessing the sources and repeating the build process.
263
264
##### Milestones
265
266
* specification of Haketilo source package format
267
* ability to specify other programs the build process depends on
268
* software to automatically build a Haketilo source package
269
270
##### Estimated time
271
272
2 weeks
273
274
## 17. Package signing in Haketilo and Hydrilla
275
276
Haketilo uses encrypted HTTPS connections to query Hydrilla API. However, to boost the security
277
and enable use of mirrors, we plan to also use PGP signatures on site resources served.
278
279
##### Milestones
280
281
* specification of a new revision of Hydrilla API and on-disk format with PGP signatures support
282
* tool for batch signing of site resources
283
* Hydrilla support for serving PGP signatures
284
* Haketilo support for downloading and verifying PGP signatures
285
* facility to manage trusted public keys within Haketilo
286
287
##### Estimated time
288
289
3 weeks
290
291
## 18. Support for custom meta-sites in Haketilo/Hydrilla
292
293
Allowing users to modify pages loaded by their browsers is our goal. Allowing them to aggregate
294
content from many sites on one page is a natural extension of it. Just as is allowing them to run
295
static web apps without having to trust some website serving them.
296
297
##### Milestones
298
299
* specification of a new revision of Hydrilla API and on-disk format with meta-sites support
300 9 koszko
* support for meta-sites in Hydrilla and Haketilo (#72)
301
302 4 koszko
303
##### Estimated time
304
305
3 weeks
306
307
## 19. Easier content management and editing within Haketilo (I)
308
309
Easy configuring and editing of site resource bundles is Haketilo's raison d'รชtre. To definitively
310
meet this expectation, any shortcomings must be identified and rethought.
311
312
##### Milestones
313
314
* testing with untrained users/consultation with "UX experts"
315
* identified annoying quirks/problems
316
* comparison with UIs of similar extensions
317
* designed alternatives to identified problems
318
* user interface mock
319
* a compiled plan for UI changes
320
321
##### Estimated time
322
323
2 weeks
324
325
## 20. Easier content management and editing within Haketilo (II)
326
327
The previously compiled plan and carefully-prepared user interface mocks will direct the
328
implementation efforts.
329
330
##### Milestones
331
332
* new Haketilo settings page interface implementation following the plan
333
* new Haketilo popup page implementation following the plan
334
* automated Haketilo GUI tests
335
336
##### Estimated time
337
338
2 weeks
339
340
## 21. REUSE specification compliance
341
342
License terms of software projects' files should be unambiguous and easy to analyze by humans
343
and computers alike. Compliance with the REUSE specification helps ensure that.
344
345
##### Milestones
346
347
* REUSE compliance in Hydrilla repository
348
* REUSE compliance in project website repository
349
* REUSE compliance in Haketilo repository
350
* REUSE compliance in custom site resources repository(ies)
351
352
##### Estimated time
353
354
1 week
355
356
## 22. Integrity constraints in Haketilo **(optional)**
357
358
One Haketilo custom site resource may depend on another, but initial versions of Haketilo did not
359
verify that dependencies are present. This and other sanity checks can be employed.
360
361
##### Milestones
362
363
* dependency checks when "installing" or upgrading a custom resource in Haketilo
364
* dependency checks when removing a custom resource from Haketilo
365
* facility for cascade removal
366
* validation of Haketilo URL patterns and other values typed in by the user
367
368
##### Estimated time
369
370
1 week
371
372
## 23. Sample meta-sites for Haketilo/Hydrilla **(optional)**
373
374
Running a static webapp like litewrite by visiting its website relies on the security of TLS and
375
network connectivity. Having it packaged as a separate browser extension requires giving it
376
excessive permissions. Running it from an HTML file is inconvenient.
377
378
##### Milestones
379
380
* at least 5 existing webapps packaged as meta-sites
381
* at least 5 meta-sites aggregating content from various client websites
382
383
##### Estimated time
384
385
3 weeks
386
387
## 24. Haketilo build system runnable from the browser **(optional)**
388
389
For portability of Haketilo's POSIX shell-based build system we avoided depending on Node.js,
390
NPM and similar tools. However, an even more portable alternative exists - to contain the build
391
system inside a standalone HTML page.
392
393
##### Milestones
394
395 16 koszko
* JavaScript-based build system in an HTML page (#47)
396
* facility to run the JavaScript-based build system from the command line
397 4 koszko
398
##### Estimated time
399
400
2 weeks
401
402
## 25. User upload of content to Hydrilla website **(optional)**
403
404
To be able to easier gather and share custom site resources within the community, we need a
405
user-friendly platform.
406
407
##### Milestones
408
409
* registrations on a Hydrilla instance
410
* upload of custom site resources to a Hydrilla instance
411
* facility to easily and efficiently moderate the content uploaded by users
412
413
##### Estimated time
414
415
3 weeks
416
417
## 26. Further development of Hydrilla platform **(optional)**
418
419
Users should be able to share not only custom site resources but also their opinions about them.
420
421
##### Milestones
422
423
* support for user comments
424
* support for user ratings
425
* support for flagging site resources that are broken or have other issues
426
* development of comment quality control systems and policies
427
428
##### Estimated time
429
430
2 weeks
431
432
## 27. Facility for setting up Hydrilla repository mirrors **(optional)**
433
434
While allowing users to set up independent instances of Hydrilla gives them greater control over
435
site content they use, it does not by itself increase the robustness and maximum throughput of
436
Hydrilla platform. Enabling the use of mirrors does.
437
438
##### Milestones
439
440
* support for setting up and automatically synchronizing Hydrilla mirrors
441
* support for announcing available mirrors in Hydrilla
442
* support for fetching repository mirrors list in Haketilo
443
* support for distributing requests over multiple repository mirrors in Haketilo
444
* documentation
445
446
##### Estimated time
447
448
2 weeks
449
450
## 28. 150 sample site resources for Haketilo **(optional)**
451
452
To maintain community growth and participation, Hydrilla's collection must be visibly alive and
453
evolve with Haketilo's feature set.
454
455
##### Milestones
456
457
* at least 20 alternative site interfaces
458
* at least 20 existing webapps packaged as meta-sites
459
* at least 150 custom site resources in total
460
461
##### Estimated time
462
463
2 weeks
464
465
## 29. 200 sample site resources for Haketilo **(optional)**
466
467
To maintain community growth and participation, Hydrilla's collection must be visibly alive and
468
evolve with Haketilo's feature set.
469
470
##### Milestones
471
472
* at least 20 accessibility-improving site changes
473
* at least 10 meta-sites aggregating content from various client websites
474
* at least 200 custom site resources in total
475
476
##### Estimated time
477
478
2 weeks
479
480
## 30. Automated building of Haketilo source packages uploaded to Hydrilla **(optional)**
481
482
Requiring packagers to upload compiled code places an extra burden on them, and complicates
483
reproducibility. Hydrilla should be able to build from source packages.
484
485
##### Milestones
486
487
* Hydrilla automated resource builds feature
488
* security consultation of the feature
489
490
##### Estimated time
491
492
2 weeks
493
494
## 31. Self-documented Haketilo **(optional)**
495
496
Now matter how user-friendly the graphical interface is, an explanation of some of the concepts
497
might be needed. The next step, after having the documentation available on the project website,
498
is bundling it with the extension itself.
499
500
##### Milestones
501
502
* Haketilo popup self-documented inline
503
* Haketilo settings page self-documented inline
504
* documentation included as extension-bundled HTML pages
505
506
##### Estimated time
507
508
2 weeks
509
510
## 32. Displaying Hypothesis annotations for given site **(optional)**
511
512
Haketilo makes site resources for websites you visit available in only a few clicks. It would be
513
useful to have the same capacity for comments. The established, libre https://hypothes.is/ provides
514
a framework for this.
515
516
##### Milestones
517
518
* support for displaying current site's Hypothesis annotations in the popup
519
* support for adding adding Hypothesis annotations in Haketilo
520
521
##### Estimated time
522
523
2 weeks
524
525
## 33. Automatic generation of independent browser extensions from Haketilo site resources **(optional)**
526
527
Haketilo's rich feature set might also be an inconvenience. It may be overwhelming or irritating to
528
some users and has a higher risk of breaking with newer browser versions than a simple extension
529
would have. Thus, an option to install just a single Haketilo resource in the browser would be
530
useful.
531
532
##### Milestones
533
534
* automatic generation of Firefox WebExtensions from Haketilo site resources
535
* automatic generation of Chromium ManifestV3 WebExtensions from Haketilo site resources
536
537
##### Estimated time
538
539
2 weeks
540
541 11 koszko
## 34. Facility to automatically convert page's "native" scripts to a Haketilo resource **(optional)** (#6)
542 4 koszko
543
Haketilo gives users control over scripts being executed on a given web page. The scripts to be
544
used need to be defined in Haketilo as a resource. Doing this manually might be time-consuming
545 16 koszko
for a user who aims to use mostly the same JavaScript a website normally serves, but served from
546 4 koszko
within Haketilo.
547
548
##### Milestones
549
550
* automatic conversion of page's inline scripts in a Haketilo resource
551
* inclusion of page's external scripts in generated resource
552 16 koszko
* inclusion of page's intrinsic JavaScript events in generated resource (#7)
553
* displaying warnings when a site's JavaScript is known to use mechanisms that might stop
554 4 koszko
  such automatic package from working properly
555
556
##### Estimated time
557
558
3 weeks
559
560 16 koszko
## 35. Use of a standalone JavaScript engine to perform unit tests in Haketilo **(optional)**
561 4 koszko
562
A Selenium-driven web browser is currently used to test parts of Haketilo. Those tests that don't
563
rely on browser APIs could as well be run outside of browser which would save time during tests.
564
565
##### Milestones
566
567 16 koszko
* selected the JavaScript engine to use for testing
568 4 koszko
* facilitated writing Haketilo tests against the chosen engine
569
* applicable existing tests modified to be run without a web browser
570
571
##### Estimated time
572
573
2 weeks
574
575
## 36. Supplemental anti-bot measures in Hydrilla **(optional)**
576
577
Limiting the number of allowed registrations and content uploads is our planned basic way to
578
prevent Hydrilla instances from being harmed by automated requests. Another measures can be
579
added to further improve platform's resilience.
580
581
##### Milestones
582
583
* email-verified registrations
584
* selected an ethical, privacy-friendly captcha solution
585
* implementation of the chosen captcha solution
586
587
##### Estimated time
588
589
2 weeks
590
591
## 37. Support for external user authentication mechanisms in Hydrilla **(optional)**
592
593
It should be possible to run Hydrilla as part of a bigger web service. Users should be able to use
594
the same set of credentials for logging in in various parts of such service.
595
596
##### Milestones
597
598
* selected an authentication mechanism to support
599
* implementation of the feature
600
601
##### Estimated time
602
603
1 week
604
605
## 38. Support for building Hydrilla and Haketilo using Autotools **(optional)**
606
607
The specificity of Haketilo and Hydrilla means a complex build system like Autotools is not
608
necessary. It could, however, be added as optional to supplement their simple build mechanisms.
609
610
##### Milestones
611
612
* Hydrilla buildable with Autotools
613
* Hydrilla out-of-source builds possible
614
* Hydrilla tarball producible with a make rule
615
* Haketilo buildable with Autotools
616
* Haketilo out-of-source builds possible
617
* Haketilo tarball producible with a make rule
618
619
##### Estimated time
620
621
1 week
622
623
## 39. Evaluation of non-WebExtension platforms for the purpose of porting Haketilo **(optional)**
624
625
WebExtensions are really a convenient platform for developing software that empowers users. But
626
this platform is also tightly controlled by big organizations and has some serious limitations and
627
shortcomings.
628
629
##### Milestones
630
631
* evaluation of existing Webkit-based browsers
632
* evaluation of XUL extensions platform still used in some Firefox forks
633
* prepared evaluation report
634
635
##### Estimated time
636
637
1 week
638
639
## 40. Development of the first non-WebExtension Haketilo port **(optional)**
640
641
Users suffer a vendor lock-in with few mainstream web browsers. Lack of their favorite extensions
642
is what stops them from switching to more user-controlled alternatives. Haketilo should not
643
contribute to that problem.
644
645
##### Milestones
646
647
* selection of a target platform based on previous evaluation
648
* specification of tasks
649
* development roadmap
650
* prototype
651
* automated tests
652
* developer documentation
653
* user documentation
654
655
##### Estimated time
656
657
7.5 weeks