Project

General

Profile

Roadmap » History » Version 11

koszko, 02/24/2022 01:17 PM
add issue links

1 1 koszko
*Note: this is derived from the project plan prepared in relation to NLnet grant received by Haketilo/Hydrilla*
2
3
# Project plan Haketilo/Hydrilla
4
At the time of this writing the browser extension Haketilo and its repository, Hydrilla, have already
5
seen their initial 0.1 demo releases. Below is a plan for their further development that will use the
6
experience gained during initial prototyping to increase stability and supply functionalities that have
7
been missing or provisional only, as well as make the tools available to a wider audience, more
8
secure and easier to use.
9
10
{{toc}}
11
12 2 koszko
## 1. Haketilo and Hydrilla 1.0 pre-release (#103)
13 1 koszko
Some big code changes to land in Haketilo and Hydrilla 1.0 will be available in a pre-release. The
14
pre-release will be made before delivery of several other side artifacts planned for 1.0.
15
16
##### Milestones
17
* ~~project plan~~[^1_prplan]
18
* ~~tentative software bill of materials~~[^1_bom_haketilo][^1_bom_hydrilla]
19
* ~~use of registerContentScript API in Firefox Haketilo port~~ (#92)[^1_haketilo_buildable_again]
20
* ~~move to the new Hydrilla JSON API prototyped at [https://hydrillabugs.koszko.org/projects/hydrilla/wiki/Repository_API
21
](/projects/hydrilla/wiki/Repository_API)~~[^1_haketilo_buildable_again]
22
* ~~most WebExtension storage.local uses replaced with IndexedDB~~ (#98)[^1_haketilo_buildable_again]
23
* ~~Python implementation of Hydrilla~~[^1_python_hydrilla]
24
25
[^1_prplan]: [[Roadmap|this very document]]
26
[^1_bom_haketilo]: [[Haketilo Software Bill of Materials]]
27
[^1_bom_hydrilla]: [[hydrilla:Hydrilla Software Bill of Materials]]
28
[^1_haketilo_buildable_again]: [commit 4c6a2323d90e9321ec2b78e226167b3013ea69ab](/projects/haketilo/repository/haketilo/revisions/4c6a2323d90e9321ec2b78e226167b3013ea69ab)
29
[^1_python_hydrilla]: [Hydrilla](/projects/hydrilla/repository/hydrilla) and [Hydrilla builder](/projects/hydrilla/repository/hydrilla-builder) repositories
30
31 2 koszko
## 2. Haketilo and Hydrilla 1.0 release (#104)
32 1 koszko
This will be the first release since receiving the NLnet grant and the first non-demo
33
release, hence it includes many improvements in various fields.
34
35
##### Milestones
36
* ~~basic automated Haketilo tests using Selenium and a Firefox-based web browser~~ (#66)
37
* ~~JSON schemas describing Hydrilla on-disk resource format, Hydrilla HTTP API and other JSON interfaces in use~~ [^2_schemas_repo]
38 3 koszko
* validation of all external JSON data in Haketilo and Hydrilla using included JSON schemas (#105)
39 1 koszko
* sample Apache2 configuration file for use with Hydrilla (#55)
40
* detailed documentation for installation and running of Hydrilla (#55)
41
* manpage for Hydrilla (#55)
42
43
[^2_schemas_repo]: [JSON schemas](/projects/hydrilla/repository/hydrilla-json-schemas) repository
44
45
##### Estimated time
46 4 koszko
1.5 weeks
47
48 5 koszko
## 3. Distribution of Hydrilla and (when applicable) Haketilo in package managers (#106)
49 4 koszko
50
It is beneficial to have tools available in a format specific to various operating system distributions.
51
While the process of inclusion in official repositories is often a complex and lengthy one, preparing
52
the actual packages, as is the goal of this task, is a good first step to making that happen.
53
54
##### Milestones
55
56
* .deb packaging of Haketilo and Hydrilla
57
* Nix packaging of Hydrilla
58
* Pacman PKGBUILDs for Haketilo and Hydrilla
59
* Guix packaging of Haketilo and Hydrilla
60
* RPM packaging of Haketilo and Hydrilla
61
62
##### Estimated time
63
64
2 weeks
65
66 5 koszko
## 4. Development of Hydrilla website part (#35)
67 4 koszko
68
A project's website makes its first impression, and therefore deserves special care. In our case the
69
website will be part of our software Hydrilla.
70
71
##### Milestones
72
73
* planning a site structure
74
* designing a landing page
75
* cross-reference with Hydrilla to ensure uniformity of design and compatibility with the on-disk format
76
* crafting of text, graphics, and any other media
77
* assembly of website
78
79
##### Estimated time
80
81
2 weeks
82
83 5 koszko
## 5. Development of a user-controlled captcha client (#107)
84 4 koszko
85
Haketilo's goal is to give internet users control over their browsing. Replacing proprietary,
86
privacy-hostile client-side programs is part of that. A tool similar to the librecaptcha Python program
87
is needed, but in the form of a javascript library.
88
89
##### Milestones
90
91
* facility for Haketilo-supplied scripts to bypass CORS
92
* free/libre javascript library for solving reCAPTCHA challenges
93
* sample Haketilo resource making use of the library on a chosen website
94
95
##### Estimated time
96
97
3 weeks
98
99 5 koszko
## 6. Permissions system for Haketilo-supplied resources (#73)
100 4 koszko
101
Custom, user-supplied resources Haketilo may deploy on viewed pages might require looser
102
restrictions than those normally employed on pages. Or, they might allow for tighter security
103
mechanisms to be employed.
104
105
##### Milestones
106
107
* specification of a new revision of Hydrilla API and on-disk format with permissions support
108 8 koszko
* facility to limit domains for which a Haketilo-supplied script is allowed to bypass CORS
109
* facility to specify what custom Content Security Policy should be used on a given pages (#88)
110 4 koszko
111
##### Estimated time
112
113
2 weeks
114
115 5 koszko
## 7. Further means of user-controlled customization of sites (#108)
116 4 koszko
117
Besides the initial function of replacing sites' javascript it is also desired to facilitate supplying
118
additional data (e.g. images) and replacing other site components.
119
120
##### Milestones
121
122 6 koszko
* facility to make arbitrary bundled data files accessible to Haketilo-supplied scripts (#69)
123
* facility to replace the entire interface of a web page with user-supplied HTML (#70)
124 4 koszko
* facility to add user-supplied CSS to a web page
125
* facility to add user-supplied fonts to a web page
126
127
##### Estimated time
128
129
3 weeks
130
131 5 koszko
## 8. 50 sample site resources for Haketilo (#109)
132 4 koszko
133
To build the community its purpose depends on, Hydrilla must be clearly ready for use. This
134
requires a representative, well-stocked library of packages.
135
136
##### Milestones
137
138
* guide describing how to make and contribute custom site resources to Hydrilla
139
* at least 5 alternative site interfaces
140
* Javascript of at least 10 free/libre web tools (like Etherpad, Ethercalc) repackaged to be run in a user-controlled way from Haketilo
141
* at least 50 different custom site resources in total
142
143
##### Estimated time
144
145
2 weeks
146
147 5 koszko
## 9. Hektilo LibrePlanet presentation (#110)
148 4 koszko
149
LibrePlanet is a conference organized by the Free Software Foundation (FSF). It is "an opportunity
150
to meet and interact with other people with both a technical and non technical background" and to
151
share experience.
152
153
##### Milestones
154
155
* applied to LibrePlanet 2022
156
* prepared presentation about giving users back the control over web browsing
157
* made the presentation at LibrePlanet 2022 (if accepted there) or posted a video presentation on Haketilo website (as a fallback case)
158
159
## 10. Localization of Haketilo and Hydrilla
160
161
To truly empower to web users all over the world, Haketilo, Hydrilla, and all associated materials
162
must be able to support languages from across the world.
163
164
##### Milestones
165
166
* automatic content language negotiation on Hydrilla pages and the website
167
* language selection option on Hydrilla pages and the website
168 10 koszko
* internationalization of Haketilo (#51)
169 4 koszko
* language selection option in Haketilo
170
* Polish translation
171
172
##### Estimated time
173
174
3 weeks
175
176
## 11. Security vetting of Haketilo and Hydrilla
177
178
As NLNet-funded projects, Haketilo and Hydrilla have the privilege of a security review from
179
Radically Open Security. To make use of this opportunity, we will ensure any findings provided are
180
properly addressed.
181
182
##### Milestones
183
184
* action on any recommendations or other findings
185
* report of how each finding from the vetting was addressed, and why
186
* note of any key issues in the developer documentation, in order to avoid repetition in the future
187
188
##### Estimated time
189
190
2 weeks
191
192
## 12. Accessibility vetting of Haketilo and Hydrilla
193
194
To empower every web user, Haketilo and Hydrilla must support the interfaces they need.
195
196
##### Milestones
197
198
* action on any recommendations or other findings
199
* report of how each finding from the vetting was addressed, and why
200
* note of any key issues in the developer documentation, in order to avoid repetition in the future
201
* certified WCAG accessible
202
203
##### Estimated time
204
205
2 weeks
206
207
## 13. Manifest V3 Haketilo port
208
209
Although highly controversial, the Manifest V3 extension format seems unavoidable.
210
211
##### Milestones
212
213
* background page replaced with Service Workers
214
* blocking webRequest operations replaced with declarativeNetRequest
215
* Haketilo working under a Chromium-based browser as a Manifest V3 extension
216
217
##### Estimated time
218
219
4 weeks
220
221
## 14. Tighter testing of Haketilo
222
223
Testing in multiple browser environments is important to ensure stability of the extension.
224
225
##### Milestones
226
227
* automated tests under each supported extension platform with at least 1 Firefox-based and Chromium-based platform
228
* integration tests of communication between Haketilo and a Hydrilla instance
229
230
##### Estimated time
231
232
2 weeks
233
234
## 15. More thorough documentation of Haketilo and Hydrilla internals
235
236
With codebase refactored and stabilized, a worthy thing is to have it properly described for others
237
to hack on.
238
239
##### Milestones
240
241
* graphical diagram(s) describing execution contexts in Haketilo and the way scripts running in various context communicate
242
* graphical diagram(s) describing the algorithm for querying by Haketilo URL patterns
243
* comprehensive description of strategies employed and APIs used for replacing scripts and CSP in Haketilo
244
* graphical diagram describing how entities (resources, mappings, licenses) depend on each another
245
* docstring documentation of every Python function
246
* HTML documentation generated from Python source code
247
* JSDoc description of every Haketilo javascript function exported from file
248
* HTML documentation generated from javascript source code
249
250
##### Estimated time
251
252
2 weeks
253
254
## 16. Tooling for building of site resources
255
256
Simple scripts don't require building before distribution. Wasm modules and bigger libraries do. For
257
users to control the resources they use in Haketilo, there needs to be some well-defined way of
258
accessing the sources and repeating the build process.
259
260
##### Milestones
261
262
* specification of Haketilo source package format
263
* ability to specify other programs the build process depends on
264
* software to automatically build a Haketilo source package
265
266
##### Estimated time
267
268
2 weeks
269
270
## 17. Package signing in Haketilo and Hydrilla
271
272
Haketilo uses encrypted HTTPS connections to query Hydrilla API. However, to boost the security
273
and enable use of mirrors, we plan to also use PGP signatures on site resources served.
274
275
##### Milestones
276
277
* specification of a new revision of Hydrilla API and on-disk format with PGP signatures support
278
* tool for batch signing of site resources
279
* Hydrilla support for serving PGP signatures
280
* Haketilo support for downloading and verifying PGP signatures
281
* facility to manage trusted public keys within Haketilo
282
283
##### Estimated time
284
285
3 weeks
286
287
## 18. Support for custom meta-sites in Haketilo/Hydrilla
288
289
Allowing users to modify pages loaded by their browsers is our goal. Allowing them to aggregate
290
content from many sites on one page is a natural extension of it. Just as is allowing them to run
291
static web apps without having to trust some website serving them.
292
293
##### Milestones
294
295
* specification of a new revision of Hydrilla API and on-disk format with meta-sites support
296 9 koszko
* support for meta-sites in Hydrilla and Haketilo (#72)
297
298 4 koszko
299
##### Estimated time
300
301
3 weeks
302
303
## 19. Easier content management and editing within Haketilo (I)
304
305
Easy configuring and editing of site resource bundles is Haketilo's raison d'รชtre. To definitively
306
meet this expectation, any shortcomings must be identified and rethought.
307
308
##### Milestones
309
310
* testing with untrained users/consultation with "UX experts"
311
* identified annoying quirks/problems
312
* comparison with UIs of similar extensions
313
* designed alternatives to identified problems
314
* user interface mock
315
* a compiled plan for UI changes
316
317
##### Estimated time
318
319
2 weeks
320
321
## 20. Easier content management and editing within Haketilo (II)
322
323
The previously compiled plan and carefully-prepared user interface mocks will direct the
324
implementation efforts.
325
326
##### Milestones
327
328
* new Haketilo settings page interface implementation following the plan
329
* new Haketilo popup page implementation following the plan
330
* automated Haketilo GUI tests
331
332
##### Estimated time
333
334
2 weeks
335
336
## 21. REUSE specification compliance
337
338
License terms of software projects' files should be unambiguous and easy to analyze by humans
339
and computers alike. Compliance with the REUSE specification helps ensure that.
340
341
##### Milestones
342
343
* REUSE compliance in Hydrilla repository
344
* REUSE compliance in project website repository
345
* REUSE compliance in Haketilo repository
346
* REUSE compliance in custom site resources repository(ies)
347
348
##### Estimated time
349
350
1 week
351
352
## 22. Integrity constraints in Haketilo **(optional)**
353
354
One Haketilo custom site resource may depend on another, but initial versions of Haketilo did not
355
verify that dependencies are present. This and other sanity checks can be employed.
356
357
##### Milestones
358
359
* dependency checks when "installing" or upgrading a custom resource in Haketilo
360
* dependency checks when removing a custom resource from Haketilo
361
* facility for cascade removal
362
* validation of Haketilo URL patterns and other values typed in by the user
363
364
##### Estimated time
365
366
1 week
367
368
## 23. Sample meta-sites for Haketilo/Hydrilla **(optional)**
369
370
Running a static webapp like litewrite by visiting its website relies on the security of TLS and
371
network connectivity. Having it packaged as a separate browser extension requires giving it
372
excessive permissions. Running it from an HTML file is inconvenient.
373
374
##### Milestones
375
376
* at least 5 existing webapps packaged as meta-sites
377
* at least 5 meta-sites aggregating content from various client websites
378
379
##### Estimated time
380
381
3 weeks
382
383
## 24. Haketilo build system runnable from the browser **(optional)**
384
385
For portability of Haketilo's POSIX shell-based build system we avoided depending on Node.js,
386
NPM and similar tools. However, an even more portable alternative exists - to contain the build
387
system inside a standalone HTML page.
388
389
##### Milestones
390
391
* Javascript-based build system in an HTML page
392
* facility to run the javascript-based build system from the command line
393
394
##### Estimated time
395
396
2 weeks
397
398
## 25. User upload of content to Hydrilla website **(optional)**
399
400
To be able to easier gather and share custom site resources within the community, we need a
401
user-friendly platform.
402
403
##### Milestones
404
405
* registrations on a Hydrilla instance
406
* upload of custom site resources to a Hydrilla instance
407
* facility to easily and efficiently moderate the content uploaded by users
408
409
##### Estimated time
410
411
3 weeks
412
413
## 26. Further development of Hydrilla platform **(optional)**
414
415
Users should be able to share not only custom site resources but also their opinions about them.
416
417
##### Milestones
418
419
* support for user comments
420
* support for user ratings
421
* support for flagging site resources that are broken or have other issues
422
* development of comment quality control systems and policies
423
424
##### Estimated time
425
426
2 weeks
427
428
## 27. Facility for setting up Hydrilla repository mirrors **(optional)**
429
430
While allowing users to set up independent instances of Hydrilla gives them greater control over
431
site content they use, it does not by itself increase the robustness and maximum throughput of
432
Hydrilla platform. Enabling the use of mirrors does.
433
434
##### Milestones
435
436
* support for setting up and automatically synchronizing Hydrilla mirrors
437
* support for announcing available mirrors in Hydrilla
438
* support for fetching repository mirrors list in Haketilo
439
* support for distributing requests over multiple repository mirrors in Haketilo
440
* documentation
441
442
##### Estimated time
443
444
2 weeks
445
446
## 28. 150 sample site resources for Haketilo **(optional)**
447
448
To maintain community growth and participation, Hydrilla's collection must be visibly alive and
449
evolve with Haketilo's feature set.
450
451
##### Milestones
452
453
* at least 20 alternative site interfaces
454
* at least 20 existing webapps packaged as meta-sites
455
* at least 150 custom site resources in total
456
457
##### Estimated time
458
459
2 weeks
460
461
## 29. 200 sample site resources for Haketilo **(optional)**
462
463
To maintain community growth and participation, Hydrilla's collection must be visibly alive and
464
evolve with Haketilo's feature set.
465
466
##### Milestones
467
468
* at least 20 accessibility-improving site changes
469
* at least 10 meta-sites aggregating content from various client websites
470
* at least 200 custom site resources in total
471
472
##### Estimated time
473
474
2 weeks
475
476
## 30. Automated building of Haketilo source packages uploaded to Hydrilla **(optional)**
477
478
Requiring packagers to upload compiled code places an extra burden on them, and complicates
479
reproducibility. Hydrilla should be able to build from source packages.
480
481
##### Milestones
482
483
* Hydrilla automated resource builds feature
484
* security consultation of the feature
485
486
##### Estimated time
487
488
2 weeks
489
490
## 31. Self-documented Haketilo **(optional)**
491
492
Now matter how user-friendly the graphical interface is, an explanation of some of the concepts
493
might be needed. The next step, after having the documentation available on the project website,
494
is bundling it with the extension itself.
495
496
##### Milestones
497
498
* Haketilo popup self-documented inline
499
* Haketilo settings page self-documented inline
500
* documentation included as extension-bundled HTML pages
501
502
##### Estimated time
503
504
2 weeks
505
506
## 32. Displaying Hypothesis annotations for given site **(optional)**
507
508
Haketilo makes site resources for websites you visit available in only a few clicks. It would be
509
useful to have the same capacity for comments. The established, libre https://hypothes.is/ provides
510
a framework for this.
511
512
##### Milestones
513
514
* support for displaying current site's Hypothesis annotations in the popup
515
* support for adding adding Hypothesis annotations in Haketilo
516
517
##### Estimated time
518
519
2 weeks
520
521
## 33. Automatic generation of independent browser extensions from Haketilo site resources **(optional)**
522
523
Haketilo's rich feature set might also be an inconvenience. It may be overwhelming or irritating to
524
some users and has a higher risk of breaking with newer browser versions than a simple extension
525
would have. Thus, an option to install just a single Haketilo resource in the browser would be
526
useful.
527
528
##### Milestones
529
530
* automatic generation of Firefox WebExtensions from Haketilo site resources
531
* automatic generation of Chromium ManifestV3 WebExtensions from Haketilo site resources
532
533
##### Estimated time
534
535
2 weeks
536
537 11 koszko
## 34. Facility to automatically convert page's "native" scripts to a Haketilo resource **(optional)** (#6)
538 4 koszko
539
Haketilo gives users control over scripts being executed on a given web page. The scripts to be
540
used need to be defined in Haketilo as a resource. Doing this manually might be time-consuming
541
for a user who aims to use mostly the same javascript a website normally serves, but served from
542
within Haketilo.
543
544
##### Milestones
545
546
* automatic conversion of page's inline scripts in a Haketilo resource
547
* inclusion of page's external scripts in generated resource
548 11 koszko
* inclusion of page's intrinsic javascript events in generated resource (#7)
549 4 koszko
* displaying warnings when a site's javascript is known to use mechanisms that might stop
550
  such automatic package from working properly
551
552
##### Estimated time
553
554
3 weeks
555
556
## 35. Use of a standalone javascript engine to perform unit tests in Haketilo **(optional)**
557
558
A Selenium-driven web browser is currently used to test parts of Haketilo. Those tests that don't
559
rely on browser APIs could as well be run outside of browser which would save time during tests.
560
561
##### Milestones
562
563
* selected the javascript engine to use for testing
564
* facilitated writing Haketilo tests against the chosen engine
565
* applicable existing tests modified to be run without a web browser
566
567
##### Estimated time
568
569
2 weeks
570
571
## 36. Supplemental anti-bot measures in Hydrilla **(optional)**
572
573
Limiting the number of allowed registrations and content uploads is our planned basic way to
574
prevent Hydrilla instances from being harmed by automated requests. Another measures can be
575
added to further improve platform's resilience.
576
577
##### Milestones
578
579
* email-verified registrations
580
* selected an ethical, privacy-friendly captcha solution
581
* implementation of the chosen captcha solution
582
583
##### Estimated time
584
585
2 weeks
586
587
## 37. Support for external user authentication mechanisms in Hydrilla **(optional)**
588
589
It should be possible to run Hydrilla as part of a bigger web service. Users should be able to use
590
the same set of credentials for logging in in various parts of such service.
591
592
##### Milestones
593
594
* selected an authentication mechanism to support
595
* implementation of the feature
596
597
##### Estimated time
598
599
1 week
600
601
## 38. Support for building Hydrilla and Haketilo using Autotools **(optional)**
602
603
The specificity of Haketilo and Hydrilla means a complex build system like Autotools is not
604
necessary. It could, however, be added as optional to supplement their simple build mechanisms.
605
606
##### Milestones
607
608
* Hydrilla buildable with Autotools
609
* Hydrilla out-of-source builds possible
610
* Hydrilla tarball producible with a make rule
611
* Haketilo buildable with Autotools
612
* Haketilo out-of-source builds possible
613
* Haketilo tarball producible with a make rule
614
615
##### Estimated time
616
617
1 week
618
619
## 39. Evaluation of non-WebExtension platforms for the purpose of porting Haketilo **(optional)**
620
621
WebExtensions are really a convenient platform for developing software that empowers users. But
622
this platform is also tightly controlled by big organizations and has some serious limitations and
623
shortcomings.
624
625
##### Milestones
626
627
* evaluation of existing Webkit-based browsers
628
* evaluation of XUL extensions platform still used in some Firefox forks
629
* prepared evaluation report
630
631
##### Estimated time
632
633
1 week
634
635
## 40. Development of the first non-WebExtension Haketilo port **(optional)**
636
637
Users suffer a vendor lock-in with few mainstream web browsers. Lack of their favorite extensions
638
is what stops them from switching to more user-controlled alternatives. Haketilo should not
639
contribute to that problem.
640
641
##### Milestones
642
643
* selection of a target platform based on previous evaluation
644
* specification of tasks
645
* development roadmap
646
* prototype
647
* automated tests
648
* developer documentation
649
* user documentation
650
651
##### Estimated time
652
653
7.5 weeks