Project

General

Profile

Roadmap » History » Version 8

koszko, 02/24/2022 12:53 PM
fix issue link

1 1 koszko
*Note: this is derived from the project plan prepared in relation to NLnet grant received by Haketilo/Hydrilla*
2
3
# Project plan Haketilo/Hydrilla
4
At the time of this writing the browser extension Haketilo and its repository, Hydrilla, have already
5
seen their initial 0.1 demo releases. Below is a plan for their further development that will use the
6
experience gained during initial prototyping to increase stability and supply functionalities that have
7
been missing or provisional only, as well as make the tools available to a wider audience, more
8
secure and easier to use.
9
10
{{toc}}
11
12 2 koszko
## 1. Haketilo and Hydrilla 1.0 pre-release (#103)
13 1 koszko
Some big code changes to land in Haketilo and Hydrilla 1.0 will be available in a pre-release. The
14
pre-release will be made before delivery of several other side artifacts planned for 1.0.
15
16
##### Milestones
17
* ~~project plan~~[^1_prplan]
18
* ~~tentative software bill of materials~~[^1_bom_haketilo][^1_bom_hydrilla]
19
* ~~use of registerContentScript API in Firefox Haketilo port~~ (#92)[^1_haketilo_buildable_again]
20
* ~~move to the new Hydrilla JSON API prototyped at [https://hydrillabugs.koszko.org/projects/hydrilla/wiki/Repository_API
21
](/projects/hydrilla/wiki/Repository_API)~~[^1_haketilo_buildable_again]
22
* ~~most WebExtension storage.local uses replaced with IndexedDB~~ (#98)[^1_haketilo_buildable_again]
23
* ~~Python implementation of Hydrilla~~[^1_python_hydrilla]
24
25
[^1_prplan]: [[Roadmap|this very document]]
26
[^1_bom_haketilo]: [[Haketilo Software Bill of Materials]]
27
[^1_bom_hydrilla]: [[hydrilla:Hydrilla Software Bill of Materials]]
28
[^1_haketilo_buildable_again]: [commit 4c6a2323d90e9321ec2b78e226167b3013ea69ab](/projects/haketilo/repository/haketilo/revisions/4c6a2323d90e9321ec2b78e226167b3013ea69ab)
29
[^1_python_hydrilla]: [Hydrilla](/projects/hydrilla/repository/hydrilla) and [Hydrilla builder](/projects/hydrilla/repository/hydrilla-builder) repositories
30
31 2 koszko
## 2. Haketilo and Hydrilla 1.0 release (#104)
32 1 koszko
This will be the first release since receiving the NLnet grant and the first non-demo
33
release, hence it includes many improvements in various fields.
34
35
##### Milestones
36
* ~~basic automated Haketilo tests using Selenium and a Firefox-based web browser~~ (#66)
37
* ~~JSON schemas describing Hydrilla on-disk resource format, Hydrilla HTTP API and other JSON interfaces in use~~ [^2_schemas_repo]
38 3 koszko
* validation of all external JSON data in Haketilo and Hydrilla using included JSON schemas (#105)
39 1 koszko
* sample Apache2 configuration file for use with Hydrilla (#55)
40
* detailed documentation for installation and running of Hydrilla (#55)
41
* manpage for Hydrilla (#55)
42
43
[^2_schemas_repo]: [JSON schemas](/projects/hydrilla/repository/hydrilla-json-schemas) repository
44
45
##### Estimated time
46 4 koszko
1.5 weeks
47
48 5 koszko
## 3. Distribution of Hydrilla and (when applicable) Haketilo in package managers (#106)
49 4 koszko
50
It is beneficial to have tools available in a format specific to various operating system distributions.
51
While the process of inclusion in official repositories is often a complex and lengthy one, preparing
52
the actual packages, as is the goal of this task, is a good first step to making that happen.
53
54
##### Milestones
55
56
* .deb packaging of Haketilo and Hydrilla
57
* Nix packaging of Hydrilla
58
* Pacman PKGBUILDs for Haketilo and Hydrilla
59
* Guix packaging of Haketilo and Hydrilla
60
* RPM packaging of Haketilo and Hydrilla
61
62
##### Estimated time
63
64
2 weeks
65
66 5 koszko
## 4. Development of Hydrilla website part (#35)
67 4 koszko
68
A project's website makes its first impression, and therefore deserves special care. In our case the
69
website will be part of our software Hydrilla.
70
71
##### Milestones
72
73
* planning a site structure
74
* designing a landing page
75
* cross-reference with Hydrilla to ensure uniformity of design and compatibility with the on-disk format
76
* crafting of text, graphics, and any other media
77
* assembly of website
78
79
##### Estimated time
80
81
2 weeks
82
83 5 koszko
## 5. Development of a user-controlled captcha client (#107)
84 4 koszko
85
Haketilo's goal is to give internet users control over their browsing. Replacing proprietary,
86
privacy-hostile client-side programs is part of that. A tool similar to the librecaptcha Python program
87
is needed, but in the form of a javascript library.
88
89
##### Milestones
90
91
* facility for Haketilo-supplied scripts to bypass CORS
92
* free/libre javascript library for solving reCAPTCHA challenges
93
* sample Haketilo resource making use of the library on a chosen website
94
95
##### Estimated time
96
97
3 weeks
98
99 5 koszko
## 6. Permissions system for Haketilo-supplied resources (#73)
100 4 koszko
101
Custom, user-supplied resources Haketilo may deploy on viewed pages might require looser
102
restrictions than those normally employed on pages. Or, they might allow for tighter security
103
mechanisms to be employed.
104
105
##### Milestones
106
107
* specification of a new revision of Hydrilla API and on-disk format with permissions support
108 8 koszko
* facility to limit domains for which a Haketilo-supplied script is allowed to bypass CORS
109
* facility to specify what custom Content Security Policy should be used on a given pages (#88)
110 4 koszko
111
##### Estimated time
112
113
2 weeks
114
115 5 koszko
## 7. Further means of user-controlled customization of sites (#108)
116 4 koszko
117
Besides the initial function of replacing sites' javascript it is also desired to facilitate supplying
118
additional data (e.g. images) and replacing other site components.
119
120
##### Milestones
121
122 6 koszko
* facility to make arbitrary bundled data files accessible to Haketilo-supplied scripts (#69)
123
* facility to replace the entire interface of a web page with user-supplied HTML (#70)
124 4 koszko
* facility to add user-supplied CSS to a web page
125
* facility to add user-supplied fonts to a web page
126
127
##### Estimated time
128
129
3 weeks
130
131 5 koszko
## 8. 50 sample site resources for Haketilo (#109)
132 4 koszko
133
To build the community its purpose depends on, Hydrilla must be clearly ready for use. This
134
requires a representative, well-stocked library of packages.
135
136
##### Milestones
137
138
* guide describing how to make and contribute custom site resources to Hydrilla
139
* at least 5 alternative site interfaces
140
* Javascript of at least 10 free/libre web tools (like Etherpad, Ethercalc) repackaged to be run in a user-controlled way from Haketilo
141
* at least 50 different custom site resources in total
142
143
##### Estimated time
144
145
2 weeks
146
147 5 koszko
## 9. Hektilo LibrePlanet presentation (#110)
148 4 koszko
149
LibrePlanet is a conference organized by the Free Software Foundation (FSF). It is "an opportunity
150
to meet and interact with other people with both a technical and non technical background" and to
151
share experience.
152
153
##### Milestones
154
155
* applied to LibrePlanet 2022
156
* prepared presentation about giving users back the control over web browsing
157
* made the presentation at LibrePlanet 2022 (if accepted there) or posted a video presentation on Haketilo website (as a fallback case)
158
159
## 10. Localization of Haketilo and Hydrilla
160
161
To truly empower to web users all over the world, Haketilo, Hydrilla, and all associated materials
162
must be able to support languages from across the world.
163
164
##### Milestones
165
166
* automatic content language negotiation on Hydrilla pages and the website
167
* language selection option on Hydrilla pages and the website
168
* internationalization of Haketilo
169
* language selection option in Haketilo
170
* Polish translation
171
172
##### Estimated time
173
174
3 weeks
175
176
## 11. Security vetting of Haketilo and Hydrilla
177
178
As NLNet-funded projects, Haketilo and Hydrilla have the privilege of a security review from
179
Radically Open Security. To make use of this opportunity, we will ensure any findings provided are
180
properly addressed.
181
182
##### Milestones
183
184
* action on any recommendations or other findings
185
* report of how each finding from the vetting was addressed, and why
186
* note of any key issues in the developer documentation, in order to avoid repetition in the future
187
188
##### Estimated time
189
190
2 weeks
191
192
## 12. Accessibility vetting of Haketilo and Hydrilla
193
194
To empower every web user, Haketilo and Hydrilla must support the interfaces they need.
195
196
##### Milestones
197
198
* action on any recommendations or other findings
199
* report of how each finding from the vetting was addressed, and why
200
* note of any key issues in the developer documentation, in order to avoid repetition in the future
201
* certified WCAG accessible
202
203
##### Estimated time
204
205
2 weeks
206
207
## 13. Manifest V3 Haketilo port
208
209
Although highly controversial, the Manifest V3 extension format seems unavoidable.
210
211
##### Milestones
212
213
* background page replaced with Service Workers
214
* blocking webRequest operations replaced with declarativeNetRequest
215
* Haketilo working under a Chromium-based browser as a Manifest V3 extension
216
217
##### Estimated time
218
219
4 weeks
220
221
## 14. Tighter testing of Haketilo
222
223
Testing in multiple browser environments is important to ensure stability of the extension.
224
225
##### Milestones
226
227
* automated tests under each supported extension platform with at least 1 Firefox-based and Chromium-based platform
228
* integration tests of communication between Haketilo and a Hydrilla instance
229
230
##### Estimated time
231
232
2 weeks
233
234
## 15. More thorough documentation of Haketilo and Hydrilla internals
235
236
With codebase refactored and stabilized, a worthy thing is to have it properly described for others
237
to hack on.
238
239
##### Milestones
240
241
* graphical diagram(s) describing execution contexts in Haketilo and the way scripts running in various context communicate
242
* graphical diagram(s) describing the algorithm for querying by Haketilo URL patterns
243
* comprehensive description of strategies employed and APIs used for replacing scripts and CSP in Haketilo
244
* graphical diagram describing how entities (resources, mappings, licenses) depend on each another
245
* docstring documentation of every Python function
246
* HTML documentation generated from Python source code
247
* JSDoc description of every Haketilo javascript function exported from file
248
* HTML documentation generated from javascript source code
249
250
##### Estimated time
251
252
2 weeks
253
254
## 16. Tooling for building of site resources
255
256
Simple scripts don't require building before distribution. Wasm modules and bigger libraries do. For
257
users to control the resources they use in Haketilo, there needs to be some well-defined way of
258
accessing the sources and repeating the build process.
259
260
##### Milestones
261
262
* specification of Haketilo source package format
263
* ability to specify other programs the build process depends on
264
* software to automatically build a Haketilo source package
265
266
##### Estimated time
267
268
2 weeks
269
270
## 17. Package signing in Haketilo and Hydrilla
271
272
Haketilo uses encrypted HTTPS connections to query Hydrilla API. However, to boost the security
273
and enable use of mirrors, we plan to also use PGP signatures on site resources served.
274
275
##### Milestones
276
277
* specification of a new revision of Hydrilla API and on-disk format with PGP signatures support
278
* tool for batch signing of site resources
279
* Hydrilla support for serving PGP signatures
280
* Haketilo support for downloading and verifying PGP signatures
281
* facility to manage trusted public keys within Haketilo
282
283
##### Estimated time
284
285
3 weeks
286
287
## 18. Support for custom meta-sites in Haketilo/Hydrilla
288
289
Allowing users to modify pages loaded by their browsers is our goal. Allowing them to aggregate
290
content from many sites on one page is a natural extension of it. Just as is allowing them to run
291
static web apps without having to trust some website serving them.
292
293
##### Milestones
294
295
* specification of a new revision of Hydrilla API and on-disk format with meta-sites support
296
* support for meta-sites in Hydrilla and Haketilo
297
298
##### Estimated time
299
300
3 weeks
301
302
## 19. Easier content management and editing within Haketilo (I)
303
304
Easy configuring and editing of site resource bundles is Haketilo's raison d'รชtre. To definitively
305
meet this expectation, any shortcomings must be identified and rethought.
306
307
##### Milestones
308
309
* testing with untrained users/consultation with "UX experts"
310
* identified annoying quirks/problems
311
* comparison with UIs of similar extensions
312
* designed alternatives to identified problems
313
* user interface mock
314
* a compiled plan for UI changes
315
316
##### Estimated time
317
318
2 weeks
319
320
## 20. Easier content management and editing within Haketilo (II)
321
322
The previously compiled plan and carefully-prepared user interface mocks will direct the
323
implementation efforts.
324
325
##### Milestones
326
327
* new Haketilo settings page interface implementation following the plan
328
* new Haketilo popup page implementation following the plan
329
* automated Haketilo GUI tests
330
331
##### Estimated time
332
333
2 weeks
334
335
## 21. REUSE specification compliance
336
337
License terms of software projects' files should be unambiguous and easy to analyze by humans
338
and computers alike. Compliance with the REUSE specification helps ensure that.
339
340
##### Milestones
341
342
* REUSE compliance in Hydrilla repository
343
* REUSE compliance in project website repository
344
* REUSE compliance in Haketilo repository
345
* REUSE compliance in custom site resources repository(ies)
346
347
##### Estimated time
348
349
1 week
350
351
## 22. Integrity constraints in Haketilo **(optional)**
352
353
One Haketilo custom site resource may depend on another, but initial versions of Haketilo did not
354
verify that dependencies are present. This and other sanity checks can be employed.
355
356
##### Milestones
357
358
* dependency checks when "installing" or upgrading a custom resource in Haketilo
359
* dependency checks when removing a custom resource from Haketilo
360
* facility for cascade removal
361
* validation of Haketilo URL patterns and other values typed in by the user
362
363
##### Estimated time
364
365
1 week
366
367
## 23. Sample meta-sites for Haketilo/Hydrilla **(optional)**
368
369
Running a static webapp like litewrite by visiting its website relies on the security of TLS and
370
network connectivity. Having it packaged as a separate browser extension requires giving it
371
excessive permissions. Running it from an HTML file is inconvenient.
372
373
##### Milestones
374
375
* at least 5 existing webapps packaged as meta-sites
376
* at least 5 meta-sites aggregating content from various client websites
377
378
##### Estimated time
379
380
3 weeks
381
382
## 24. Haketilo build system runnable from the browser **(optional)**
383
384
For portability of Haketilo's POSIX shell-based build system we avoided depending on Node.js,
385
NPM and similar tools. However, an even more portable alternative exists - to contain the build
386
system inside a standalone HTML page.
387
388
##### Milestones
389
390
* Javascript-based build system in an HTML page
391
* facility to run the javascript-based build system from the command line
392
393
##### Estimated time
394
395
2 weeks
396
397
## 25. User upload of content to Hydrilla website **(optional)**
398
399
To be able to easier gather and share custom site resources within the community, we need a
400
user-friendly platform.
401
402
##### Milestones
403
404
* registrations on a Hydrilla instance
405
* upload of custom site resources to a Hydrilla instance
406
* facility to easily and efficiently moderate the content uploaded by users
407
408
##### Estimated time
409
410
3 weeks
411
412
## 26. Further development of Hydrilla platform **(optional)**
413
414
Users should be able to share not only custom site resources but also their opinions about them.
415
416
##### Milestones
417
418
* support for user comments
419
* support for user ratings
420
* support for flagging site resources that are broken or have other issues
421
* development of comment quality control systems and policies
422
423
##### Estimated time
424
425
2 weeks
426
427
## 27. Facility for setting up Hydrilla repository mirrors **(optional)**
428
429
While allowing users to set up independent instances of Hydrilla gives them greater control over
430
site content they use, it does not by itself increase the robustness and maximum throughput of
431
Hydrilla platform. Enabling the use of mirrors does.
432
433
##### Milestones
434
435
* support for setting up and automatically synchronizing Hydrilla mirrors
436
* support for announcing available mirrors in Hydrilla
437
* support for fetching repository mirrors list in Haketilo
438
* support for distributing requests over multiple repository mirrors in Haketilo
439
* documentation
440
441
##### Estimated time
442
443
2 weeks
444
445
## 28. 150 sample site resources for Haketilo **(optional)**
446
447
To maintain community growth and participation, Hydrilla's collection must be visibly alive and
448
evolve with Haketilo's feature set.
449
450
##### Milestones
451
452
* at least 20 alternative site interfaces
453
* at least 20 existing webapps packaged as meta-sites
454
* at least 150 custom site resources in total
455
456
##### Estimated time
457
458
2 weeks
459
460
## 29. 200 sample site resources for Haketilo **(optional)**
461
462
To maintain community growth and participation, Hydrilla's collection must be visibly alive and
463
evolve with Haketilo's feature set.
464
465
##### Milestones
466
467
* at least 20 accessibility-improving site changes
468
* at least 10 meta-sites aggregating content from various client websites
469
* at least 200 custom site resources in total
470
471
##### Estimated time
472
473
2 weeks
474
475
## 30. Automated building of Haketilo source packages uploaded to Hydrilla **(optional)**
476
477
Requiring packagers to upload compiled code places an extra burden on them, and complicates
478
reproducibility. Hydrilla should be able to build from source packages.
479
480
##### Milestones
481
482
* Hydrilla automated resource builds feature
483
* security consultation of the feature
484
485
##### Estimated time
486
487
2 weeks
488
489
## 31. Self-documented Haketilo **(optional)**
490
491
Now matter how user-friendly the graphical interface is, an explanation of some of the concepts
492
might be needed. The next step, after having the documentation available on the project website,
493
is bundling it with the extension itself.
494
495
##### Milestones
496
497
* Haketilo popup self-documented inline
498
* Haketilo settings page self-documented inline
499
* documentation included as extension-bundled HTML pages
500
501
##### Estimated time
502
503
2 weeks
504
505
## 32. Displaying Hypothesis annotations for given site **(optional)**
506
507
Haketilo makes site resources for websites you visit available in only a few clicks. It would be
508
useful to have the same capacity for comments. The established, libre https://hypothes.is/ provides
509
a framework for this.
510
511
##### Milestones
512
513
* support for displaying current site's Hypothesis annotations in the popup
514
* support for adding adding Hypothesis annotations in Haketilo
515
516
##### Estimated time
517
518
2 weeks
519
520
## 33. Automatic generation of independent browser extensions from Haketilo site resources **(optional)**
521
522
Haketilo's rich feature set might also be an inconvenience. It may be overwhelming or irritating to
523
some users and has a higher risk of breaking with newer browser versions than a simple extension
524
would have. Thus, an option to install just a single Haketilo resource in the browser would be
525
useful.
526
527
##### Milestones
528
529
* automatic generation of Firefox WebExtensions from Haketilo site resources
530
* automatic generation of Chromium ManifestV3 WebExtensions from Haketilo site resources
531
532
##### Estimated time
533
534
2 weeks
535
536
## 34. Facility to automatically convert page's "native" scripts to a Haketilo resource **(optional)**
537
538
Haketilo gives users control over scripts being executed on a given web page. The scripts to be
539
used need to be defined in Haketilo as a resource. Doing this manually might be time-consuming
540
for a user who aims to use mostly the same javascript a website normally serves, but served from
541
within Haketilo.
542
543
##### Milestones
544
545
* automatic conversion of page's inline scripts in a Haketilo resource
546
* inclusion of page's external scripts in generated resource
547
* inclusion of page's intrinsic javascript events in generated resource
548
* displaying warnings when a site's javascript is known to use mechanisms that might stop
549
  such automatic package from working properly
550
551
##### Estimated time
552
553
3 weeks
554
555
## 35. Use of a standalone javascript engine to perform unit tests in Haketilo **(optional)**
556
557
A Selenium-driven web browser is currently used to test parts of Haketilo. Those tests that don't
558
rely on browser APIs could as well be run outside of browser which would save time during tests.
559
560
##### Milestones
561
562
* selected the javascript engine to use for testing
563
* facilitated writing Haketilo tests against the chosen engine
564
* applicable existing tests modified to be run without a web browser
565
566
##### Estimated time
567
568
2 weeks
569
570
## 36. Supplemental anti-bot measures in Hydrilla **(optional)**
571
572
Limiting the number of allowed registrations and content uploads is our planned basic way to
573
prevent Hydrilla instances from being harmed by automated requests. Another measures can be
574
added to further improve platform's resilience.
575
576
##### Milestones
577
578
* email-verified registrations
579
* selected an ethical, privacy-friendly captcha solution
580
* implementation of the chosen captcha solution
581
582
##### Estimated time
583
584
2 weeks
585
586
## 37. Support for external user authentication mechanisms in Hydrilla **(optional)**
587
588
It should be possible to run Hydrilla as part of a bigger web service. Users should be able to use
589
the same set of credentials for logging in in various parts of such service.
590
591
##### Milestones
592
593
* selected an authentication mechanism to support
594
* implementation of the feature
595
596
##### Estimated time
597
598
1 week
599
600
## 38. Support for building Hydrilla and Haketilo using Autotools **(optional)**
601
602
The specificity of Haketilo and Hydrilla means a complex build system like Autotools is not
603
necessary. It could, however, be added as optional to supplement their simple build mechanisms.
604
605
##### Milestones
606
607
* Hydrilla buildable with Autotools
608
* Hydrilla out-of-source builds possible
609
* Hydrilla tarball producible with a make rule
610
* Haketilo buildable with Autotools
611
* Haketilo out-of-source builds possible
612
* Haketilo tarball producible with a make rule
613
614
##### Estimated time
615
616
1 week
617
618
## 39. Evaluation of non-WebExtension platforms for the purpose of porting Haketilo **(optional)**
619
620
WebExtensions are really a convenient platform for developing software that empowers users. But
621
this platform is also tightly controlled by big organizations and has some serious limitations and
622
shortcomings.
623
624
##### Milestones
625
626
* evaluation of existing Webkit-based browsers
627
* evaluation of XUL extensions platform still used in some Firefox forks
628
* prepared evaluation report
629
630
##### Estimated time
631
632
1 week
633
634
## 40. Development of the first non-WebExtension Haketilo port **(optional)**
635
636
Users suffer a vendor lock-in with few mainstream web browsers. Lack of their favorite extensions
637
is what stops them from switching to more user-controlled alternatives. Haketilo should not
638
contribute to that problem.
639
640
##### Milestones
641
642
* selection of a target platform based on previous evaluation
643
* specification of tasks
644
* development roadmap
645
* prototype
646
* automated tests
647
* developer documentation
648
* user documentation
649
650
##### Estimated time
651
652
7.5 weeks