Project

General

Profile

Feature #66

Write tests

Added by koszko 4 months ago. Updated 2 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Start date:
11/27/2021
Due date:
% Done:

50%

Estimated time:
(Total: 0.00 h)

Description

It seems problematic to test software that is meant to run as a browser extension - and it indeed is, especially when it comes to testing stuff that uses WebExtension APIs, not to mention inter-context messaging. However, there are some ways of testing code in-browser and we can always come up with our own ones. It is also not impossible to mock an environment with sites to inject scripts to (by meddling with /etc/hosts or employing an HTTP proxy).

Creativity wanted


Subtasks

Feature #97: Make tests system parametrizable through configure scriptNew

Actions

History

#1

Updated by koszko 4 months ago

  • Description updated (diff)
#2

Updated by jahoti 4 months ago

Mocking sites is definitely critical, albeit probably better done with a hijacking proxy of some sort (my words, not a known technical term) than with the extremely limited options of /etc/hosts.

As for conducting the actual tests themselves, shouldn't it be enough to use a browser driver for all the UI interactions and then do the rest by adding JS to "test builds" of the extension, which can communicate with the main test suite using AJAX? Perhaps- quite likely- I'm completely misunderstanding the issue.

#3

Updated by koszko 4 months ago

jahoti wrote:

Mocking sites is definitely critical, albeit probably better done with a hijacking proxy of some sort (my words, not a known technical term) than with the extremely limited options of /etc/hosts.

If it can allow us to do tests without having to run a VM/chroot/container/UML/Linux namespace, then proxy is preferable.

As for conducting the actual tests themselves, shouldn't it be enough to use a browser driver for all the UI interactions

I haven't used browser drivers before, but they indeed seem to be the way to go.

and then do the rest by adding JS to "test builds" of the extension, which can communicate with the main test suite using AJAX?

Yep, that's what we should do.

Perhaps- quite likely- I'm completely misunderstanding the issue.

You are not :)

#4

Updated by jahoti 4 months ago

  • Assignee set to jahoti

This is now off to a (very slow) start.

It's currently in a separate folder to Hachette; should that continue, or would it be best to add the test suite to the extension repository.

#5

Updated by koszko 4 months ago

jahoti wrote:

This is now off to a (very slow) start.

It's currently in a separate folder to Hachette; should that continue, or would it be best to add the test suite to the extension repository.

I see no problem in having it in the repo. After all, it is somehow related to the code (i.e. changes in code force changes in tests).

Please for now only focus on things that are not going to change quickly. Btw, currently considered changes are:

  1. Giving the same semantics to *** in both domain and path parts of a pattern
  2. Removing page_info_server
#6

Updated by jahoti 4 months ago

Please for now only focus on things that are not going to change quickly.

I'll make sure to once it gets to that stage; currently the entire extent of the test suite is a stripped down MITM proxy :).

#7

Updated by jahoti 3 months ago

  • % Done changed from 0 to 10

The basic infrastructure to support creating a "virtual network" in now in the jahoti branch, and can be used on its own (with some extra web pages and bypassing an error Chromium throws).

As soon as I finally manage to get Selenium working, this can actually be put to use in the way it was intended.

#8

Updated by koszko 3 months ago

Have you considered using UML (no, not that diagraming language, I mean User Mode Linux) to run tests inside? I'm suggesting this just in case

#9

Updated by jahoti 3 months ago

Have you considered using UML (no, not that diagraming language, I mean User Mode Linux) to run tests inside? I'm suggesting this just in case

I didn't actually- thanks for suggesting it! Once the basic test system is running and reporting results, that will be the next job before finally working on a thorough set of tests once (hopefully) the core features and interface are stable.

#10

Updated by koszko 5 days ago

  • Status changed from New to In Progress
  • Assignee changed from jahoti to koszko

jahoti wrote:

Have you considered using UML (no, not that diagraming language, I mean User Mode Linux) to run tests inside? I'm suggesting this just in case

I didn't actually- thanks for suggesting it! Once the basic test system is running and reporting results, that will be the next job before finally working on a thorough set of tests once (hopefully) the core features and interface are stable.

I've been thinking more about this recently. We want some way to isolate the test environment from other netowrking on the machine to avoid interference. We could use either a full virtual machine, UML or some container (e.g. a Docker one). I earlier suggested UML because full virtual machine seems like on overkill and I personally like UML more than Docker (trauma after having to resist Docker Hub's nonfree js back when I was a student). Whether we choose Docker or UML, we still need operating system's entire userspace to be installed in it. While I don't consider it unacceptably unfeasible, it is a bit of an inconvenience.

We can instead use Linux network namespaces. This way we can still use host's programs. I know, it adds the requirement that host OS needs to be using Linux (which would also be the case if we used UML), yet I think this is acceptable (it is the build system that has to be portable). We don't want to require root access to run tests, so we need to use user namespaces. Those used to be disabled by default on Debian kernels which is yet another inconvenience that I think we should just swallow. However, here I use jxself's linux-libre .deb package and it seems to have user namespaces enabled by default :)

So, to spawn a shell in its own network namespace with uid=0 and gid=0, we do:

unshare -Urn

Now we have not only network isolation but also the ability to mess with the firewall.
Please look into unshare's manpage to find more details. Or don't. I will be working on this crazy namespace stuff now :)

#11

Updated by koszko 2 days ago

  • % Done changed from 10 to 50

unshare doesn't seem to work when in chroot and I currently run tests agains a browser I have installed inside a chroot. I will try to modify the makefile rule to test if unshare can be used and only use it if it can. Otherwise, port 1337 must be free the our internal test proxy to listen on.

I added to the Python test code from jahoti branch. The koszko branch now has a make test target added which uses Selenium WebDriver to spawn a headless IceCat instance which then accesses an example page over HTTPS through our internet-mocking proxy server. There's also a make test-environment target that spawns a headed IceCat instance as well as a Python console where WebDriver attached to this IceCat instance can be accessed under the driver variable.

In both make test and make test-environment the IceCat instance is made to accessed a mocked internet through a proxy which serves content as defined in test/world_wide_library.py.

Current single test case only checks if a mocked page gets loaded properly. More tests that will check Haketilo's code will be added. This will probably involve doing #79.

In my case, IceCat was loading some globally-installed extensions that would interfere with the test, so I included a minimal profile that disables them. This is obviously just a workaround that will not work everywhere...
Nevertheless, I suggest we instead rely on Firefox' safe mode in unit tests and only use the minimal profile when testing Haketilo as a whole. Currently make test and make test-environment use safe mode exclusively.

For all this I've been using Parabola's IceCat 60 (the older browser we use for testing, the less likely we are to later learn that we're unknowingly relying on some feature that is unavailable somewhere else). Chromium-based tests would also be cool to have, but I am not considering them a priority

EDIT
You currently need the following to run tests:

  • a supported browser (currently Parabola's IceCat60, would be cool if you could add support for other ones)
  • python3
  • pytest
  • Python binding for Selenium (python-selenium in Parabola)
  • geckodriver (geckodriver in Parabola)

Also available in: Atom PDF