Feature #66
Updated by koszko over 1 year ago
It seems problematic to test software that is meant to run as a browser extension - and it indeed is, especially when it comes to testing stuff that uses WebExtension APIs, not to mention inter-context messaging. However, there are some ways of testing code in-browser and we can always come up with our own ones. It is also not impossible to mock an environment with sites to inject scripts to (by meddling with `/etc/hosts` or employing an HTTP proxy).
Creativity wanted
Tests system quirks (listed here to save time of those who ever try to hack around it):
* It seems OK to navigate a Selenium driver to a page in a Pytest fixture but **only** when the fixture is function-scoped.
* Browser takes a lot of time to start, so it only makes sense to reuse a browser instance in subsequent tests. Here we achieve this by Pytest fixture scoping.
* Pytest fixtures that are not function-scoped **have to** be defined in `conftest.py` to work properly.
* Surprisingly, it is possible install a WebExtension when running Firefox in safe mode, but the extension has to be installed as **temporary**.
* There seems to be no straightforward way to learn extension's internal UUID (needed to navigate to extension's bundled files/pages) using Selenium. For unit tests the solution is to make the test extension open its own page when started (or redirect a certain HTTP request to its own page).
* Even when Firefox is started from Selenium with an empty profile, it loads the extensions that are installed globally on the system. Those can interfere with the tests and should be disabled, for example by a [properly-crafted `extensions.json` file](/projects/haketilo/repository/hachette/revisions/8475676ac6d631e2a34db25662aa88359a5fd98d/entry/test/default_profiles/icecat_empty/extensions.json). When running an a different system, it might be necessary to adjust the `extensions.json`. Because of that it is preferred to just test in Firefox' safe mode when possible.
* Selenium driver's `execute_script` method runs the script in some special, one-time context. If we want to run the script in page's context, we can inject it by adding a `<script>` tag to the page.
* If executed script returns a Promise, Selenium's `execute_script` method will wait for the promise to resolve and will return its result.
* When an error happens in a script injected via the `<script>` tag, it is a pain to debug it... Nevertheless, some useful info might then be obtained from the `geckodriver.log` file Selenium creates. `console.log`-based debugging is also possible if we disable headless mode in Firefox (which in our case is enabled by defining an environment variable in Makefile's test rule) and make the test code sleep in the relevant part of the failing test.
* Want to start a Selenium-driven IceCat instance together with a Python command-line where the driver can be manipulated? Run `make test-environment`.
* Want to only run a single test? Instead of `make test` run `pytest -vv -k <test-function-name>`. Optionally add `MOZ_HEADLESS=<whatever>` before the command to stop IceCat window from appearing. Rn running all tests on RockPro64 takes over 19s and running a single one takes over 10s.
The above were noted when testing with IceCat60. Might or might not be true for newer browsers...
Creativity wanted
Tests system quirks (listed here to save time of those who ever try to hack around it):
* It seems OK to navigate a Selenium driver to a page in a Pytest fixture but **only** when the fixture is function-scoped.
* Browser takes a lot of time to start, so it only makes sense to reuse a browser instance in subsequent tests. Here we achieve this by Pytest fixture scoping.
* Pytest fixtures that are not function-scoped **have to** be defined in `conftest.py` to work properly.
* Surprisingly, it is possible install a WebExtension when running Firefox in safe mode, but the extension has to be installed as **temporary**.
* There seems to be no straightforward way to learn extension's internal UUID (needed to navigate to extension's bundled files/pages) using Selenium. For unit tests the solution is to make the test extension open its own page when started (or redirect a certain HTTP request to its own page).
* Even when Firefox is started from Selenium with an empty profile, it loads the extensions that are installed globally on the system. Those can interfere with the tests and should be disabled, for example by a [properly-crafted `extensions.json` file](/projects/haketilo/repository/hachette/revisions/8475676ac6d631e2a34db25662aa88359a5fd98d/entry/test/default_profiles/icecat_empty/extensions.json). When running an a different system, it might be necessary to adjust the `extensions.json`. Because of that it is preferred to just test in Firefox' safe mode when possible.
* Selenium driver's `execute_script` method runs the script in some special, one-time context. If we want to run the script in page's context, we can inject it by adding a `<script>` tag to the page.
* If executed script returns a Promise, Selenium's `execute_script` method will wait for the promise to resolve and will return its result.
* When an error happens in a script injected via the `<script>` tag, it is a pain to debug it... Nevertheless, some useful info might then be obtained from the `geckodriver.log` file Selenium creates. `console.log`-based debugging is also possible if we disable headless mode in Firefox (which in our case is enabled by defining an environment variable in Makefile's test rule) and make the test code sleep in the relevant part of the failing test.
* Want to start a Selenium-driven IceCat instance together with a Python command-line where the driver can be manipulated? Run `make test-environment`.
* Want to only run a single test? Instead of `make test` run `pytest -vv -k <test-function-name>`. Optionally add `MOZ_HEADLESS=<whatever>` before the command to stop IceCat window from appearing. Rn running all tests on RockPro64 takes over 19s and running a single one takes over 10s.
The above were noted when testing with IceCat60. Might or might not be true for newer browsers...