http://hydrillabugs.koszko.org/http://hydrillabugs.koszko.org/favicon.ico?15861920342021-07-26T14:14:38ZHydrilla issue trackerHaketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=1242021-07-26T14:14:38Zkoszkokoszko@koszko.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/124/diff?detail_id=91">diff</a>)</li></ul> Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=1302021-07-28T05:27:58Zjahoti
<ul></ul><p>Mocking sites is definitely critical, albeit probably better done with a hijacking proxy of some sort (my words, not a known technical term) than with the extremely limited options of <code>/etc/hosts</code>.</p>
<p>As for conducting the actual tests themselves, shouldn't it be enough to use a browser driver for all the UI interactions and then do the rest by adding JS to "test builds" of the extension, which can communicate with the main test suite using AJAX? Perhaps- quite likely- I'm completely misunderstanding the issue.</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=1362021-07-28T07:24:04Zkoszkokoszko@koszko.org
<ul></ul><p>jahoti wrote:</p>
<blockquote>
<p>Mocking sites is definitely critical, albeit probably better done with a hijacking proxy of some sort (my words, not a known technical term) than with the extremely limited options of <code>/etc/hosts</code>.</p>
</blockquote>
<p>If it can allow us to do tests without having to run a VM/chroot/container/UML/Linux namespace, then proxy is preferable.</p>
<blockquote>
<p>As for conducting the actual tests themselves, shouldn't it be enough to use a browser driver for all the UI interactions</p>
</blockquote>
<p>I haven't used browser drivers before, but they indeed seem to be the way to go.</p>
<blockquote>
<p>and then do the rest by adding JS to "test builds" of the extension, which can communicate with the main test suite using AJAX?</p>
</blockquote>
<p>Yep, that's what we should do.</p>
<blockquote>
<p>Perhaps- quite likely- I'm completely misunderstanding the issue.</p>
</blockquote>
<p>You are not :)</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=2012021-08-05T09:47:30Zjahoti
<ul><li><strong>Assignee</strong> set to <i>jahoti</i></li></ul><p>This is now off to a (very slow) start.</p>
<p>It's currently in a separate folder to Hachette; should that continue, or would it be best to add the test suite to the extension repository. </p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=2032021-08-05T10:30:16Zkoszkokoszko@koszko.org
<ul></ul><p>jahoti wrote:</p>
<blockquote>
<p>This is now off to a (very slow) start.</p>
<p>It's currently in a separate folder to Hachette; should that continue, or would it be best to add the test suite to the extension repository.</p>
</blockquote>
<p>I see no problem in having it in the repo. After all, it is somehow related to the code (i.e. changes in code force changes in tests).</p>
<p>Please for now only focus on things that are not going to change quickly. Btw, currently considered changes are:</p>
<ol>
<li>Giving the same semantics to <code>***</code> in both domain and path parts of a pattern</li>
<li>Removing <code>page_info_server</code></li>
</ol>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=2042021-08-06T00:42:24Zjahoti
<ul></ul><blockquote>
<p>Please for now only focus on things that are not going to change quickly. </p>
</blockquote>
<p>I'll make sure to once it gets to that stage; currently the entire extent of the test suite is a stripped down MITM proxy :).</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=3132021-09-05T02:29:57Zjahoti
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul><p>The basic infrastructure to support creating a "virtual network" in now in the <code>jahoti</code> branch, and can be used on its own (with some extra web pages and bypassing an error Chromium throws).</p>
<p>As soon as I finally manage to get Selenium working, this can actually be put to use in the way it was intended.</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=3472021-09-09T11:52:05Zkoszkokoszko@koszko.org
<ul></ul><p>Have you considered using UML (no, not that diagraming language, I mean User Mode Linux) to run tests inside? I'm suggesting this just in case</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=3562021-09-11T02:44:12Zjahoti
<ul></ul><blockquote>
<p>Have you considered using UML (no, not that diagraming language, I mean User Mode Linux) to run tests inside? I'm suggesting this just in case</p>
</blockquote>
<p>I didn't actually- thanks for suggesting it! Once the basic test system is running and reporting results, that will be the next job before finally working on a thorough set of tests once (hopefully) the core features and interface are stable.</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=4392021-11-24T08:38:18Zkoszkokoszko@koszko.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>jahoti</i> to <i>koszko</i></li></ul><p>jahoti wrote:</p>
<blockquote>
<blockquote>
<p>Have you considered using UML (no, not that diagraming language, I mean User Mode Linux) to run tests inside? I'm suggesting this just in case</p>
</blockquote>
<p>I didn't actually- thanks for suggesting it! Once the basic test system is running and reporting results, that will be the next job before finally working on a thorough set of tests once (hopefully) the core features and interface are stable.</p>
</blockquote>
<p>I've been thinking more about this recently. We want some way to isolate the test environment from other netowrking on the machine to avoid interference. We could use either a full virtual machine, UML or some container (e.g. a Docker one). I earlier suggested UML because full virtual machine seems like on overkill and I personally like UML more than Docker (trauma after having to resist Docker Hub's nonfree js back when I was a student). Whether we choose Docker or UML, we still need operating system's entire userspace to be installed in it. While I don't consider it unacceptably unfeasible, it is a bit of an inconvenience.</p>
<p>We can instead use Linux network namespaces. This way we can still use host's programs. I know, it adds the requirement that host OS needs to be using Linux (which would also be the case if we used UML), yet I think this is acceptable (it is the build system that has to be portable). We don't want to require root access to run tests, so we need to use user namespaces. Those used to be disabled by default on Debian kernels which is yet another inconvenience that I think we should just swallow. However, here I use jxself's linux-libre .deb package and it seems to have user namespaces enabled by default :)</p>
<p>So, to spawn a shell in its own network namespace with uid=0 and gid=0, we do:</p>
<pre><code class="shell syntaxhl">unshare <span class="nt">-Urn</span>
</code></pre>
<p>Now we have not only network isolation but also the ability to mess with the firewall.<br>
Please look into unshare's manpage to find more details. Or don't. I will be working on this crazy namespace stuff now :)</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=4462021-11-27T12:51:25Zkoszkokoszko@koszko.org
<ul><li><strong>% Done</strong> changed from <i>10</i> to <i>50</i></li></ul><p><code>unshare</code> doesn't seem to work when in chroot and I currently run tests agains a browser I have installed inside a chroot. I will try to modify the makefile rule to test if unshare can be used and only use it if it can. Otherwise, port 1337 must be free the our internal test proxy to listen on.</p>
<p>I added to the Python test code from <code>jahoti</code> branch. The <code>koszko</code> branch now has a <code>make test</code> target added which uses Selenium WebDriver to spawn a headless IceCat instance which then accesses an example page over HTTPS through our internet-mocking proxy server. There's also a <code>make test-environment</code> target that spawns a headed IceCat instance as well as a Python console where WebDriver attached to this IceCat instance can be accessed under the <code>driver</code> variable.</p>
<p>In both <code>make test</code> and <code>make test-environment</code> the IceCat instance is made to accessed a mocked internet through a proxy which serves content as defined in <code>test/world_wide_library.py</code>.</p>
<p>Current single test case only checks if a mocked page gets loaded properly. More tests that will check Haketilo's code will be added. This will probably involve doing <a class="issue tracker-2 status-5 priority-1 priority-lowest closed" title="Feature: Improve the build script by using awk (Closed)" href="http://hydrillabugs.koszko.org/issues/79">#79</a>.</p>
<p>In my case, IceCat was loading some globally-installed extensions that would interfere with the test, so I included a minimal profile that disables them. This is obviously just a workaround that will not work everywhere...<br>
Nevertheless, I suggest we instead rely on Firefox' safe mode in unit tests and only use the minimal profile when testing Haketilo as a whole. Currently <code>make test</code> and <code>make test-environment</code> use safe mode exclusively.</p>
<p>For all this I've been using Parabola's IceCat 60 (the older browser we use for testing, the less likely we are to later learn that we're unknowingly relying on some feature that is unavailable somewhere else). Chromium-based tests would also be cool to have, but I am not considering them a priority</p>
<p><strong>EDIT</strong><br>
You currently need the following to run tests:</p>
<ul>
<li>a supported browser (currently Parabola's IceCat60, would be cool if you could add support for other ones)</li>
<li>python3</li>
<li>pytest</li>
<li>Python binding for Selenium (<code>python-selenium</code> in Parabola)</li>
<li>geckodriver (<code>geckodriver</code> in Parabola)</li>
</ul>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=4502021-11-30T03:07:35Zjahoti
<ul></ul><p>It's working nicely! I've had a look at the unshare manpage as well, and while it deserves a more thorough read you've definitely found the right tool.</p>
<blockquote>
<p>unshare doesn't seem to work when in chroot and I currently run tests agains a browser I have installed inside a chroot. I will try to modify the makefile rule to test if unshare can be used and only use it if it can. Otherwise, port 1337 must be free the our internal test proxy to listen on.</p>
</blockquote>
<p>Alternatively, the default of <code>1337</code> could be removed; there's a function <code>free_port</code> in <code>selenium.webdriver.common.utils</code> which returns a free port.</p>
<blockquote>
<p>I added to the Python test code from jahoti branch. The koszko branch now has a make test target added which uses Selenium WebDriver to spawn a headless IceCat instance which then accesses an example page over HTTPS through our internet-mocking proxy server. There's also a make test-environment target that spawns a headed IceCat instance as well as a Python console where WebDriver attached to this IceCat instance can be accessed under the driver variable.</p>
<p>In both make test and make test-environment the IceCat instance is made to accessed a mocked internet through a proxy which serves content as defined in test/world_wide_library.py.</p>
<p>Current single test case only checks if a mocked page gets loaded properly. More tests that will check Haketilo's code will be added. This will probably involve doing <a class="issue tracker-2 status-5 priority-1 priority-lowest closed" title="Feature: Improve the build script by using awk (Closed)" href="http://hydrillabugs.koszko.org/issues/79">#79</a>.</p>
</blockquote>
<p>I see you've started on that!</p>
<blockquote>
<p>In my case, IceCat was loading some globally-installed extensions that would interfere with the test, so I included a minimal profile that disables them. This is obviously just a workaround that will not work everywhere...<br>
Nevertheless, I suggest we instead rely on Firefox' safe mode in unit tests and only use the minimal profile when testing Haketilo as a whole. Currently make test and make test-environment use safe mode exclusively.</p>
</blockquote>
<p>Definitely- they're hardly unit tests otherwise :).</p>
<blockquote>
<p>For all this I've been using Parabola's IceCat 60 (the older browser we use for testing, the less likely we are to later learn that we're unknowingly relying on some feature that is unavailable somewhere else). Chromium-based tests would also be cool to have, but I am not considering them a priority</p>
</blockquote>
<p>To save some debugging for whoever takes on Chromium, some notes:</p>
<ul>
<li>Extensions <em>cannot be used</em> in headless mode</li>
<li>While it's probably been fixed, <code>driver.quit()</code> doesn't kill the actual browser with my selenium installation</li>
<li>The driver needs to be the same version as the browser to work</li>
</ul>
<blockquote>
<p>EDIT<br>
You currently need the following to run tests:</p>
<ul>
<li>a supported browser (currently Parabola's IceCat60, would be cool if you could add support for other ones)</li>
</ul>
</blockquote>
<p>I'll try <a class="issue tracker-2 status-5 priority-2 priority-default closed child" title="Feature: Make tests system parametrizable through configure script (Closed)" href="http://hydrillabugs.koszko.org/issues/97">#97</a> and see what other customizations need to be made after that.</p>
<blockquote>
<ul>
<li>python3</li>
<li>pytest</li>
<li>Python binding for Selenium (python-selenium in Parabola)</li>
<li>geckodriver (geckodriver in Parabola)</li>
</ul>
</blockquote>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=4562021-12-01T13:10:53Zkoszkokoszko@koszko.org
<ul></ul><p><code>koszko</code> branch now has a facility to load .js files together with their dependencies to a page in selenium-driven browser.</p>
<p><strong>UPDATE</strong><br>
I improved the facility. It is now possible to use selenium to execute scripts in page's global scope. I had to do workarounds to enable reporting of errors from such scripts.</p>
<p>It now works like this: if <code>a.js</code> depends on <code>b.js</code> which depends on <code>c.js</code>, our new facility can load them into a test page this way:</p>
<ol>
<li><code>exports_init.js</code> is loaded, raw</li>
<li><code>c.js</code> is loaded, wrapped in an anonymous function, with its exports available through <code>window</code> object</li>
<li><code>b.js</code> is loaded in the same fashion</li>
<li><code>a.js</code> is loaded <strong>without</strong> a wrapping anonymous function but with imports satisfied</li>
</ol>
<p>Then, subsequent scripts can be loaded and they can access private global variables from <code>a.js</code></p>
<p>See <code>test/unit/test_patterns.py</code> for an example.</p>
<p>Other quirks I fought:</p>
<ul>
<li>Custom-scoped pytest fixtures myst be either in the same file that uses them or in <code>conftest.py</code>. I have no idea why...</li>
<li>When IceCat60 reports javascript syntax error on line 172 in index.html, it means the error is on line 172 of the script, not 172 of the html page.</li>
<li>Selenium driver's <code>execute_script</code> method executes scripts in some kind of one-time scope. They have access to <code>window</code> (and hence variables declared by page's scripts using <code>var</code>) but no access to variables declared by <code>const</code> or <code>let</code> nor to variables declared in previous calls to <code>execute_script</code>. That's the reason for all the hassle :)</li>
<li>"<code>get_log</code> is not implemented by Firefox driver."</li>
</ul>
<p>Selenium tests seem to be resource-intensive. On my RockPro64 Merely starting IceCat60 takes ~10s and the few test cases we have now already take another 2s. I guess there's little we can do about it right now...</p>
<p>I am now working on querying of URL patterns. Breakages in code pushed to <code>koszko</code> might occur ;)</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=4602021-12-02T23:57:04Zjahoti
<ul></ul><p>This seems to be coming together well!</p>
<p>I'm in the process of addressing <a class="issue tracker-2 status-5 priority-2 priority-default closed child" title="Feature: Make tests system parametrizable through configure script (Closed)" href="http://hydrillabugs.koszko.org/issues/97">#97</a>, which (along with a few other build system parameters) should make it to the <code>build-sys</code> branch tomorrow at the very latest.</p>
Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=4692021-12-13T18:19:17Zkoszkokoszko@koszko.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/469/diff?detail_id=270">diff</a>)</li></ul><p><code>koszko</code> branch now has code to use test extensions in unit tests</p>
<p>Typical scenario:<br>
You have some code that uses WebExtension APIs. You use the <code>webextension</code> Pytest fixture from <code>conftest.py</code>, parameterizing it through a <code>pytest.mark</code> decorator:</p>
<pre><code class="python syntaxhl"><span class="c1"># You could as well pass an empty dict and use the defaults.
</span><span class="o">@</span><span class="n">pytest</span><span class="p">.</span><span class="n">mark</span><span class="p">.</span><span class="n">ext_data</span><span class="p">({</span>
<span class="s">'background_script'</span><span class="p">:</span> <span class="s">'console.log("this will run in background page");'</span><span class="p">,</span>
<span class="s">'content_script'</span><span class="p">:</span> <span class="s">'console.log("this will run at document_start on every page");'</span><span class="p">,</span>
<span class="s">'test_page'</span><span class="p">:</span> <span class="s">'<h1>this will be the extension-bundled page opened when the test function is called</h1>'</span>
<span class="p">})</span>
<span class="k">def</span> <span class="nf">test_something</span><span class="p">(</span><span class="n">driver</span><span class="p">,</span> <span class="n">execute_in_page</span><span class="p">):</span>
<span class="c1"># do something
</span> <span class="k">assert</span> <span class="n">something</span>
</code></pre> Haketilo - Feature #66: Write testshttp://hydrillabugs.koszko.org/issues/66?journal_id=5202022-02-19T10:22:28Zkoszkokoszko@koszko.org
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>50</i> to <i>100</i></li></ul><p>Quite a lot of test cases are now in <code>master</code></p>