Site fixes

Thanks, this has a potential to be very useful :)

Upon registering, have you noticed anything bad in GitHub's ToS that we should be aware of?

Btw, in cases where one mapping expects another to be installed (e.g. to make an iframe on some page functional), you can use the new "required_mappings" key in index.json to indicate that. This feature is available starting from Hydrilla 1.1 and Haketilo 2.0. It's my fault that I didn't document it on the wiki... Anyway, you can see (and I suppose you've already seen) an example here :)

Strangely I don't remember GitHub asking me to agree to a terms of service. I did find a terms page [1], but there's a lot of pages, so I haven't read them all yet. The only thing I've noticed so far is that the GitHub Terms of Service [2] says "One person or legal entity may maintain no more than one free Account", which I definitely violated in testing (and, I've made multiple accounts for school before too, before I knew about free software). But of course that's not a problem inherent to the free software client, just something that's a bit more likely to occur when developing a client (Though, now that I know the launch code entry page doesn't require JavaScript, I can just test getting to that page, as getting to that page doesn't require actually accessing the email used.). I'll make another comment if I find something else more concerning in the terms.

[1] https://docs.github.com/en/site-policy/github-terms

[2] https://docs.github.com/en/site-policy/github-terms/github-terms-of-service

Oh also, thanks for the required_mappings example. For some reason I didn't think to look at Hacktcha for an example... For some reason I thought I would need multiple resources but then one mapping that links all of them, but I guess required_mappings is better because that means you can still install individual lower level components by themselves if you want (e.g. just the Octocaptcha).

jacobk wrote:

Oh also, thanks for the required_mappings example.

YW :)

Whether multiple payloads would actually suffice in this case - I don't know for sure. I can't find any evidence of Octocaptcha being used somewhere else... but you also have some code to integrate with funcaptcha. And this one seems to also be used by other websites, is that correct?

I doubt there are other sites using Octocaptcha, as I think the name refers to the GitHub mascot, but it is possible to solve the Octocaptcha by itself for testing [1], and the URL has query strings with makes me think maybe some other parts of the site use Octocaptcha differently?

I think Octocaptcha is basically GitHub's custom frontend for FunCaptcha, so I think other sites do use FunCaptcha. If I remember correctly, Roblox uses FunCaptcha, but I think it's a different kind of FunCaptcha so I'm not sure if the code I have already written would carry over.

Also, for some reason I was unable to get Hydrilla to serve the GitHub scripts without upgrading to from Hydrilla 1.0 to Hydrilla 1.1b. Other packages seemed to work fine on 1.0.

[1] https://octocaptcha.com/?origin_page=github_signup_next&responsive=true&require_ack=true&version=2

Also, for some reason I was unable to get Hydrilla to serve the GitHub scripts without upgrading to from Hydrilla 1.0 to Hydrilla 1.1b.

The convenient "mapping_and_resource" was only added in 1.1b1. Well, it's only applicable to Hydrilla builder (the definitions it produces don't use such shortcuts), so serving stuff that was already built should also be possible with 1.0. Which is probably not very useful under local development setting since server requires exactly the same version of builder to be installed...

With the current version of the fix, using the HTTPS Everywhere extension with "Encrypt All Sites Eligible" enabled will break the visual component of the script. I have not investigated why this happens yet. Firefox/Abrowser's built-in "HTTPS-Only Mode" does not seem to cause problems.

Also, I've been occasionally running into a problem when updating fixes, in which Haketilo for some reason keeps requesting an old resource after downloading a new mapping, and removing the resource leads to getting a 404 when trying to install the fix (as the old resource is no longer in the build directory). Removing the mapping in Haketilo too does not seem to fix it, but neither does deleting and restoring the build directory, so I think the problem is with Haketilo.

I haven't seen this happen on my profile that only has Haketilo installed, so I can't say for sure that it's not the interference of another extension that's causing the problem, but it would be weird if it was another extension's fault since the requests are happening in the context of the extension and not a web page. If I catch it happening on Haketilo by itself, then I'll open a separate issue for it.

jacobk wrote:

so I think the problem is with Haketilo.

No, it's a problem with Hydrilla 😅

Browser caches some of the responses. It does not cache the ones that are generated dynamically with Python code but it can cache files served with flask's send_file() function. This line needs to be changed.

I experienced that bug before but didn't actually prioritize fixing it since the issue only manifests itself when running a development server of Hydrilla. So far I've been just using Ctrl+Shift+del Firefox shortcut to quickly clear the browser's cache

No need for you to create a new issue for it - I'll correct this behavior in the next release!

The fix as it is now is maybe good enough to publish. It's now usable entirely through the GUI, but audio CAPTCHA is still the only option, and there are some values sent in requests that I am not sure how to get/generate, namely "bda" and "bio", which apparently aren't actually necessary, but sending "???" for them makes it very easy for GitHub to tell that people are using the script, if they decide they want to block it for some reason, or if they decide that most user of the script were actually bots, and they want to ban all users that used the script (not sure if they'd really do that though).

Both "bda" and "bio" are very long strings of variable length that start with "ey", and I've seen strings that match that description outside of GitHub (e.g. on Microsoft login if I remember correctly, though in that case I was able to tell where the value came from IIRC), and I did find a website that might be talking about something similar to what GitHub does, but I'm not sure: https://aws.amazon.com/premiumsupport/knowledge-center/appsync-wrong-query-item-number-dynamodb/

Oops, the duplicate comments are because of Abrowser showing an HTTPS-Only Mode warning when I try to submit, but then when I go back the message is still in the edit box so I think it wasn't sent, but it actually was sent despite the error message.

Not sure if that's an Abrowser/Firefox issue or a Hydrilla-issue-tracker/Redmine issue.

jacobk wrote:

The fix as it is now is maybe good enough to publish.

Thanks! I'll look at it soon.

there are some values sent in requests that I am not sure how to get/generate

I had a similar problem with reCAPTCHA (although the strings there used a different format) and after some thinking I decided not to worry about it too much. Especially considering how much precious time RE-ing the generation of those strings could take.

Oops, the duplicate comments are because of Abrowser showing an HTTPS-Only Mode warning when I try to submit [...]

No problem, I removed the duplicates :)

Not sure if that's an Abrowser/Firefox issue or a Hydrilla-issue-tracker/Redmine issue.

It could be a hydrillabugs issue as at some point, after renaming "Hachette" to "Haketilo", I used Apache's mod_html to replace all "hachette" links in pages being served with "haketilo" links... I later realized this was a mistake as one of the side-effects was the rendering of hydrillabugs in Quirks mode. Perhaps this also caused the HTTP-only mode issue?

I changed the Apache configuration to not perform any HTML rewrites (I should've done that long ago...). Let's see if the issues persist now.

I found some information about "bda" that looks like it would be very useful if GitHub decides to start checking for it. It looks like it is for fingerprinting.

https://github.com/keepitLowkey/py-funcaptcha/blob/master/py_funcaptcha.py

Also this looks like it has some information about solving visual CAPTCHAs, but I haven't looked into it much: https://archive.ph/GbaCZ

Disgusting. Well, good to know at least.

Would be cool to at some point spend more time RE-ing this kind of stuff (it's actually more entertaining than work on Haketilo itself)