This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 6, 2023. It is now read-only.
Full Fetch Test: Pending for March 2020 #18866
Description
Announcing to our ruleset maintainers that I plan to run this soon. So far, I have amended old config files and currently updating check_rules.py from the suggestions given from the last fetch test.
The tasks listed from the last time that are complete:
- Instead of disabling the entire ruleset, remove individual hosts (Update https-everywhere-checker to disable single target hosts rather than entire rules #1998)
- The report that is generated when a ruleset is disabled (beside the
Disabled by https-everywhere-checker becausetext) should include a timestamp
Before Each Run
- Update the cipher list in https://github.com/EFForg/https-everywhere/blob/c7da7b7eb704a697cc3d44e406c1912e45b954a9/test/rules/src/https_everywhere_checker
/http_client.py#L48-L49 with ciphers from an updated version of Firefox ESR (Update Cipher List for Pending Fetch Test #18964) - Create an issue (or send an email) announcing that there will be a forthcoming run of the full fetch test
- Update certificate bundle https://github.com/EFForg/https-everywhere/blob/master/utils/create-platform-certs/README.md
- Update User agent in
http.checker.configUse Tor Browser user agent for fetch tests #19004
Also, I wanted to be able to run this test and have it be reviewable for the ruleset maintainers when it is pushed. So I open to what can make PR's more reviewable in the future.
I am also going to track this in a project in the repo so that way progress can be tracked with fetch tests in the future and hopefully run this quarterly and not every 2-3 years anymore.
Activity
J0WI commentedon Feb 20, 2020
We should review some of the failed top-* rulesets before submitting a new ruleset update. Probalby also for .onion rules, because the results might not be very reliable there.
zoracon commentedon Feb 21, 2020
@J0WI Should I hold off on the ruleset update for this week? Or are you referencing the Fetch Test update?
J0WI commentedon Feb 21, 2020
The scheduled release is not affected. The fetch test will cause a bunch of disabled rules and we probably do not want to ship a ruleset update to the API right afterwards that disables important rules for all users.
J0WI commentedon Feb 21, 2020
The whitelist needs to be updated after each run. See also #18967.
pipboy96 commentedon Feb 22, 2020
@zoracon We also need to update the certificate bundle.
zoracon commentedon Feb 23, 2020
@J0WI I am doing my best to figure out a reviewable fetch test result this time around. I don't expect the results the fetch test to be all patched at once, but I will likely spend time over the next month (or two) implementing the results with the rest of the maintainers here.
cschanaj commentedon Mar 4, 2020
We might also want to update the user-agent of the https-everywhere-checker to reflect the current browser requirement before the next full fetch test. This prevent absurd fetch errors when some websites perform user-agent sniffing and return different content.
https-everywhere/test/rules/http.checker.config
Line 19 in bc3c6a9
pipboy96 commentedon Mar 4, 2020
@cschanaj I think we should use current Tor Browser's user agent.
cschanaj commentedon Mar 4, 2020
I agree it is a good idea to follow the Tor Browser configuration, a relatively up-to-date Tor Blog entry can be our reference: https://blog.torproject.org/browser-fingerprinting-introduction-and-challenges-ahead
zoracon commentedon Mar 11, 2020
As an update, it seems like all steps are covered with updates and adjustments. Now I am running a soft test to adjust the new workflow for reviewable patches.
For context, the current git flow for the fetch tests runs
git diff > /patch/diff.patch. I am adjusting the portion to utilizegit format-patch.Right now I am testing a utility that breaks up the patch by file.
That way I can write a script that runs
git apply --check <patchfile>and then give over reviewable patches over to the maintainers and myself for subsequent review.zoracon commentedon Mar 12, 2020
Soft run complete, took about a full day, thinking about posting up the fully disabled rulesets in a PR so the actual fetch test rounds can just focus on the ones with targets disabled.
pipboy96 commentedon Mar 12, 2020
@zoracon That's a good idea! Looking forward to your PR. Also, can we fully delete rulesets if all targets return NXDOMAIN?
J0WI commentedon Mar 18, 2020
#18964 has introduced a regression in ruleset testing: #19034
cschanaj commentedon Jun 3, 2020
blocked by #19330
zoracon commentedon Oct 22, 2020
Closing this out since this process has been run, reviewed, and merged.
Postmortem here: https://github.com/EFForg/https-everywhere/wiki/Postmortem:-Fetch-Test-April-2020