Proposal to start automating the process of updating imported WPT tests inside WebKit.

# Carlos Alberto Lopez Perez (a day ago)

Hello webkit-dev,

I would like to start a discussion about our current process for updating WPT tests, and to see if we can agree on some process to semi-automate it.

Currently the process of updating WPT tests its a very manual task which involves:

  1. Raise the WPT commit in LayoutTests/imported/w3c/resources/TestRepositories
  2. Run the importer script: Tools/Scripts/import-w3c-tests
  3. Check that what the script did makes sense (usually those changes are so big that's impossible to check manually)
  4. Run layout tests to generate new expectations for new tests.
  5. Upload the patch to bugzilla, wait for the EWS.
  6. Check what failed in each platform.
  7. Open new bugs for the new failures in reftests.
  8. Add the new failures to the TestExpectations.
  9. Repeat from 5. until everything is green in the EWS.
  10. Ask for review: the reviewer is usually also unable to review the changes in details because the patch is so big that its impossible to review in detail.
  11. Land it

This is so tedious task, that the last time we updated this globally was on 2018-September (that its the date of WPT commit 0313d9f which is what its pointed on the file LayoutTests/imported/w3c/resources/TestRepositories)

Engineers interested only in a subset of WPT tests (ex: domparsing/) what they do, its to update individual folders from WPT ToT (instead of updating all the imported WPT checkout). But this has its own problems: sometimes the updated folder depends on file from other folder that its not updated and unexpected problems happpen (example: webkit.org/b/205445)

I think that having the whole imported WPT checkout inside WebKit updated often would be a benefit for everybody.

So, we have the idea of trying to semi-automate this. On the steps above it would mean automating everything except the step of reviewing (10.)

Our proposal would be that a bot runs weekly and generates a patch with the update, then uploads it to Bugzilla, waits for the EWS, opens bugs for new failures and updates expectations accordingly; and repeats until everything is green in the EWS. Then the bot would CC the reviewers and ask for r? and cq?.

Note that this bot would not try to import new folders from WPT (folders for new specs) into WebKit. It would only update the ones already imported.

The bigger benefit would be having our set of imported WPT tests updated often, allowing WebKit engineers to run the regression tests with the last version of WPT tests and to get noticed early if some update on a WPT test breaks.

Also, engineers interested in WPT tests would not need anymore to manually update the tests since that its done now for them. Only they would need to import the new tests if those tests belong to a WPT folder (spec) still not imported inside WebKit.

Other side benefit of this is that reviewing the patch with the WPT checkout update that the bot generates would be something humanly possible to review. Since we would do that each few days, the changes hopefully would be in the dozens of files instead of the thousands of files.

The are also disadvantages to this, like the extra gardening work that will be generated due to this constant update of tests. Specially regarding flakies or failures not catched by the EWS (this may be a big problem for platforms that don't have an EWS running layout tests. [1])

So there its a trade-off to be made regarding the frequency of updates (daily, weekly, etc). Maybe we can start with weekly and see how it goes.

Just out of curiosity, I have checked how often Chromium updates their WPT imported tests, and they do that several times per day and fully automated [2].

We previously outlined this proposed process in this document: hackmd.io/b4ViVtDsSk6Qg8fAycexVA#3-Import-w3c-tests-running-in-a-bot

Thanks for reading so far.

Please let me know your thoughts.

Best regards!

[1] For GTK and WPE this should not be a problem as we are already looking forward to make our EWS bots run layout tests. [2] See: sprunge.us/a56RIo and chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md#Automatic

Contact us to advertise here

Want more features?

Request early access to our private beta of readable email premium.