Safari Tech Preview 96 available on wpt.fyi!

# Stephen Mcgruer (9 days ago)

Excited to announce that Safari Tech Preview 96 is now available on wpt.fyi!

Example run: wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned

Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but very close): wpt.fyi/results/?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013

If you are surprised by the results in the diff view, I would love to hear about it. One thing that surprised me is that the release blog post[0] mentioned Web Animations being enabled by default, but we see no differences in the test diff for web-animations/. It's possible we enable some flag that turned on Web Animations already, but I can't see it obviously in our safari setup[1].

[0]: webkit.org/blog/9658/release-notes-for-safari-technology-preview-96 [1]: web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml

Contact us to advertise here
# Antoine Quint (8 days ago)

On 27 Nov 2019, at 16:07, Stephen Mcgruer <smcgruer at chromium.org> wrote:

One thing that surprised me is that the release blog post[0] mentioned Web Animations being enabled by default, but we see no differences in the test diff for web-animations/. It's possible we enable some flag that turned on Web Animations already, but I can't see it obviously in our safari setup[1].

Safari Technology Preview had been shipping with Web Animations on by default prior to that change. The value DEFAULT_EXPERIMENTAL_FEATURES_ENABLED is true for STP builds. So it’s correct that you don’t see any change.

Antoine

# Stephen Mcgruer (8 days ago)

Safari Technology Preview had been shipping with Web Animations on by default prior to that change. The value DEFAULT_EXPERIMENTAL_FEATURES_ENABLED is true for STP builds. So it’s correct that you don’t see any change.

Thanks for the explanation! :)

# Maciej Stachowiak (4 days ago)

There’s a number of mysterious timeouts with 96. Not sure if flakiness or real?

The new WebCrypto failures are surprising, but likely real and should be investigated: wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013, wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is:different&run_id=347530011&run_id=381930013

Many large new chunks of not-run tests are caused by a harness error or timeout. E.g. html/ and webauthn/

# Jiewen Tan (4 days ago)

On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak <mjs at apple.com> wrote:

There’s a number of mysterious timeouts with 96. Not sure if flakiness or real?

The new WebCrypto failures are surprising, but likely real and should be investigated: wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013, wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is:different&run_id=347530011&run_id=381930013

I believe these tests are flaky. I have made a PR to improve it a while ago. I should probably get those improvement landed sometime. web-platform-tests/wpt#6102, web-platform-tests/wpt#6102

Many large new chunks of not-run tests are caused by a harness error or timeout. E.g. html/ and webauthn/

STP 96 has enabled user agent UI for WebAuthn, which could cause crashes. This could be the reason why webauthn is 0 all the time.

Best, Jiewen

# Stephen Mcgruer (4 days ago)

There’s a number of mysterious timeouts with 96. Not sure if flakiness or real? Many large new chunks of not-run tests are caused by a harness error or timeout. E.g. html/ and webauthn

We have unfortunately struggled historically with getting reliable + consistent results for Safari, with spurious infrastructure + other problems. That said, Igalia has been working on improving reliability and have made good progress, and we have also invested in projects like the WebkitGTK runs to help with this problem (e.g. if a test fails in both Safari and WebKitGTK, there's a better chance it's a real failure[0])

We have had many Safari TP 96 runs since my announcement ( wpt.fyi/runs?label=master&max-count=100&product=safari); you can click on a single run (the icon) to view the run itself, or click on two runs to get a diff view - which can be useful for spotting flakes (as most SHAs are only changing a few tests, so differences between consecutive runs are usually flakes). There is also an explicit flake analyzer on our insights page - wpt.fyi/insights (it's a little clunky but can still be useful).

I believe these tests are flaky. I have made a PR to improve it a while

ago. I should probably get those improvement landed sometime.

Improvements would be great! Looking at that PR, it is quite old, so will need to be rebased and force pushed. Let me know if you need any help on that :).

STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.

This could be the reason why webauthn is 0 all the time.

It does look like there are still some harness failures in further STP 96 runs (e.g. wpt.fyi/results/webauthn?run_id=361070008); I see a mention in the failing tests of a "UnexpectedAlertOpenException: unexpected alert open (500):" - is that what you mean? Is there some flag we could be setting on STP, or something Safaridriver could be doing to avoid these?

[0]: wpt.fyi/results/?label=master&product=safari%5Bexperimental%5D&product=webkitgtk&product=chrome%5Bexperimental%5D&product=firefox%5Bexperimental%5D&aligned&q=safari%3Afail%20webkitgtk%3Afail%20chrome%3Apass%20firefox%3Apass is an example search for tests that fail in both Safari + WebKitGTK, but pass in both Chrome and Firefox.

# Philip Jägenstedt (3 days ago)

On Tue, Dec 3, 2019 at 3:05 AM Jiewen Tan <jiewen_tan at apple.com> wrote:

Hi Maciej,

On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak <mjs at apple.com> wrote:

There’s a number of mysterious timeouts with 96. Not sure if flakiness or real?

The new WebCrypto failures are surprising, but likely real and should be investigated: wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013, wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is:different&run_id=347530011&run_id=381930013

I believe these tests are flaky. I have made a PR to improve it a while ago. I should probably get those improvement landed sometime. web-platform-tests/wpt#6102

That sounds very doable. I added you to the reviewers team to make this easier and commented here: web-platform-tests/wpt#6102

Want more features?

Request early access to our private beta of readable email premium.