Unifying testsuite policy and getting rid of CSS exceptions

# Geoffrey Sneddon (20 days ago)

(BCC'd to www-style, to keep people in the loop, but can we keep all replies on the one list?)

There was some discussion in the CSS WG a fortnight ago (minutes: www.w3.org/mid/CADhPm3t6F0RGzPsz=sG4JoCyPk6XEX+FJ+kzHyjHjkF_Q_H4tw@mail.gmail.com under the "Test Metadata" heading) about where new test suites should go.

When the CSS testsuite was merged into wpt, we did it with the entire old repository just being put in css/, and adding a few extra lints (all of these to keep the build system working for the sake of keeping test.csswg.org/harness working, a requirement of the CSS WG being okay with the merge; these are a unique filename constraint, a requirement for <link rel=help> links, support files being in a

support directory, though these don't suffice for all the requirements the build system).

At that time, we had a few CSS test suites at the top level of WPT: cssom, cssom-view, and a few Houdini specs. Shortly after the merge, some people (including, I believe, the editor) decided to merge the entire cssom and cssom-view test suites into the top level directories, rather than having them split across multiple places. Following this, the code that runs test.csswg.org/harness was updated to build the CSS testsuite from the root of wpt, instead of just the css/ directory (though the lints only apply to css/).

Beyond the extra lints, there's one other major difference: the directories in css/ are versioned (i.e., the directories map to the versioned short names), which has typically been an anti-goal given implementors rarely look at anything except the latest ED. However, given the general policy of not needlessly moving tests, we've mostly avoided doing that, so we now have some level 4 tests in level 3 folders. Note the CSS build system currently only cares about <link rel=help> links, so directory names are irrelevant.

The question is where we go from here, and there are three sides to this.

  1. What we do with existing tests:

a. We keep them where they are now.

b. We move all CSS WG spec test suites into css/.

c. We move all CSS WG spec test suites to the top-level.

  1. Do we remove the versioning from future CSS WG test suites:

a. Yes.

b. No.

  1. Where do new test suites go:

a. At the top level.

b. In css/.

c. Up to the editor of the individual spec.

  1. Do we move existing tests to match the above?

a. Yes.

b. No.

  1. Where do we run the extra CSS lints:

a. Only in css/.

b. In all directories containing tests for CSS WG specs.

As far as I'm aware, the status-quo is 1a, 2a, 3b, 4a. The CSS WG's preference is 1b, 2b, 3c, and I assume 4b (given without it the CSS WG's tooling is likely to get pretty broken pretty fast).

There had been some talk about dropping the requirement for <link rel=help>, but this requires versioned folders (as you need to require

somewhere has the version) for the build system's metadata requirements, and it would rely on folders within the spec mapping to anchors within the spec (and anchors aren't stable across versions, because specs get refactored).

Does anyone have any preferences?

Contact us to advertise here
# Chris Lilley (20 days ago)

<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>515fcf60ca3e2cda30c9a587a0e9433cbcf8bb958ecb586bdce814c2061d400810301d1d35521cb53d91e5ee7084ea5b331dd8655a3204f174a34ea74c92f9bd/original.md</Key><RequestId>78E0014E70992767</RequestId><HostId>eDin40LccVfUfUKNwORe7CjHlbpDH+57YFWZK8Ob02g4BRhZ62nZo2tY+e8mxw5vx2ZEDMvC/d8=</HostId></Error>

# Geoffrey Sneddon (20 days ago)

On 14/09/17 19:23, Geoffrey Sneddon wrote:

The question is where we go from here, and there are three sides to this.

  1. What we do with existing tests:

a. We keep them where they are now.

b. We move all CSS WG spec test suites into css/.

c. We move all CSS WG spec test suites to the top-level.

  1. Do we remove the versioning from future CSS WG test suites:

a. Yes.

b. No.

  1. Where do new test suites go:

a. At the top level.

b. In css/.

c. Up to the editor of the individual spec.

  1. Do we move existing tests to match the above?

a. Yes.

b. No.

  1. Where do we run the extra CSS lints:

a. Only in css/.

b. In all directories containing tests for CSS WG specs.

As far as I'm aware, the status-quo is 1a, 2a, 3b, 4a. The CSS WG's preference is 1b, 2b, 3c, and I assume 4b (given without it the CSS WG's tooling is likely to get pretty broken pretty fast).

Well, I screwed up here.

The survey meant to end up as:

  1. Do we put new test suites for CSS WG specs in versioned directories:

a. Yes

b. No

  1. Where do new test suites for CSS WG specs go:

a. At the top level

b. In css/

c. Either at the top level or in css/, with no preference between either except for being consistently in one

d. Up to the editor of the individual spec as to whether they want to use the CSS WG tooling

  1. Do we move existing tests to match the above:

a. Yes

b. No

  1. Where do we run the extra CSS-only lints:

a. Only in css/

b. In all directories containing tests for CSS WG specs

c. Nowhere (and almost certainly break the CSS WG's tooling)

And the CSS WG's preference is 1: b, 2: b, 3: a, 4: a/b (they're identical given previous answers).

I believe the preference of the WPT admins is 1: b, 2: a, 3: a, 4: c (this is essentially "remove all special-casing for CSS testsuites").

# Geoffrey Sneddon (20 days ago)

On 15/09/17 00:46, Geoffrey Sneddon wrote:

On 14/09/17 19:23, Geoffrey Sneddon wrote:

The question is where we go from here, and there are three sides to this.

  1. What we do with existing tests:

a. We keep them where they are now.

b. We move all CSS WG spec test suites into css/.

c. We move all CSS WG spec test suites to the top-level.

  1. Do we remove the versioning from future CSS WG test suites:

a. Yes.

b. No.

  1. Where do new test suites go:

a. At the top level.

b. In css/.

c. Up to the editor of the individual spec.

  1. Do we move existing tests to match the above?

a. Yes.

b. No.

  1. Where do we run the extra CSS lints:

a. Only in css/.

b. In all directories containing tests for CSS WG specs.

As far as I'm aware, the status-quo is 1a, 2a, 3b, 4a. The CSS WG's preference is 1b, 2b, 3c, and I assume 4b (given without it the CSS WG's tooling is likely to get pretty broken pretty fast).

Well, I screwed up here.

The survey meant to end up as:

  1. Do we put new test suites for CSS WG specs in versioned directories:

a. Yes

b. No

  1. Where do new test suites for CSS WG specs go:

a. At the top level

b. In css/

c. Either at the top level or in css/, with no preference between either except for being consistently in one

d. Up to the editor of the individual spec as to whether they want to use the CSS WG tooling

  1. Do we move existing tests to match the above:

a. Yes

b. No

  1. Where do we run the extra CSS-only lints:

a. Only in css/

b. In all directories containing tests for CSS WG specs

c. Nowhere (and almost certainly break the CSS WG's tooling)

And the CSS WG's preference is 1: b, 2: b, 3: a, 4: a/b (they're identical given previous answers).

I believe the preference of the WPT admins is 1: b, 2: a, 3: a, 4: c (this is essentially "remove all special-casing for CSS testsuites").

And given we probably want to garner responses without a million emails: docs.google.com/forms/d/1KlaK6h0lq4KcoC4_uNMzjXSIqBdmS5lwkbxykNKn9zM/edit

(I realise sending an extra email for this is causing an extra million emails, sorry!)

# fantasai (18 days ago)

On 09/14/2017 02:23 PM, Geoffrey Sneddon wrote: >

  1. Do we put new test suites for CSS WG specs in versioned directories: a. Yes b. No

On the one hand, if a feature moves across levels then either the test has to move or its location isn't quite right anymore. (From the CSSWG tooling perspective, it doesn't matter where the test is, but it's nice when things match.)

On the other, modules split (and occasionally merge) as well, which creates the same problem, and that's more likely to affect existing tests if we use unlevelled directories.

Also, if there is a desire to have per-section sub-directories, that works better with levelled directories since spec reorganization of a later spec won't affect test-spec matching in an earlier one.

  1. Where do new test suites for CSS WG specs go: a. At the top level b. In css/ c. Either at the top level or in css/, with no preference between either except for being consistently in one d. Up to the editor of the individual spec as to whether they want to use the CSS WG tooling

My preference would be for keeping them all under the css/ subdirectory.

This makes it easier to find all the CSS tests since they're together, which is useful for things like wpt.fyi.

It also means that the modularization of the CSS specs won't be (literally) doubling the number of top-level directories in WPT, which probably makes it easier for anyone who isn't looking for CSS tests to navigate the repo.

  1. Do we move existing tests to match the above: a. Yes b. No

Yes.

Having inconsistent (time-dependent, author-dependent, etc.) locations means that tests belonging to a single spec are split across multiple directories, making it harder to find the tests one is looking for, or to check for existing tests before adding new ones in order to avoid duplicate work. Furthermore there is no clearly correct way forward for someone adding new tests, which makes that process more confusing than it needs to be imho.

  1. Where do we run the extra CSS-only lints: a. Only in css/ b. In all directories containing tests for CSS WG specs c. Nowhere (and almost certainly break the CSS WG's tooling)

All CSSWG specs, unless someone is volunteering to compile implementation reports for all of our specs for the next 10 years by some other method. The CSSWG does not consider "we have 500 tests for module X, that should be enough" to be a sufficiently comprehensive method for evaluating test coverage.

~fantasai

# fantasai (18 days ago)

On 09/14/2017 07:50 PM, Geoffrey Sneddon wrote:

And given we probably want to garner responses without a million emails: docs.google.com/forms/d/1KlaK6h0lq4KcoC4_uNMzjXSIqBdmS5lwkbxykNKn9zM/edit

Surveys like that are useful if you want to run a popularity contest. Usually at W3C we make decisions by evaluating the characteristics of each proposal, not by voting on them without discussion of their merits.

~fantasai

# Geoffrey Sneddon (a day ago)

On Thu, Sep 14, 2017 at 7:23 PM, Geoffrey Sneddon me@gsnedders.com wrote:

(BCC'd to www-style, to keep people in the loop, but can we keep all replies on the one list?)

There was some discussion in the CSS WG a fortnight ago (minutes: www.w3.org/mid/CADhPm3t6F0RGzPsz=sG4JoCyPk6XEX+FJ+kzHyjHjkF_Q_H4tw@mail.gmail.com under the "Test Metadata" heading) about where new test suites should go.

When the CSS testsuite was merged into wpt, we did it with the entire old repository just being put in css/, and adding a few extra lints (all of these to keep the build system working for the sake of keeping test.csswg.org/harness working, a requirement of the CSS WG being okay with the merge; these are a unique filename constraint, a requirement for <link rel=help> links, support files being in a support directory, though these don't suffice for all the requirements the build system).

At that time, we had a few CSS test suites at the top level of WPT: cssom, cssom-view, and a few Houdini specs. Shortly after the merge, some people (including, I believe, the editor) decided to merge the entire cssom and cssom-view test suites into the top level directories, rather than having them split across multiple places. Following this, the code that runs test.csswg.org/harness was updated to build the CSS testsuite from the root of wpt, instead of just the css/ directory (though the lints only apply to css/).

Beyond the extra lints, there's one other major difference: the directories in css/ are versioned (i.e., the directories map to the versioned short names), which has typically been an anti-goal given implementors rarely look at anything except the latest ED. However, given the general policy of not needlessly moving tests, we've mostly avoided doing that, so we now have some level 4 tests in level 3 folders. Note the CSS build system currently only cares about <link rel=help> links, so directory names are irrelevant.

The question is where we go from here, and there are three sides to this.

  1. What we do with existing tests:

a. We keep them where they are now.

b. We move all CSS WG spec test suites into css/.

c. We move all CSS WG spec test suites to the top-level.

  1. Do we remove the versioning from future CSS WG test suites:

a. Yes.

b. No.

  1. Where do new test suites go:

a. At the top level.

b. In css/.

c. Up to the editor of the individual spec.

  1. Do we move existing tests to match the above?

a. Yes.

b. No.

  1. Where do we run the extra CSS lints:

a. Only in css/.

b. In all directories containing tests for CSS WG specs.

As far as I'm aware, the status-quo is 1a, 2a, 3b, 4a. The CSS WG's preference is 1b, 2b, 3c, and I assume 4b (given without it the CSS WG's tooling is likely to get pretty broken pretty fast).

There had been some talk about dropping the requirement for <link rel=help>, but this requires versioned folders (as you need to require somewhere has the version) for the build system's metadata requirements, and it would rely on folders within the spec mapping to anchors within the spec (and anchors aren't stable across versions, because specs get refactored).

Does anyone have any preferences?

/Geoffrey

Hello CSS WG!

The conclusion, from the discussion on public-test-infra and others around WPT, is that from the WPT project there is a preference for:

  • Unversioned directories, probably under css/ to make the lint rules simpler and easier to understand for contributors.
  • Keep requiring <link rel=help> for CSS WG stuff.
  • Require versioned spec links.
  • Don't require links to specific spec sections, but encourage them (but don't block on it indefinitely!).

Peter agreed on IRC to update the build system and test harness so that tests in no section and in unknown sections still appear (the status quo is that they don't appear anywhere in any of the CSS WG's tooling, which is awkward when test anchors change!).

Is the WG happy with this?

# Florian Rivoal (a day ago)

On Oct 4, 2017, at 0:25, Geoffrey Sneddon me@gsnedders.com wrote:

Hello CSS WG!

The conclusion, from the discussion on public-test-infra and others around WPT, is that from the WPT project there is a preference for:

  • Unversioned directories, probably under css/ to make the lint rules simpler and easier to understand for contributors.
  • Keep requiring <link rel=help> for CSS WG stuff.
  • Require versioned spec links.
  • Don't require links to specific spec sections, but encourage them (but don't block on it indefinitely!).

Peter agreed on IRC to update the build system and test harness so that tests in no section and in unknown sections still appear (the status quo is that they don't appear anywhere in any of the CSS WG's tooling, which is awkward when test anchors change!).

Is the WG happy with this?

I am.

I also hope the same requirement about rel=help links will be generalized to the rest of wpt, but that's not a blocking condition to do the above.

Also, I believe that while these rules should be applied going forward, we may need to grant a one time exception: when a browser vendor first starts syncing their existing tests with ours, they may have a bunch of legacy tests which do not have rel=help links. I don't think it is practical to require relabeling of these before starting the synchronization (although allowing links without a specific section should make it easier). If they can be labelled, great, but as long as the vendor starts requiring rel=help links for all new tests going forward, I'd be OK with allowing existing tests to be imported as is.

I think there are three other things we may want to synchronize on (but again, these don't have to block the above):

1) When a test is manual, we currently have multiple ways of declaring it so:

  • <meta name="flags" content="interact">

  • <meta name="flags" content="animated">

  • <meta name="flags" content="userstyle">

  • <meta name="flags" content="font">

  • -manual suffix after the main filename but before the extension

The different flags imply different reasons for a test being manual, but they all boil down to the test having to be run manually. I do not think the distinction in reason actually serves a useful purpose. We should continue documenting what these means for historical reasons, but I propose that going forward, the CSSWG aligns with the rest of wpt, and adopts the -manual suffix after the filename to identify newly written manual tests (i.e. tests other than reftests or testharness.js tests).

2) Should the CSS tests and the rest of WPT converge on the use of <meta name="flags">

I suggest that we do, and that the only flags that are worth bothering about are "may" and "should". CSSWG test should stop using the rest, and WPT should start using these two.

Not that the rest is entirely meaningless, but given how little benefit we derive from them, they're not worth perpetuating the "the csswg has excessive and annoying metadata requirements that nobody can be bothered to remember" meme for.

3) What about test assertions?

web-platform-tests.org/writing-tests/css-metadata.html#test-assertions

They currently are described as optional, and that should stay that way. There should be an expectation that the test reviewer will push back on a test if isn't clear what the test is testing, but <meta name=assert> is only one way to do that. Comments or explicit assertions in tests that use JS are equally fine.

This should not be specific to the CSSWG, and so the test-assertions section should be moved from the CSSWG documentation into the general WPT documentation, identifying <meta name=assert> as one of the options available to make a test self documenting.


Hopefully, if we get all that in place, we can stop having distinct requirements for csswg tests and the rest of wpt. As long as we do not lose essential information (and I think the above proposals does not), I think it is worth eliminating the mental overhead of separate requirements.

—Florian

Want more features?

Request early access to our private beta of readable email premium.