Proposal: adopt a "test required" policy for spec changes

# Daniel Veditz (20 days ago)

The W3 leadership has been emphasizing the importance of ensuring interoperability through Web Platform Tests. Some groups have adopted a policy of requiring corresponding web-platform-tests pull requests for before landing normative spec changes. Since interoperability is part of getting a spec to become a Recommendation this makes sense especially for specs that are in or entering CR. Should we adopt such a policy?

The Web Performance group adopted the following: [[ ALL normative spec changes are generally expected to have a corresponding pull request in web-platforms-tests, either in the form of new tests or modifications to existing tests, or must include the rationale for why test updates are not required for the proposed update. [...] ]] w3c/web-per <goog_1391446905>

formance/blob/gh-pages/CONTRIB <goog_1391446905>UTING.md

... and the CSS Working Group adopted one last week: [[ For normative changes for any specification in CR or later as well as the pre-CR specifications listed below, a corresponding web-platform-tests PR must be provided, except if testing is not practical; for other specifications it is usually appreciated. Typically, both PRs will be merged at the same time. Note that a test change that contradicts the spec should not be merged before the corresponding spec change. [...] ]] w3c/csswg-drafts/blob/master/CONTRIBUTING.md

Good idea? Objections? Respond on list and we can talk about it on our next call (Sept 20).

-Dan Veditz

Contact us to advertise here
# Mike West (18 days ago)

I'm in favor of adopting a policy along the lines of Web Performance's. Requiring tests for changes to a document in CR, in particular, seems like a no-brainer, since we ought to have enough implementation experience by that point to make writing tests against an implementation trivial.

I'm a little more skeptical about a test-first approach to designing a feature in the first place, though, as I see some marginal risk of locking ourselves into a bad design just due to inertia and sunk cost. Chrome's incubation-first strategy seems like a reasonable way of mitigating this from a working group perspective (e.g. we'd only adopt documents (and therefore apply a test-first policy to them)) once we were reasonably confident that their general shape was itself reasonable.

+Philip who's been thinking about this a lot, both from Chrome's perspective, and from the perspective of the WHATWG (where such a policy is already solidly in place [https://whatwg.org/working-mode](https://whatwg.org/working-mode))).

# Philip Jägenstedt (2 days ago)

Quite excited about this, is this blocked on having a meeting?

# Mike West (16 hours ago)

Hrm. We rushed the call last week (my fault!), and ended up skipping over this entirely.

I'm still in favor of adopting a policy along the lines of Web Performance's, I'm still a little bit concerned about velocity, but I'm still assuaged that an incubation-first strategy should reduce the costs as most of the work we're doing in this WG should be polishing as opposed to making radical changes in the direction of an unproven design.

Wendy, would adopting such a policy require any changes to formal W3C docs/charter/etc, or would adding a CONTRIBUTING file be something we could decide to do without asking permission? :)

-mike

On Wed, Sep 27, 2017 at 1:49 PM, Philip Jägenstedt foolip@google.com

wrote:

# Wendy Seltzer (13 hours ago)

On 09/28/2017 06:45 AM, Mike West wrote:

Hrm. We rushed the call last week (my fault!), and ended up skipping over this entirely.

I'm still in favor of adopting a policy along the lines of Web Performance's, I'm still a little bit concerned about velocity, but I'm still assuaged that an incubation-first strategy should reduce the costs as most of the work we're doing in this WG should be polishing as opposed to making radical changes in the direction of an unproven design.

Wendy, would adopting such a policy require any changes to formal W3C docs/charter/etc, or would adding a CONTRIBUTING file be something we could decide to do without asking permission? :)

Procedurally, a group decision is sufficient. Dan, when you think the group has consensus, go ahead!

Want more features?

Request early access to our private beta of readable email premium.