Extended attributes vs. new types

# Domenic Denicola (4 months ago)

(cc public-script-coord for future reference),

I'm looking at tackling the shared array buffer + Web IDL issues discussed in www.w3.org/Bugs/Public/show_bug.cgi?id=29388 and related threads, and would appreciate some extra help walking me through the problems with extended attributes and union types you allude to in bugzilla.mozilla.org/show_bug.cgi?id=1231687#c17 (and which have motivated things like USVString in the past). I know you explained it briefly there, but I'm still a little lost, and feel that I need to get a solid grip on the issue if I'm to be productive here. So a more extensive explanation would be really appreciated.

Here are some questions whose answers might help me:

What exact steps would go wrong in the spec if we used [AllowShared] Uint8Array, or [EnsureUTF16] DOMString, instead of MaybeSharedUint8Array/USVString? What are the problematic Web IDL snippets, JS inputs, and spec steps that we end up at?

Why is this different than things like [Clamp] or [EnforceRange]? The way that they insert themselves into the JS -> Web IDL conversion steps seems totally reasonable to me. Does [Clamp] also break down when trying to use it with union types?

What spec surgery would be involved in fixing this whole situation? I guess the idea would be to make the extended attributes apply to the types instead of the arguments, but how exactly would that help?

Contact us to advertise here
# Boris Zbarsky (4 months ago)

On 1/11/17 6:11 PM, Domenic Denicola wrote:

What exact steps would go wrong in the spec if we used [AllowShared] Uint8Array, or [EnsureUTF16] DOMString, instead of MaybeSharedUint8Array/USVString? What are the problematic Web IDL snippets, JS inputs, and spec steps that we end up at?

Consider the following IDL definitions:

dictionary Dict { USVString member1; sequence<USVString> member2; };

interface Iface { void foo(optional Dict arg); void bar((Iface or USVString) arg); void baz(sequence<USVString> arg); void something(record<DOMString, USVString> arg); attribute USVString attr; };

How would we handle these using "[EnsureUTF16] DOMString" instead? It seems to me like we'd need the following:

1) The ability to specify [EnsureUTF16] on dictionary members. 2) The ability to specify [EnsureUTF16] on attributes. 3) The ability to specify [EnsureUTF16] on operation attributes. 4) Steps in heycam.github.io/webidl/#create-sequence-from-iterable to propagate through the [EnsureUTF16] to the element conversions. 5) Steps in heycam.github.io/webidl/#es-to-record to propagate through the [EnsureUTF16] to conversions. For the special case of USVString there's the additional complication of the key and value maybe wanting different behavior, but that's not a problem for [AllowShared].

It's hard to say whether heycam.github.io/webidl/#es-to-union to union would need changes too. It would depend on exactly how the phrasing went for the [EnsureUTF16] thing.

If we add new parametrized types they would need to know about [EnsureUTF16] too. If we add more types parametrized on more than one type (a la record), things get pretty annoying.

This is all on the spec end. On the implementation end, you obviously have to explicitly thread the [EnsureUTF16] thing and any other such flags through sequence conversions, record conversions, and union conversions. It makes things somewhat more complicated.

Why is this different than things like [Clamp] or [EnforceRange]?

It's not.

[Clamp] and [EnforceRange] have (1), (2), (3) above. They don't have (4) or (5). If you want to pass a sequence of clamped integers, you're out of luck.

If you want to have a clamped integer in a union, you're out of luck too, because, for example, heycam.github.io/webidl/#Clamp says:

The [Clamp] extended attribute must not appear on a read only attribute, or an attribute, operation argument or dictionary member that is not of an integer type.

Does [Clamp] also break down when trying to use it with union types?

Well, it's not allowed to be specified on union types at the moment.

If it were allowed, then in terms of the spec I think it would be OK, since you could make the argument that the value being converted in heycam.github.io/webidl/#abstract-opdef-converttoint is the value that was passed for the union type, and hence "V is being passed as an operation argument annotated with the [Clamp] extended attribute" or one of the other two conditions in step 6 is true.

In terms of implementation, unions would need to know to pass through the "I was tagged with [Clamp] down to all their conversions". Note that you can't just build this into the union itself, because the same union can be used in both [Clamp] and non-[Clamp] situations, right? Consider:

typedef (long or USVString) MyUnion;

interface Iface { void foo([Clamp] MyUnion arg); void bar(MyUnion arg); };

A binding generator would presumably generate a single instance of MyUnion, which would have to have conditional behavior based on whether it's used in a [Clamp] or non-[Clamp] context. And also based on whether it's used in [EnforceRange]. And [EnsureUTF16], if that were a thing.

What spec surgery would be involved in fixing this whole situation?

I think the above more or less describes it?

I guess the idea would be to make the extended attributes apply to the types instead of the arguments, but how exactly would that help?

At that point, how is that really different from just having a different type? Unless you can combine the extended attributes, of course. But if they're mutually exclusive (and [Clamp] and [EnforceRange] are specified as mutually exclusive), there is no real semantic difference between that and a new type...

There can be the argument that you can apply [Clamp] to a bunch of different types, so you would get a sort of "type explosion" if you created separate types for all of those. So it might make sense to have [Clamp] that attaches directly to integer types, from that point of view. That would not have the "have to thread this information through" problem I describe above, of course, nor the "hey, we're parametrized over two types, which one should be clamped?" problem. Nor the "any time we add a new parametrized thing we have to teach it about all these frobs" problem.

I hope that helps; happy to try to clarify anything above that's still being confusing.

# Domenic Denicola (4 months ago)

Thanks! I think this does spell out the issue. Now let's turn to shared array buffers and see if I can apply my newfound knowledge.

My main concern is the type explosion issue. It's not the worst thing in the world to add 11 new "MaybeShared" types (9 typed arrays + DataView + ArrayBuffer) and two new typedefs (ArrayBufferView + BufferSource). But it feels pretty silly, and might lead to some repetitive specing and bindings generator code (or some new layer of abstraction to avoid the repetition). And if we end up wanting to capture the by-reference vs. by-copy semantics in the type system, we're in real trouble; now we're up to 44 types and 8 typedefs for the various permutations.

Anne's suggestion ( bugzilla.mozilla.org/show_bug.cgi?id=1231687#c18 ) is to couple MaybeShared and by-reference, and use prose for the remaining cases. Maybe that's the way to go. But it sure would be nice if we could just get away with a few extended attributes... which brings us to the last part of the message.

I guess the idea would be to make the extended attributes apply to the types instead of the arguments, but how exactly would that help?

At that point, how is that really different from just having a different type? Unless you can combine the extended attributes, of course. But if they're mutually exclusive (and [Clamp] and [EnforceRange] are specified as mutually exclusive), there is no real semantic difference between that and a new type...

There can be the argument that you can apply [Clamp] to a bunch of different types, so you would get a sort of "type explosion" if you created separate types for all of those. So it might make sense to have [Clamp] that attaches directly to integer types, from that point of view. That would not have the "have to thread this information through" problem I describe above, of course, nor the "hey, we're parametrized over two types, which one should be clamped?" problem. Nor the "any time we add a new parametrized thing we have to teach it about all these frobs" problem.

One of the big things your message drove home is how extended attributes apply to syntactic productions like dictionary members, attributes, methods, interfaces, etc. and not to types. I guess I knew this intellectually, but my intuition wasn't correctly aligned.

It does seem more intuitive for certain "behaviors", like clamping/range enforcement, and maybe the null-treatment behaviors, to be part of the type. (Corresponding to the current extended attributes [Clamp], [EnforceRange], [TreatNonObjectAsNull], and [TreatNullAs].) In general whenever an extended attribute's processing model shows up entirely inside the ES -> Web IDL type conversion section, often with preludes like "is being assigned to an attribute... is passed as an operation argument... is being used as a dictionary member...", being part of the type would make more sense.

I think it would be nice to introduce this concept into the spec, and retrofit it onto those extended attributes, but I'm not sure it's worth the trouble for implementers.

It also comes with questions like---should we invent a new concept and syntax, separate from extended attributes? Or should we say that certain extended attributes apply exclusively to types, just like certain of them apply to attributes/interfaces/etc. I guess that latter option would introduce a dependency from the parser to the semantics; to interpret void f([X] long foo) you'd be unsure whether [X] is associated with the parameter or the type until you knew if X was one of the type-only extended attributes. (Unless we banned extended attributes on parameters entirely? Are only these type-applicable extended attributes currently used on parameters?)

Thoughts? Mainly on the "is it worth it" question, I suppose.

# Boris Zbarsky (4 months ago)

On 1/12/17 2:23 PM, Domenic Denicola wrote:

And if we end up wanting to capture the by-reference vs. by-copy semantics in the type system

Right, at that point we're sort of off into multiple orthogonal annotations: copy/reference semantics and allow/disallow sharing...

One of the big things your message drove home is how extended attributes apply to syntactic productions like dictionary members, attributes, methods, interfaces, etc. and not to types.

Hmm. That's a good point.

In particular, the obvious corollary is that we can't just redefine Type, syntactically, to include extended attributes, because then this:

void foo([SomeAttr] int arg);

would be syntactically ambiguous.

But what we could do is have a TypeWithAttributes which is defined as "ExtendedAttributeList Type" We could use TypeWithAttributes in almost all places where an ES-to-IDL conversion would be involved, except where a grammar production that can be preceded by ExtendedAttributeList can start with Type. Looking through the grammar really quickly, the things that involve ES-to-IDL conversions and can start with Type would be OptionalOrRequiredArgument and DictionaryMember. For those we could specify that certain extended attributes specified on the argument or dictionary member get propagated to its type instead. In other places we'd use TypeWithAttributes instead of Type.

Note that in looking through this stuff just now I found two places that would need "threading through" that I had missed initially: setlike<V>

and maplike<K,V> where you want to specify whether V can be a

shared/clamped/whatever thing or not...

Anyway, there would need to be a bit of careful handling of cases like this:

typedef [Clamp] long ClampedLong;

void foo([EnforceRange] ClampedLong arg);

which should error out, because fundamentally both the [Clamp] and the [EnforceRange] are being stuck on the "long", right?

There will be this weird inconsistency where you write:

dictionary Dict { [Clamp] required long member; };

and

interface Iface { void foo([Clamp] optional long arg); };

vs

interface Iface { attribute [Clamp] long attr; };

but maybe we can rejigger dictionary required member grammar and operation optional arg grammar so the case when the keyword ("required" or "optional") is present is more separate (and followed by TypeWithAttributes) and the case when the keyword is absent starts with Type and propagates extended attrs. Then this:

interface Iface { void foo([Clamp] long arg); };

would quack very similarly to this:

interface Iface { void foo(optional [Clamp] long arg); };

even if the under-the-hood reasons it worked were different.

I think it would be nice to introduce this concept into the spec, and retrofit it onto those extended attributes, but I'm not sure it's worth the trouble for implementers.

I think the ability to apply the existing extended attrs to things like union members, maplike/setlike values, etc, is worth a certain amount of trouble. I don't think it will be that much trouble in practice if we do it this way. Definitely less trouble than threading things through conversion code.

It also comes with questions like---should we invent a new concept and syntax, separate from extended attributes? Or should we say that certain extended attributes apply exclusively to types, just like certain of them apply to attributes/interfaces/etc. I guess that latter option would introduce a dependency from the parser to the semantics; to interpret void f([X] long foo) you'd be unsure whether [X] is associated with the parameter or the type until you knew if X was one of the type-only extended attributes.

In terms of the parser it would have to be associated with the argument, of course. But the argument could propagate it through to its type per above suggestion.

(Unless we banned extended attributes on parameters entirely? Are only these type-applicable extended attributes currently used on parameters?)

The extended attributes that currently apply to operation arguments are: [Clamp], [EnforceRange], [TreatNullAs]. The first two of those also apply to dictionary members, but [TreatNullAs] does not (though there's no obvious reason it can't).

Thoughts? Mainly on the "is it worth it" question, I suppose.

I suspect it probably is...

# mostafa shahdadi (4 months ago)

Sent from my iPad

# Domenic Denicola (4 months ago)

I've started working on this. One question in the nitty-gritty so far...

But what we could do is have a TypeWithAttributes which is defined as "ExtendedAttributeList Type" We could use TypeWithAttributes in almost all places where an ES-to-IDL conversion would be involved, except where a grammar production that can be preceded by ExtendedAttributeList can start with Type. Looking through the grammar really quickly, the things that involve ES-to-IDL conversions and can start with Type would be OptionalOrRequiredArgument and DictionaryMember. For those we could specify that certain extended attributes specified on the argument or dictionary member get propagated to its type instead. In other places we'd use TypeWithAttributes instead of Type.

It seems the simplest thing to do is to just remove the ExtendedAttributesList that precedes such productions. You'd still be able to annotate the types with extended attributes, but not the arguments or dictionary members themselves.

As you noted below only [Clamp], [EnforceRange], and [TreatNullAs] apply to arguments or dictionary members. So as long as we allow those on the Type production, we can remove ExtendedAttributeList on arguments and dictionary members, I believe.

# Boris Zbarsky (4 months ago)

On 1/25/17 5:20 PM, Domenic Denicola wrote:

It seems the simplest thing to do is to just remove the ExtendedAttributesList that precedes such productions.

That could break some IDL consumers, no? (I mean, I know for a fact it would break Gecko.)

As you noted below only [Clamp], [EnforceRange], and [TreatNullAs] apply to arguments or dictionary members.

Of the extended attributes defined in the IDL spec. But extended attributes are allowed to be defined by other specs, and are supposed to serve as a general extension point for cases when IDL needs to express implementation-specific things that there is no existing IDL syntax for.

Just looking at Gecko's IDL, we support [TreatNonCallableAsNull] on operation arguments to simplify the implementation of some legacy interfaces. We also support all of our conditional-exposure extended attributes (things like "pretend this doesn't exist if some pref is not set", "pretend this does not exist unless we're in a sufficiently privileged context", etc) on dictionary members. I would not be terribly surprised if at some point we want to support [SecureContext] on dictionary members, for similar reasons.

Want more features?

Request early access to our private beta of readable email premium.