EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)

# Aakash Jain (4 days ago)

Many people didn't like the noise by the EWS comments, and we removed the comments as per previous discussion in: lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html.

I agree with your point that having some kind of notification might be useful.

I proposed some ideas in lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html, but didn't get much feedback. If we can all agree on a solution, I can look into implementing it.

Thanks Aakash

Contact us to advertise here
# Maciej Stachowiak (4 days ago)

How about only a single comment when the first failure occurs? (Or else when all bots pass, if there is never a failure.)

This should help the author, the reviewer, and anyone else cc’d, without being too spammy.

# Aakash Jain (4 days ago)

Sounds good. I prefer the single comment when the first failure occur. That way notification would be sent as soon as the first failure happens.

I'll implement that (assuming it's acceptable to everyone).

Thanks Aakash

# Alexey Proskuryakov (3 days ago)

My preference is still e-mailing the patch author directly (possibly, also having an option to opt in for anyone). Bugzilla comments will always be irrelevant for most people CC'ed on the bug, and they are almost always undesirable to keep within the discussion flow.

# Maciej Stachowiak (2 days ago)

I think they are useful to actual and potential reviewers. Direct email to the patch author is not something anyone can Cc themselves on, and is not archived, so seems like a strictly worse form of communication.

# Alexey Proskuryakov (a day ago)

Can you elaborate on that, how exactly is e-mailing on first failure useful to reviewers?

Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree with this feedback. So I think that comments are generally undesirable.

Since I don't understand what your precise scenario is, I may be make straw man arguments below, but here are some things that I think make the proposed behavior unhelpful (add a comment on first failure, or when all EWSes pass).

  1. EWS comments in Bugzilla are so annoying that some people take the radical step of manually hiding them. EWS history is archived anyway, there is no need to look into comments for it.

  2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a slight annoyance, discouraging further participation in the project.

  3. I believe that for most reviewers, the mode of operation is one of the two: (1) do it when pinged directly, or (2) go over the review queue when one has the time. Getting EWS comments helps neither.

  4. Commenting when all EWSes pass is not very practical - it's too often that we have some stragglers that take days (or forever). I don't think that we can make it reliable even if we start actively policing EWS responsiveness.

  5. The reviewer likely wants to know the state of multiple EWSes if they are going to wait for EWS at all. What exactly are they going to do after getting an e-mail that one EWS failed?

  6. More bugmail delays response, especially for active project members who are CC'ed on a lot of bugs. I personally started reading bugmail more frequently now, knowing that there is more signal and less noise.

I can see the usefulness in the somewhat unusual case of a super urgent patch. We may want multiple people to watch it, so that members of CC list would go and ask the patch author to update it with more urgency than e-mail allows for. I think that opt-in is a better mechanism for that, so that people who opted in would receive information about each EWS data point.

# Ryosuke Niwa (a day ago)

On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov <ap at webkit.org> wrote:

Can you elaborate on that, how exactly is e-mailing on first failure useful to reviewers?

Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree with this feedback. So I think that comments are generally undesirable.

Since I don't understand what your precise scenario is, I may be make straw man arguments below, but here are some things that I think make the proposed behavior unhelpful (add a comment on first failure, or when all EWSes pass).

  1. EWS comments in Bugzilla are so annoying that some people take the radical step of manually hiding them. EWS history is archived anyway, there is no need to look into comments for it.

  2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a slight annoyance, discouraging further participation in the project.

  3. I believe that for most reviewers, the mode of operation is one of the two: (1) do it when pinged directly, or (2) go over the review queue when one has the time. Getting EWS comments helps neither.

  4. Commenting when all EWSes pass is not very practical - it's too often that we have some stragglers that take days (or forever). I don't think that we can make it reliable even if we start actively policing EWS responsiveness.

  5. The reviewer likely wants to know the state of multiple EWSes if they are going to wait for EWS at all. What exactly are they going to do after getting an e-mail that one EWS failed?

I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, a bug mail will stay in my inbox as one of items to get to.

I can see the usefulness in the somewhat unusual case of a super urgent

patch. We may want multiple people to watch it, so that members of CC list would go and ask the patch author to update it with more urgency than e-mail allows for. I think that opt-in is a better mechanism for that, so that people who opted in would receive information about each EWS data point.

I think there is a value in knowing that a patch isn't ready instead of having to open the bug to realize that.

  • R. Niwa
# Alexey Proskuryakov (7 hours ago)

4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa <rniwa at webkit.org> написал(а):

On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov <ap at webkit.org <mailto:ap at webkit.org>> wrote:

Can you elaborate on that, how exactly is e-mailing on first failure useful to reviewers?

Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree with this feedback. So I think that comments are generally undesirable.

Since I don't understand what your precise scenario is, I may be make straw man arguments below, but here are some things that I think make the proposed behavior unhelpful (add a comment on first failure, or when all EWSes pass).

  1. EWS comments in Bugzilla are so annoying that some people take the radical step of manually hiding them. EWS history is archived anyway, there is no need to look into comments for it.

  2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a slight annoyance, discouraging further participation in the project.

  3. I believe that for most reviewers, the mode of operation is one of the two: (1) do it when pinged directly, or (2) go over the review queue when one has the time. Getting EWS comments helps neither.

  4. Commenting when all EWSes pass is not very practical - it's too often that we have some stragglers that take days (or forever). I don't think that we can make it reliable even if we start actively policing EWS responsiveness.

  5. The reviewer likely wants to know the state of multiple EWSes if they are going to wait for EWS at all. What exactly are they going to do after getting an e-mail that one EWS failed?

I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, a bug mail will stay in my inbox as one of items to get to.

I can see the usefulness in the somewhat unusual case of a super urgent patch. We may want multiple people to watch it, so that members of CC list would go and ask the patch author to update it with more urgency than e-mail allows for. I think that opt-in is a better mechanism for that, so that people who opted in would receive information about each EWS data point.

I think there is a value in knowing that a patch isn't ready instead of having to open the bug to realize that.

So just to clarify,

  • a major part of how you get to review bugs is by being CC'ed, and you review them when you have the time to read bugmail;
  • and you don't open the bug in Bugzilla if there is already an EWS failure by the time you read the e-mail where review is requested?

That's clearly a valid benefit. In my mind, it probably doesn't outweigh the downsides. On the other hand, yours is a voice of someone who reviews way more patches than Maciej and me combined these days, so maybe more e-mail is an overall benefit to many of the reviewers.

# Devin Rousso (6 hours ago)

I may be alone here, but I actually found build/test(s) failure emails to be super useful. The part that I did not find useful was that each time a build/test(s) would fail on a particular platform, there would be two comments/emails, which furthermore split the useful information into two pieces. IIRC, the first comment/email would contain information about what build/test(s) failed, and the second comment/email was just an archive upload of the layout test results (if applicable).

I would find it very useful to get an email each time one of the bots had something go "wrong", and ideally include a link directly to the 'errors' log that Aakash mention in the beginning of this thread (or to the layout test results in the case of a test(s) failure). Right now, if something goes "wrong", I have to be the one to discover it and then click through a few pages until I find the data that will actually help me solve the problem.

As a patch author, this is useful in letting me not have to remember to check my patch once a day in order to see if there are any red bubbles (something which I've forgotten to do, leaving a broken patch up for review thinking all was fine). I can upload a patch and ideally leave it be, as I will be notified if the build/test(s) fails.

As a patch reviewer, I'm not sure this would be as useful, as I have a saved query that I use for finding all patches that need to be reviewed, and on a search result page there are no EWS bubbles anyways, so I can't see the info there. I have to click into the bug regardless. To Ryosuke's point, it may still be useful to get an email as a reviewer simply so I know that something failed and can potentially remember that fact before I click on a bug in the search results page, but I know personally that my mind doesn't really work that way :P

Thanks, Devin

Want more features?

Request early access to our private beta of readable email premium.