Tools/TestWebKitAPI

# Michael Catanzaro (2 days ago)

I started writing this mail to propose creating a separate ChangeLog for Tools/TestWebKitAPI, to make the Tools/ ChangeLog more focused on, you know, actual tools. So many of my changes under Tools/ are to the API tests.

But this reminded me that TestWebKitAPI is the only testsuite we have under Tools/. All the rest are toplevel directories:

JSTests/ LayoutTests/ ManualTests/ PerformanceTests/ WebDriverTests/ WebPlatformTests/

So it almost doesn't fit under Tools/ at all, right? Would there be any objections to moving it to a toplevel WebKitAPITests/ or just APITests/ directory? That would nicely parallel all our other tests.

(The practical objection is that moving anything is difficult for someone without familiarity with both XCode and CMake. I'd only be able to help with the CMake portion of the move. And then Apple internal build will inevitably break too, so someone would need to volunteer to care for that.)

If we don't want to move it, I guess a new ChangeLog file would be fine? :)

Michael

Contact us to advertise here
# Ross.Kirsling at sony.com (2 days ago)

APITests/ would seem like an improvement to me...although given that this would be the seventh *Tests/ directly, it also makes me wonder if those should all be moved under a top-level Tests/ directory? __

(Either way, I'd be happy to help with Xcode rearranging if needed, at least to the point of pleasing the external builds.)

Ross

On 2/5/19, 12:13 PM, "webkit-dev on behalf of Michael Catanzaro" <webkit-dev-bounces at lists.webkit.org on behalf of mcatanzaro at igalia.com> wrote:

Hi,

I started writing this mail to propose creating a separate ChangeLog 
for Tools/TestWebKitAPI, to make the Tools/ ChangeLog more focused on, 
you know, actual tools. So many of my changes under Tools/ are to the 
API tests.

But this reminded me that TestWebKitAPI is the only testsuite we have 
under Tools/. All the rest are toplevel directories:

JSTests/
LayoutTests/
ManualTests/
PerformanceTests/
WebDriverTests/
WebPlatformTests/

So it almost doesn't fit under Tools/ at all, right? Would there be any 
objections to moving it to a toplevel WebKitAPITests/ or just APITests/ 
directory? That would nicely parallel all our other tests.

(The practical objection is that moving anything is difficult for 
someone without familiarity with both XCode and CMake. I'd only be able 
to help with the CMake portion of the move. And then Apple internal 
build will inevitably break too, so someone would need to volunteer to 
care for that.)

If we don't want to move it, I guess a new ChangeLog file would be 
fine? :)

Michael

_______________________________________________
webkit-dev mailing list
webkit-dev at lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit
# Ryosuke Niwa (2 days ago)

Moving it to top-level APITests directory sounds good to me although I'm not volunteering to fix any Xcode or Apple internal build related issues.

  • R. Niwa
# Ross.Kirsling at sony.com (2 days ago)

That said, I would expect specifically the Tools/TestWebKitAPI/Tests subdirectory to be the thing moved to APITests/, since the code that actually runs the tests is as much a “tool” as DRT, WKTR, or various other test-running scripts under Tools/Scripts are.

Ross

From: Ryosuke Niwa <rniwa at webkit.org>

Date: Tuesday, February 5, 2019 at 1:44 PM To: "Kirsling, Ross" <Ross.Kirsling at sony.com>

Cc: Michael Catanzaro <mcatanzaro at igalia.com>, WebKit Development <webkit-dev at lists.webkit.org>

Subject: Re: [webkit-dev] Tools/TestWebKitAPI

Moving it to top-level APITests directory sounds good to me although I'm not volunteering to fix any Xcode or Apple internal build related issues.

  • R. Niwa

On Tue, Feb 5, 2019 at 1:07 PM <Ross.Kirsling at sony.com<mailto:Ross.Kirsling at sony.com>> wrote:

APITests/ would seem like an improvement to me...although given that this would be the seventh *Tests/ directly, it also makes me wonder if those should all be moved under a top-level Tests/ directory? __

(Either way, I'd be happy to help with Xcode rearranging if needed, at least to the point of pleasing the external builds.)

Ross

On 2/5/19, 12:13 PM, "webkit-dev on behalf of Michael Catanzaro" <webkit-dev-bounces at lists.webkit.org<mailto:webkit-dev-bounces at lists.webkit.org> on behalf of mcatanzaro at igalia.com<mailto:mcatanzaro at igalia.com>> wrote:

Hi,

I started writing this mail to propose creating a separate ChangeLog
for Tools/TestWebKitAPI, to make the Tools/ ChangeLog more focused on,
you know, actual tools. So many of my changes under Tools/ are to the
API tests.

But this reminded me that TestWebKitAPI is the only testsuite we have
under Tools/. All the rest are toplevel directories:

JSTests/
LayoutTests/
ManualTests/
PerformanceTests/
WebDriverTests/
WebPlatformTests/

So it almost doesn't fit under Tools/ at all, right? Would there be any
objections to moving it to a toplevel WebKitAPITests/ or just APITests/
directory? That would nicely parallel all our other tests.

(The practical objection is that moving anything is difficult for
someone without familiarity with both XCode and CMake. I'd only be able
to help with the CMake portion of the move. And then Apple internal
build will inevitably break too, so someone would need to volunteer to
care for that.)

If we don't want to move it, I guess a new ChangeLog file would be
fine? :)

Michael

_______________________________________________
webkit-dev mailing list
webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>
https://lists.webkit.org/mailman/listinfo/webkit
# Brian Burg (2 days ago)

I am in favor of a top-level Tests directory which would contain the various *Tests/ directories within it. Any relative directory paths (i.e., <script src=> in LayoutTests) should work without further changes.

If you post a patch that gets open source bits to work, I'd be willing to spend [some] time fixing Apple internal business. I can't guarantee it will work, but I would give it a try. I'm particularly wary of breaking bots that assume the location of, say, LayoutTests.

However, before we do that, perhaps it would be a useful intermediate step to follow Michael's OP and move Tools/TestWebKitAPI/ to APITests/. That alone will reveal a lot of bad assumptions and make later moves easier.

# Michael Catanzaro (2 days ago)

On Tue, Feb 5, 2019 at 4:01 PM, Ross.Kirsling at sony.com wrote:

That said, I would expect specifically the Tools/TestWebKitAPI/Tests subdirectory to be the thing moved to APITests/, since the code that actually runs the tests is as much a “tool” as DRT, WKTR, or various other test-running scripts under Tools/Scripts are.

I dunno, we could do it either way, but there's value in keeping related code together. E.g. our tests subclass lots of TestWebKitAPI classes, so we'd have code in APITests/ subclassing stuff under Tools/TestWebKitAPI/. It's much more tightly-coupled than, say, the layout tests are to TestController/WKTR, which communicates with layout tests over well-defined interfaces. In contrast, our API tests really depend on implementation details of TestWebKitAPI and they often have to be changed in tandem.

Michael

# Ryosuke Niwa (2 days ago)

On Tue, Feb 5, 2019 at 5:06 PM Michael Catanzaro <mcatanzaro at igalia.com>

wrote:

On Tue, Feb 5, 2019 at 4:01 PM, Ross.Kirsling at sony.com wrote:

That said, I would expect specifically the Tools/TestWebKitAPI/Tests subdirectory to be the thing moved to APITests/, since the code that actually runs the tests is as much a “tool” as DRT, WKTR, or various other test-running scripts under Tools/Scripts are.

I dunno, we could do it either way, but there's value in keeping related code together. E.g. our tests subclass lots of TestWebKitAPI classes, so we'd have code in APITests/ subclassing stuff under Tools/TestWebKitAPI/. It's much more tightly-coupled than, say, the layout tests are to TestController/WKTR, which communicates with layout tests over well-defined interfaces. In contrast, our API tests really depend on implementation details of TestWebKitAPI and they often have to be changed in tandem.

Yeah, I'd prefer keeping all the related code for TestWebKitAPI in one place. We have build scripts in Source too so there is a precedence for this.

We can probably push the harness code into APITests/harness or something.

  • R. Niwa

Want more features?

Request early access to our private beta of readable email premium.