PSA: Changes in the recommended GTK/WPE developer workflow

# Philippe Normand (5 days ago)

Until now the most recommended way to work on the GTK and WPE ports was to use a custom JHBuild[0] moduleset ([1][2]) for build/runtime dependencies. This setup was deployed almost 10 years ago [3] and while it is an opt-in approach, it is highly recommended, especially if you want to run layout tests locally.

If you ever executed Tools/Scripts/update-webkitgtk-libs or Tools/Scripts/update-webkitwpe-libs it means you currently have a DependenciesGTK and/or DependenciesWPE folder in WebKitBuild/ and that your workflow is JHBuild-based.

JHBuild has various shortcomings, even though it served us well all these years:

  • if the moduleset is modified, the bots remove the whole Dependencies{GTK,WPE} folders and rebuild everything from scratch
  • no cross-compilation support
  • no clear separation between the host system and the JHBuild-based sysroot
  • poor system requirements management (See the install-dependencies scripts in Tools/{wpe,gtk})
  • time lost rebuilding the dependencies (can take up to an hour on my build machine)

So in order to improve our lives a bit, we decided to try a new workflow, this time based on Flatpak[4]. Instead of having every developer locally build the dependencies, the built sysroot will be packaged in a Flatpak SDK/Runtime and downloaded to the developer machine. Currently the SDK is built for X86_64 but additional architectures can be supported (ARMv7, Aarch64 at least).

The advantages of this new approach:

  • cross-compilation can easily be achieved
  • clear separation between the host system and the SDK sysroot
  • the SDK runs in a sandbox
  • unified toolchain for WebKit build (currently both GCC 9.2.0 and clang 8.0.1)
  • no time lost rebuilding dependencies :)
  • consistent layout tests coverage across different hosts
  • separate build directories for each port (example: WebKitBuild/GTK/Release mounted as /app/WebKit/WebKitBuild/Release in the SDK sandbox)

As a bonus, this setup allows for:

  • integration with sccache, for faster WebKit builds
  • builtin IceCC support without additional manual steps (you still need a scheduler running on the host system though)

The only disadvantage of this new approach is that hacking on dependencies is currently not trivial to accomplish. We plan to enable this most likely via the Flapjack[5] tool.

Once the patch from bug#205658 [6] lands, this workflow will be available. How do I use it? Easy:

  1. Make sure your flatpak version is recent enough, 1.4.4 is the minimum version we support. Backports for Debian stable are available and for Ubuntu LTS, a PPA is available. See the Flathub instructions (though you don't need to manually add the flathub remote) [7].

  2. Run update-webkitgtk-libs or update-webkitwpe-libs as usual. This will download and install the SDK in WebKitBuild/UserFlatpak/

  3. Run build-webkit as usual

The SDK will be used when running the tests (API, layout, webkitpy, etc) and the MiniBrowser.

For the time being we keep the JHBuild workflow in the SVN, although if everything goes well, it will be removed in the coming months. Hence we encourage all GTK/WPE developers to try this new workflow!

If you still want to keep using JHBuild, make sure to set the WEBKIT_JHBUILD=1 environment variable in your shell. But keep in mind that we intend to remove support for JHBuild if Flatpak works as expected (so please try to test it, and report any issue with it). The GTK/WPE buildbots and EWS are currently running with JHBuild and will be switched one-by-one to Flatpak soon.

The SDK sources are currently hosted on Github[8], but we might move them to WebKit's SVN.

Philippe

[0] developer.gnome.org/jhbuild/stable [1] trac.webkit.org/browser/trunk/Tools/gtk/jhbuild.modules [2] trac.webkit.org/browser/trunk/Tools/wpe/jhbuild.modules [3] bugs.webkit.org/show_bug.cgi?id=73425 [4] flatpak.org [5] endlessm/flapjack [6] bugs.webkit.org/show_bug.cgi?id=205658 [7] flatpak.org/setup [8] igalia/webkit

Contact us to advertise here
# Philippe Normand (4 days ago)

On Tue, 2020-03-17 at 17:31 +0000, Philippe Normand wrote:

The only disadvantage of this new approach is that hacking on dependencies is currently not trivial to accomplish. We plan to enable this most likely via the Flapjack[5] tool.

I forgot to mention (quoting the ChangeLog):

As an additional commodity, the new environment supports the
GStreamer gst-build-based workflow. Just set the GST_BUILD_PATH
environment variable to your gst-build path. This feature was
contributed by Thibault Saunier.

Once the patch from bug#205658 [6] lands, this workflow will be available.

The patch landed in r258626.

Happy hacking! Philippe

Want more features?

Request early access to our private beta of readable email premium.