Moving to Python 3

# Jonathan Bedard (6 days ago)

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

  1. Make any no-cost Python 3 compatibility changes, in particular

    • print foo -> print(foo)
    • import .foo -> import webkitpy.foo
  2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results)

  3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like:

    • dict.iteritems() -> dict.items()
    • dict.items() -> list(dict.items())
  4. Install Python 3 on macOS Sierra and Mojave bots

  5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch)
  6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems():
    ...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language.

There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3.

Jonathan

Contact us to advertise here
# Tim Horton (6 days ago)

On Jul 12, 2019, at 12:18 PM, Jonathan Bedard <jbedard at apple.com> wrote:

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

  1. Make any no-cost Python 3 compatibility changes, in particular
    • print foo -> print(foo)
    • import .foo -> import webkitpy.foo
  2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results)
  3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like:
    • dict.iteritems() -> dict.items()
    • dict.items() -> list(dict.items())
  4. Install Python 3 on macOS Sierra and Mojave bots
  5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch)
  6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems():
    ...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to this, actually??

legacy.python.org/dev/peps/pep-0469

See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me.

# Robert Ma (6 days ago)

Any thoughts on bytes and Unicode strings, especially the string literals in the code base?

# Don.Olmstead at sony.com (6 days ago)

Mentioned this on the other thread but here’s it again.

python-modernize.readthedocs.io/en/latest and python-future.org/automatic_conversion.html might be of use considering the sheer amount of code. They’re both mentioned in docs.python.org/3/howto/pyporting.html for migration.

You can run individual rules one by one over the code. That might be ok rather than running all the rules at once. Especially for review. Modernize seemed to work pretty well but it wasn’t smart enough to detect shebang lines.

From: webkit-dev <webkit-dev-bounces+don.olmstead=am.sony.com at lists.webkit.org> On Behalf Of Tim Horton

Sent: Friday, July 12, 2019 12:38 PM To: Jonathan Bedard <jbedard at apple.com>

Cc: webkit-dev at lists.webkit.org Subject: Re: [webkit-dev] Moving to Python 3

On Jul 12, 2019, at 12:18 PM, Jonathan Bedard <jbedard at apple.com<mailto:jbedard at apple.com>> wrote:

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

  1. Make any no-cost Python 3 compatibility changes, in particular

    • print foo -> print(foo)
    • import .foo -> import webkitpy.foo
  2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results)

  3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like:

    • dict.iteritems() -> dict.items()
    • dict.items() -> list(dict.items())
  4. Install Python 3 on macOS Sierra and Mojave bots

  5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch)
  6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems():
    ...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to this, actually??

legacy.python.org/dev/peps/pep-0469

See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me.

There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3.

Jonathan

# Michael Catanzaro (6 days ago)

On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard <jbedard at apple.com>

wrote:

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

Is python2 support required for a well-motivated transitional purpose?

I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already.

Then million-dollar question is: what shebangs will we use for our scripts? Will #!/usr/bin/env python3 work for Apple?

Michael

# Jonathan Bedard (6 days ago)

On Jul 12, 2019, at 12:45 PM, Robert Ma <robertma at chromium.org> wrote:

Any thoughts on bytes and Unicode strings, especially the string literals in the code base?

My experience with this has been you mostly have to pay attention to where your code interfaces with other processes. In webkitpy, I suspect that most of the our work here will probably be where we are calling other commands or reading files.

Because we call most commands through either the Executive class or the FileSystem class, I think there is a good chance we can make changes here that effect most of the codebase.

Jonathan

# Adrian Perez de Castro (6 days ago)

Hello,

On Fri, 12 Jul 2019 12:37:43 -0700, Tim Horton <timothy_horton at apple.com> wrote:

On Jul 12, 2019, at 12:18 PM, Jonathan Bedard <jbedard at apple.com> wrote:

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the new Mac developer tools come with Python 3. As a result, we need to continue the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

Given that GNU/Linux distributions have started already a while ago switching it is great that MacOS is now following suit as well \o/

We have a bug already for this: bugs.webkit.org/show_bug.cgi?id=184986

If one has to make code compatible with both Python 2.7 and 3.x, my advice would be to use the Six module, which provides ready-made helpers which behave consistently across both versions: six.readthedocs.io

I propose that, over the next 9 months, we do the following:

  1. Make any no-cost Python 3 compatibility changes, in particular
    • print foo -> print(foo)
    • import .foo -> import webkitpy.foo
  2. Convert any scripts not used in automation to Python 3 ASAP (scripts like bisect-builds, block-spammers, compare-results)
  3. Make most Python 3 compatibility changes which sacrifice efficiency, subject to a case-by-case audit. These would be things like:
    • dict.iteritems() -> dict.items()
    • dict.items() -> list(dict.items())
  4. Install Python 3 on macOS Sierra and Mojave bots
  5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like clean-webkit, merge-results-json, webkit-patch)
  6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

for expectation_string, expectation_enum in test_expectations.TestExpectations.EXPECTATIONS.iteritems():
    ...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In Python 2.7, iteritems() gives us an iterator instead of creating a new list, like items() would. In Python 3, iteritems() doesn’t exist, but items() does, and now gives us an iterator instead of creating a new list. The trouble here is that, in this case, creating a new list will be very expensive, expensive enough that we might manage to impact the testing run. There isn’t really an elegant way around this problem if we want to support both Python 2.7 and Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to this, actually??

legacy.python.org/dev/peps/pep-0469

See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me.

Instead of rolling our own, I would rather use Six (mentioned above). It covers all the differences with the different “.iter*()” methods:

six.readthedocs.io/#six.iterkeys, six.readthedocs.io/#six.itervalues, six.readthedocs.io/#six.iteritems, six.readthedocs.io/#six.iterlists ...

There are other small gotchas as well. For example, ‘%’ is no longer a protected character, which can actually change the behavior of regexes. That’s why I think it’s better to just try and directly convert things instead of attempting to be compatible with both Python 2.7 and Python 3.

In my experience some of the major troubles making a codebase compatible with both Pythons are the string types:

  • In Python 2, “str” is also usable for binary data (basically it's backed by a char[], while “unicode” is actual text which may contain any code point.
  • In Python 3, “string” is kind of the old “unicode” (textual data), and “bytes” is kind of “string” but only holds binary data and in general cannot be used where a string would have been used.

The Six module helps a bit with the text types (see “six.text_type” and “six.binary_type”.

# Jonathan Bedard (6 days ago)

On Jul 12, 2019, at 12:49 PM, Michael Catanzaro <mcatanzaro at igalia.com> wrote:

On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard <jbedard at apple.com> wrote:

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

Is python2 support required for a well-motivated transitional purpose?

I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already.

We still need to figure out whether (and for how long) we intend to retain Python 2 support. Over the last few months, opinions on this have changed quite a bit, so I’m trying to determine what our path forward is going to be.

In my opinion, a few months after Catalina GMs, we no longer need to maintain Python 2 support, assuming that we provide adequate automation for installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about shebangs.

Then million-dollar question is: what shebangs will we use for our scripts? Will #!/usr/bin/env python3 work for Apple?

If we move straight to Python 3, we would need to use the Python 3 shebang. Catalina has both Python 2.7 (name ‘python’) and Python 3 (named ‘python3’). I think that this is what we would want to do because it allows us to pretty explicitly convert scripts one at a time.

# Jonathan Bedard (6 days ago)

On Jul 12, 2019, at 1:07 PM, Keith Rollin <krollin at apple.com> wrote:

On Jul 12, 2019, at 13:37, Tim Horton <timothy_horton at apple.com> wrote:

See "Migrating to the common subset of Python 2 and 3” — you define different iteritems() helpers in the two cases. Seems pretty reasonable to me.

I did something like this in webkit-triage. Search for “viewitems” for the compatibility function I used, as well as for “PY3” to see some of the other thunks I put into place.

— Keith

Actually, hadn’t seen those tricks yet, thanks Tim and Keith for calling them out!

Those would allow us to keep scripts both Python 2.7 and Python 3 compatible for longer.

Jonathan

# Ryosuke Niwa (6 days ago)

On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard <jbedard at apple.com> wrote:

On Jul 12, 2019, at 12:49 PM, Michael Catanzaro <mcatanzaro at igalia.com> wrote:

On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard <jbedard at apple.com> wrote:

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

Is python2 support required for a well-motivated transitional purpose?

I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already.

We still need to figure out whether (and for how long) we intend to retain Python 2 support. Over the last few months, opinions on this have changed quite a bit, so I’m trying to determine what our path forward is going to be.

In my opinion, a few months after Catalina GMs, we no longer need to maintain Python 2 support, assuming that we provide adequate automation for installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about shebangs.

I don't think it's acceptable to require installation of Python 3 just to build & run tests on WebKit unless the installation itself is automated and compartmentalized to WebKit's development given how bad Python is with respect to having multiple versions of Python's and managing packages between them.

I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble.

  • R. Niwa
# Michael Catanzaro (6 days ago)

On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble.

I understand it would be a hassle. :/ But please consider it relative to the additional difficulty of rewriting all of webkitpy to support multiple versions of python at the same time, or setting up a wrapper layer of bash scripts around all of our python scripts to enter a virtualenv before executing the real script.

Michael

# Ryosuke Niwa (6 days ago)

On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro <mcatanzaro at igalia.com>

wrote:

On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble.

I understand it would be a hassle. :/ But please consider it relative to the additional difficulty of rewriting all of webkitpy to support multiple versions of python at the same time, or setting up a wrapper layer of bash scripts around all of our python scripts to enter a virtualenv before executing the real script.

Yeah, and it sounds strictly better that the trouble is handled by people who maintain Python code who presumably more about Python than a random WebKit contributor who may not know how to setup virtual environment in Python, etc...

Again, the argument that the difficulty can be overcome and it's a minor inconvenience isn't convincing. I can, for example, suggest that we should just build WebKit using the latest version of clang. Anyone who uses a system that doesn't come with the latest release of clang should just build clang instead. There is a significant cost in having to support MSVC and GC++ so we should just use clang everywhere and only the latest version like Chromium does.

Each team & person has a different preference & perspective when it comes to tooling. Please don't break someone else's working workflow based on your preference.

  • R. Niwa
# Maciej Stachowiak (6 days ago)

On Jul 12, 2019, at 3:23 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:



On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard <jbedard at apple.com> wrote:

On Jul 12, 2019, at 12:49 PM, Michael Catanzaro <mcatanzaro at igalia.com> wrote:

On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard <jbedard at apple.com> wrote:

The trouble I foresee us encountering with any scheme which attempts a conversion which retains both Python 2.7 and Python 3 compatibility is code like this:

Is python2 support required for a well-motivated transitional purpose?

I had previously proposed making all our scripts work with both python2 and python3 only because I thought Apple was going to require python2 indefinitely. Now that you're interested in this transition, there's probably no need to continue python2 support. Anyone building WebKit on older versions of macOS can reasonably be expected to manually install python3, right? And it's clear that you're prepared to do this for infrastructure/bots already.

We still need to figure out whether (and for how long) we intend to retain Python 2 support. Over the last few months, opinions on this have changed quite a bit, so I’m trying to determine what our path forward is going to be.

In my opinion, a few months after Catalina GMs, we no longer need to maintain Python 2 support, assuming that we provide adequate automation for installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about shebangs.

I don't think it's acceptable to require installation of Python 3 just to build & run tests on WebKit unless the installation itself is automated and compartmentalized to WebKit's development given how bad Python is with respect to having multiple versions of Python's and managing packages between them.

I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble.

It’s possible install a python without installing it on the system, and install modules and any needed additions, using virtualenv: virtualenv.pypa.io/en/stable

This is the pro way to use python without caring about what happens to be on the system.

I suggest we proceed by gradually converting our scripts to use a Python 3 virtualenv. This will also spare us the need to install Python modules onto the system. The tricky part will be that webkitpy would have to work both ways until conversion is complete (or at the extreme I guess we could fork it).

# Jonathan Bedard (6 days ago)

I would agree that if we move to Python 3, we need a script which installs Python 3 in an impossible to mess-up way on Mojave and High Sierra.

I don’t think the clang comparison is fair here. Python 2 is officially deprecated in 2020, we can’t expect security updates to the language or any libraries we depend on 6 months from now. It’s not really a question of if we stop supporting Python 2, but rather, when we stop supporting Python 2.

It’s also worth noting that in Catalina, we will need some script to install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer included by default in the new Xcode. Given this, it doesn’t seem terrible if the script responsible for installing XCode CL tools also installs Python 3 for older Mac versions.

Jonathan

# Ryosuke Niwa (5 days ago)

On Fri, Jul 12, 2019 at 11:22 PM Jonathan Bedard <jbedard at apple.com> wrote:

I would agree that if we move to Python 3, we need a script which installs Python 3 in an impossible to mess-up way on Mojave and High Sierra.

I don’t think the clang comparison is fair here. Python 2 is officially deprecated in 2020, we can’t expect security updates to the language or any libraries we depend on 6 months from now. It’s not really a question of if we stop supporting Python 2, but rather, when we stop supporting Python 2.

I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system.

It’s also worth noting that in Catalina, we will need some script to install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer included by default in the new Xcode. Given this, it doesn’t seem terrible if the script responsible for installing XCode CL tools also installs Python 3 for older Mac versions.

Yes, as long as it doesn’t replace or break existing Python2.7 and/or other python scripts in the system.

On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:



On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro <mcatanzaro at igalia.com> wrote:

On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

I frequently do WebKit development in older versions of macOS to diagnose old OS specific regressions, and having to install Python 3 each time I install an old OS is too much of a trouble.

I understand it would be a hassle. :/ But please consider it relative to the additional difficulty of rewriting all of webkitpy to support multiple versions of python at the same time, or setting up a wrapper layer of bash scripts around all of our python scripts to enter a virtualenv before executing the real script.

Yeah, and it sounds strictly better that the trouble is handled by people who maintain Python code who presumably more about Python than a random WebKit contributor who may not know how to setup virtual environment in Python, etc...

Again, the argument that the difficulty can be overcome and it's a minor inconvenience isn't convincing. I can, for example, suggest that we should just build WebKit using the latest version of clang. Anyone who uses a system that doesn't come with the latest release of clang should just build clang instead. There is a significant cost in having to support MSVC and GC++ so we should just use clang everywhere and only the latest version like Chromium does.

Each team & person has a different preference & perspective when it comes to tooling. Please don't break someone else's working workflow based on your preference.

  • R. Niwa

--

  • R. Niwa
# Maciej Stachowiak (5 days ago)

On Jul 13, 2019, at 3:44 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:



On Fri, Jul 12, 2019 at 11:22 PM Jonathan Bedard <jbedard at apple.com> wrote: I would agree that if we move to Python 3, we need a script which installs Python 3 in an impossible to mess-up way on Mojave and High Sierra.

I don’t think the clang comparison is fair here. Python 2 is officially deprecated in 2020, we can’t expect security updates to the language or any libraries we depend on 6 months from now. It’s not really a question of if we stop supporting Python 2, but rather, when we stop supporting Python 2.

I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system.

It’s also worth noting that in Catalina, we will need some script to install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer included by default in the new Xcode. Given this, it doesn’t seem terrible if the script responsible for installing XCode CL tools also installs Python 3 for older Mac versions.

Yes, as long as it doesn’t replace or break existing Python2.7 and/or other python scripts in the system.

This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3.

# Michael Catanzaro (5 days ago)

On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak <mjs at apple.com>

wrote:

This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3.

I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code.

Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK?

Michael

# Ryosuke Niwa (5 days ago)

On Sat, Jul 13, 2019 at 4:14 PM Michael Catanzaro <mcatanzaro at igalia.com>

wrote:

On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak <mjs at apple.com> wrote:

This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3.

I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code.

Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK?

That seems okay given macOS / iOS ports don’t use CMake right now.

Virtually all ports that use CMake as the build system would have python3 in the base installation or already require installations of a bunch of software (e.g. Windows port), right?

  • R. Niwa
# Maciej Stachowiak (5 days ago)

On Jul 13, 2019, at 4:14 PM, Michael Catanzaro <mcatanzaro at igalia.com> wrote:

On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak <mjs at apple.com> wrote:

This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3.

I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code.

Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK?

Can you clarify why this is needed?

# Maciej Stachowiak (5 days ago)

On Jul 13, 2019, at 7:26 PM, Maciej Stachowiak <mjs at apple.com> wrote:

On Jul 13, 2019, at 4:14 PM, Michael Catanzaro <mcatanzaro at igalia.com> wrote:

On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak <mjs at apple.com> wrote:

This is exactly what virtualenvs can do. And this is how we should do it IMO, even for systems that natively have some version of Python 3.

I guess that's fine for everything not required by the CMake build, e.g. build-webkit and webkitpy. That's probably 99% of our python code.

Scripts required by the CMake build should use system python3 though, not the virtualenv. I guess that's 1% or less of our python. OK?

Can you clarify why this is needed?

(Also which specific Python scripts are needed by the CMake build?)

# Adrien Destugues (5 days ago)

I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system.

Python3 would normally install a "python3" executable that's independant from the "python" one from Python2? It's fairly common to have these side by side precisely because they are not compatible.

# Michael Catanzaro (4 days ago)

On Sat, Jul 13, 2019 at 9:26 PM, Maciej Stachowiak <mjs at apple.com>

wrote:

Can you clarify why this is needed?

Well it just wouldn't seem very kosher to use a virtualenv for the serious work of performing real distro builds, right? In contrast to developer scripts for developer convenience, where I'd say it's perfectly fine to do whatever we want. But official builds should surely be performed using system dependencies only.

Anyway, it doesn't seem like a problem. From searching for 'python' in my build.ninja, I find:

JSC:

ud_itab.py generateWasmOpsHeader.py generateWasmValidateInlinesHeader.py generateWasmB3IRGeneratorInlinesHeader.py create_regex_tables generateYarrUnicodePropertyTables.py generateIntlCanonicalizeLanguage.py KeywordLookupGenerator.py generate-inspector-protocol-bindings.py generate-js-builtins.py generateYarrCanonicalizeUnicode generate-combined-inspector-json.py jsmin.py cssmin.py make-js-file-arrays.py

WebCore:

create-html-entity-table create-http-header-name-table makeSelectorPseudoClassAndCompatibilityElementMap.py makeSelectorPseudoElementsMap.py

WebKit:

generate-message-receiver.py generate-messages-header.py

WebInspectorUI:

copy-user-interface-resources.pl (perl script that runs some python)

Tools:

generate-inspector-gresource-manifest.py

It's possible I've missed some, but that's probably most of them. Basically all the scripts under Source/ -- the scripts that are really required to build -- and that one platform-specific exception under Tools/, shouldn't require a virtualenv IMO. That doesn't seem too difficult to ensure. We could either have them use the virtualenv only optionally if it's somehow already available (I'm not familiar with how it works) or only if the scripts detect that webkitpy exists, or just make this subset of scripts compatible with both python2 and python3 for the next couple years.

In contrast, the vast majority of our python scripts are not required to build. They live under Tools/Scripts, are only used by developers, and are not included in release tarballs at all. That includes all of webkitpy, anything used by build-webkit, anything used by run-webkit-tests, etc. I'd say we can go crazy here. You get a virtualenv, you get a virtualenv, everybody gets a virtualenv!

Michael

# Fujii Hironori (3 days ago)

On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa <rniwa at webkit.org> wrote:

I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system.

Just out of curiosity. As far as I know, installing Python 3 breaks nothing. What and why are they got broken?

# Guillaume Emont (2 days ago)

Quoting Fujii Hironori (2019-07-16 08:04:09)

On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa <rniwa at webkit.org> wrote:

I don’t think anyone is arguing that we’d eventually need to move to
Python3. I’m arguing that it’s not okay to require random WebKit
contributor to know some obscure python insanity to install Python 3, or
have a script that installs Python 3 and breaks all other python scripts in
the system.

Just out of curiosity. As far as I know, installing Python 3 breaks nothing. What and why are they got broken?

I suspect Ryosuke is talking about a case where python 3 has already been installed on the OS (but is not part of the original OS), and we install python 3 also, and the scripts that were using the first python 3 installed end up using WebKit's python 3 instead, which could lack a python module required by these scripts, hence breaking them.

I think this situation should be easy to avoid with a virtualenv though.

Best

# Konstantin Tokarev (2 days ago)

16.07.2019, 14:33, "Guillaume Emont" <guijemont at igalia.com>:

Quoting Fujii Hironori (2019-07-16 08:04:09)

On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa <rniwa at webkit.org> wrote:

 I don’t think anyone is arguing that we’d eventually need to move to
 Python3. I’m arguing that it’s not okay to require random WebKit
 contributor to know some obscure python insanity to install Python 3, or
 have a script that installs Python 3 and breaks all other python scripts in
 the system.

Just out of curiosity. As far as I know, installing Python 3 breaks nothing. What and why are they got broken?

I suspect Ryosuke is talking about a case where python 3 has already been installed on the OS (but is not part of the original OS), and we install python 3 also, and the scripts that were using the first python 3 installed end up using WebKit's python 3 instead, which could lack a python module required by these scripts, hence breaking them.

But we have autoinstaller which handles this situation.

# Alexey Proskuryakov (2 days ago)

15 июля 2019 г., в 23:04, Fujii Hironori <fujii.hironori at gmail.com> написал(а):

On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa <rniwa at webkit.org <mailto:rniwa at webkit.org>> wrote:

I don’t think anyone is arguing that we’d eventually need to move to Python3. I’m arguing that it’s not okay to require random WebKit contributor to know some obscure python insanity to install Python 3, or have a script that installs Python 3 and breaks all other python scripts in the system.

Just out of curiosity. As far as I know, installing Python 3 breaks nothing. What and why are they got broken?

As always, it will be up to the people doing the work to decide how it's done. The feedback is clear:

  • They shouldn't make it excessively difficult to do WebKit engineering on older versions of macOS.

"Excessively" is not clearly defined, but it seems obvious that there is a tradeoff between tools work difficulty, and difficulty for engineers who go back to older macOS versions (and implicitly, difficulty for people who set up bots, QA engineers, and others involved).

I don't think that anyone ever suggested that it will be up to each engineer to figure out the best way to install Python 3.

  • virtualenv works great for some projects.

Definitely worth looking at it as one of the primary paths forward. It's not just Python itself, but we don't want to pollute the system with modules either. Although the latter already has a pretty nice solution in WebKit with autoinstall.

  • virtualenv shouldn't be required when building WebKit, and dynamic dependencies in general are not OK in this scenario.
# Darin Adler (2 days ago)

On Jul 16, 2019, at 12:46 PM, Alexey Proskuryakov <ap at webkit.org> wrote:

  • They shouldn't make it excessively difficult to do WebKit engineering on older versions of macOS.

"Excessively" is not clearly defined, but it seems obvious that there is a tradeoff between tools work difficulty, and difficulty for engineers who go back to older macOS versions (and implicitly, difficulty for people who set up bots, QA engineers, and others involved).

I don't think that anyone ever suggested that it will be up to each engineer to figure out the best way to install Python 3.

Lets keep in mind our strategy to keep development of WebKit on macOS easy. We want to preserve this. The steps (assuming git) are:

webkit.org/build-tools • download and install Xcode from Apple % xcode-select --install

webkit.org/getting-the-code % git clone git://git.webkit.org/WebKit.git WebKit % Tools/Scripts/webkit-patch setup-git-clone

webkit.org/building-webkit % build-webkit --debug

webkit.org/running-webkit % run-safari

We’d really like to keep it to a small number of steps. Having to download and install anything else would be a significant additional step.

— Darin

Want more features?

Request early access to our private beta of readable email premium.