Python 3 support

Bug #1785119 reported by Thomas Pointhuber on 2018-08-02
36
This bug affects 8 people
Affects Status Importance Assigned to Milestone
KiCad
Undecided
Thomas Pointhuber

Bug Description

Python 2 will be deprecated in about 1 Year and 4 Months (https://pythonclock.org). It would be a good idea to add Python 3 support and drop Python 2 (or at least print a warning if anyone uses it).

According to this blog post SIP and SWIG can coexist: https://kicad.mmccoo.com/2017/11/23/learnings-from-moving-kicad-to-wxpython-4-0/

How are the current plans?

Related branches

This have to be well planned because will also affect the wxPython now used.
As a python plugin developer I already start to be concerned about and test plugin and pythons scripts that interact with KiCad in both Py2 and Py3 (respectively wxPython3 and wxPython4).
Also wxPython 4 use gtk3 (and wxPython3 use gtk2) witch may affect more parts of KiCad for a good migration.

Nick Østergaard (nickoe) wrote :

I am not sure why Hildo is that concerned in #1. I don't see anything that interact with kicad with python3. I don't see what the issue is with gtk2 vs gtk3, as we want to migrate to gtk3 on linux.

I think the biggest hurdle is to see a proff of a patch working with phoenix.

Changed in kicad:
assignee: nobody → Thomas Pointhuber (pointhi)
Thomas Pointhuber (pointhi) wrote :

I started with a python3 branch: https://github.com/pointhi/kicad-source-mirror/commits/python3

For now, it successfully creates a python module, but I'm not able to import it:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/pcbnew.py", line 38, in <module>
    import _pcbnew
ImportError: dynamic module does not define module export function (PyInit__pcbnew)
```

Any ideas?

Wayne Stambaugh (stambaughw) wrote :

Please keep in mind that KiCad has to build on all three major platforms without building dependencies from source. This is the criteria for merging changes with new dependencies. This wont be an issue on linux as I already see phoenix packages in Debian testing. The problem child is going to be windows. There is currently no phoenix package for the msys2 platform and may be as difficult as getting phoenix to build and run on linux. I have already tried building phoenix from source on msys (mingw32/mingw64) with no success so there is definitely some work to there. I think there is plenty of time because AFAIK, phoenix still isn't widely available on most linux distros yet.

Thomas Pointhuber (pointhi) wrote :

It seems wxPython is available for Windows as well: https://pypi.org/project/wxPython/#files

There is some discussion going on about that in the forum: https://forum.kicad.info/t/moving-from-python-to-lisp-lua/11691/48

I don't think there is plenty of time. With the current release cycle, python 2 will be unmaintained before the next major release. The implications could even be that python 2 will no longer be available in the package manager because security/platform fixes are no longer delivered. I highly recommend switching to Python 3 with the KiCad 5.1 release to prevent future issues, as well as to make the switch sooner than later. People will have to port their scripts anyway at some point.

I assume we cannot use Python2+3 inside the GUI concurrent due to symbol issues (like with GTK2+GTK3), but at least when used as CLI script support for both versions should be possible.

Changed in kicad:
status: New → In Progress

On 8/4/2018 5:02 AM, Thomas Pointhuber wrote:
> It seems wxPython is available for Windows as well:
> https://pypi.org/project/wxPython/#files

None of these windows packages are build with gcc and mingw which is a
non-starter for developers and our package builder. wxPython 4 has to
build on mingw32 and mingw64 using the msys2 shell. This is not going
to be a straight forward build because the wxPython build scripts always
assume windows builds use msvc. I haven't spent much time trying to
build wxpython 4 on msys2 but I can tell you that there will be some
work involved.

>
> There is some discussion going on about that in the forum:
> https://forum.kicad.info/t/moving-from-python-to-lisp-lua/11691/48

I have no intention of moving from Python to any other scripting
language. If someone wants to spend the time to implement this I'm fine
with merging it as a alternate scripting language as long as it comes
with a developer to support it.

>
> I don't think there is plenty of time. With the current release cycle,
> python 2 will be unmaintained before the next major release. The
> implications could even be that python 2 will no longer be available in
> the package manager because security/platform fixes are no longer
> delivered. I highly recommend switching to Python 3 with the KiCad 5.1
> release to prevent future issues, as well as to make the switch sooner
> than later. People will have to port their scripts anyway at some point.

If all of the platforms are supported and it is stable then I see no
issue in releasing python3 support in 5.1. However, the current
manpower focus is resolving the gtk2/gtk3 issues so we can even enable
wxpython support so that has to happen first.

>
> I assume we cannot use Python2+3 inside the GUI concurrent due to symbol
> issues (like with GTK2+GTK3), but at least when used as CLI script
> support for both versions should be possible.
>
> ** Changed in: kicad
> Status: New => In Progress
>

Thomas Pointhuber (pointhi) wrote :

I think the current state of my branch (https://github.com/pointhi/kicad-source-mirror/commits/python3) is good enough to do more widespread testings. Using the pcbnew module inside python3 scripts as well as opening simple action scripts is working on Linux. Windows and Mac is not tested so far.

My plan would be to do some basic tests on win/mac and then merge the initial support. I added a cmake flag which enables python3, so it is purely optional at this state. There are some changes to python scripts, but all changes are made in a way to result in the same behavior in both Python versions.

wxPython is another thing. I will try to look into it after initial python3 support is added.

@Wayne: the forum entry started with other scripting languages, but later changed to a pure python discussion (I linked because of the later part). There is also a good idea for the Python 3 roadmap:
* get Python 3 support into KiCad before 5.1, but stay on Python 2 when building
* enable Python 3 as default in the nightlies after the 5.1 release.

Wayne Stambaugh (stambaughw) wrote :

On 8/13/2018 12:27 PM, Thomas Pointhuber wrote:
> I think the current state of my branch (https://github.com/pointhi
> /kicad-source-mirror/commits/python3) is good enough to do more
> widespread testings. Using the pcbnew module inside python3 scripts as
> well as opening simple action scripts is working on Linux. Windows and
> Mac is not tested so far.
>
> My plan would be to do some basic tests on win/mac and then merge the
> initial support. I added a cmake flag which enables python3, so it is
> purely optional at this state. There are some changes to python scripts,
> but all changes are made in a way to result in the same behavior in both
> Python versions.
>
> wxPython is another thing. I will try to look into it after initial
> python3 support is added.
>
> @Wayne: the forum entry started with other scripting languages, but later changed to a pure python discussion (I linked because of the later part). There is also a good idea for the Python 3 roadmap:
> * get Python 3 support into KiCad before 5.1, but stay on Python 2 when building
> * enable Python 3 as default in the nightlies after the 5.1 release.
>

@Thomas, this is an issue. We already have users complaining about the
lack of wxPython support on certain platforms so disabling wxPython just
for python 3 support on platforms where this is supported is not going
to go over well. If we want to provide a separate python 3 with no
wxpython nightly builds for users then I'm fine with that (assuming our
package devs are on board) but throwing wxpython users under the bus is
not a going to go over well.

Nick Østergaard (nickoe) wrote :

@Wayne, I think you misread the discussion here.

The patch has a conditional to enable python3 support. So getting it merged early just helps packagers test and migrate completely away from python2. Many linux platforms can not provide wxpython now.

It has been verified on windows and linux, but lack testing on macos. This is the only reason that I have not pushed to get this merged yet.

On top of that Thomas also have a branch with wxphoenix WORKING, meaning that we don't really loose the wxpython feature. But I don't think the python3 with wxpython (phoenix) has been tested that much, but it looks like some minor commits.

Wayne Stambaugh (stambaughw) wrote :

On 8/23/2018 12:28 PM, Nick Østergaard wrote:
> @Wayne, I think you misread the discussion here.
>
> The patch has a conditional to enable python3 support. So getting it
> merged early just helps packagers test and migrate completely away from
> python2. Many linux platforms can not provide wxpython now.

Thomas was suggesting switching nightly builds to python 3 which means
no wxpython even on platforms where this still works (osx and windows).
 All I was suggesting is that we keep nightly builds as is on platforms
where wxpython with python 2 still works. It doesn't matter for
platforms where wxpython doesn't work so we can provide packages for
both configurations for users rather then pull the plug on wxpython2.

>
> It has been verified on windows and linux, but lack testing on macos.
> This is the only reason that I have not pushed to get this merged yet.
>
> On top of that Thomas also have a branch with wxphoenix WORKING, meaning
> that we don't really loose the wxpython feature. But I don't think the
> python3 with wxpython (phoenix) has been tested that much, but it looks
> like some minor commits.
>

Has anyone been able get phoenix to build on msys2? I haven't tried it
in a while. This is critical to be able to support phoenix on windows.

Nick Østergaard (nickoe) wrote :

Well, yes he suggested that, but only after the 5.1 release. And as I see it that is still a good chunk of time in the future. So merging the code for optional python3 support seems like a good idea to me.

Wayne Stambaugh (stambaughw) wrote :

I'm fine with merging the source changes once it's ready to go.

On 8/23/2018 2:06 PM, Nick Østergaard wrote:
> Well, yes he suggested that, but only after the 5.1 release. And as I
> see it that is still a good chunk of time in the future. So merging the
> code for optional python3 support seems like a good idea to me.
>

Thomas Pointhuber (pointhi) wrote :

I propose to get Python3 support merged, which can be found on my branch: https://github.com/pointhi/kicad-source-mirror/tree/python3

Reasons:

* no known python2/3 regressions on Linux as well as Windows builds (still waiting on mac)
* Possibility of Python 2 regressions is quite low. The number of code path changes when compiling with Python2 is <10 LOC (for the c code) and mostly bugfixes. The Python 3 version could contain overlooked regressions but they do not affect normal builds. All known regressions where fixed and most of them were caused by the differences in string/Unicode handling.
* I verified all changed python scripts with both Python 2.7 as well as Python 3.6 and there are no known regressions.
* This change would allow more users to test Python 3, as well as allow distributions which drop python 2 to still build with python support

I also made patches for wxPhoenix (https://github.com/pointhi/kicad-source-mirror/tree/wx-python), but they need some more work (there seems to be a windows regression with the old wxpython). I would like to get this stuff discussed and tested after python 3 support is merged.

Changed in kicad:
milestone: none → 5.1.0
Changed in kicad:
status: In Progress → Fix Committed
Changed in kicad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers