Qt Widgets Binding to Python

1. Introduction

Shiboken2 is a Python binding generator. It uses CLang to parse and analyze the structure of the C++ code (both from Qt, and our own widget), and typesystem to create the generated code through instructions.

docs/images/qtforpython-underthehood.png

Qt uses Shiboken2 to create its Python bindings, and already provides typesystem files for all its libraries. This is how PySide2 is built.

The mayor issue of Shiboken2, is that it uses CMake3 and a huge CMakeList.txt that is very difficult to understand. There is no other build system supported. In this page, I will explain how Shiboken2 works, and how the CMakeLists.txt file was translated.

2. Dependencies

  • Qt installed.

  • PySide2: PySide2 are the C++ wrappers and python bindings, normally installed through PIP.

  • shiboken2: Shiboken2 PIP module are the runtime dependencies for PySide2 or any Shiboken2 binded python library.

  • shiboken2_generator: These are the code-generation tools shiboken needs.

All of them must be in the same version. The easiest way is to figure out the base C++ Qt installation version, and install pyside2 from Qt PIP repository.

Warning

We need at least version 5.14.1, due to a bug present in 5.14.0 code generation include faults.

Warning

At this moment (DevEnv 2.2.0), shiboken2_generator is not present in devenv. Please install it yourself using these instructions:

pip install --index-url=http://download.qt.io/official_releases/QtForPython/ --trusted-host download.qt.io shiboken2==5.14.1 pyside2==5.14.1 shiboken2_generator==5.14.1

Once installed, you can try this example https://gitlab.eso.org/ahoffsta/cut/-/tree/master/examples/widget_example:

git clone https://gitlab.eso.org/ahoffsta/cut.git
cd cut/examples/widget_example
waf configure build install
cutwidgetsshowcasepy

3. Input for Bindings

Input files for binding generation in Shiboken2 are simple. Two files are needed:

  • C++ header file, that includes every class we need to bind.

  • Typesetting XML file, that indicates which will be the name of the target Python module, and which classes will be bind into said module.

The header file is just a list of #include sentences, no extra code is needed. Every class that needs binding has to be included in this file, through the use of common #include macro instructions. If two classes are defined in the same file, then only one include is needed.

1
2
3
4
5
#ifndef BINDINGS_H
#define BINDINGS_H
#define QT_ANNOTATE_ACCESS_SPECIFIER(a) __attribute__((annotate(#a)))
#include "QeWidgetExample.h"
#endif // BINDINGS_H

There are only two important lines, 3 and 4.

  • Line 3 indicates to the C++ preprocessor that it needs to make classifiers used by Qt visible. These classifiers are used in QObjects. (signals:, slots: in the header files).

  • In Line 4, you should include every widget you need. In here, we recommend to use only file named as the resulting target name. This will ensure compabitility with Python module names, allowing both C++ and Python version of the widget to provide correct include/import statements.

The typesystem file is a bit more complex.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
<?xml version="1.0"?>
<typesystem package="QeWidgetExample">
    <load-typesystem name="typesystem_core.xml" generate="no"/>
    <load-typesystem name="typesystem_core_common.xml" generate="no" />
    <load-typesystem name="typesystem_core_x11.xml" generate="no" />
    <load-typesystem name="typesystem_widgets.xml" generate="no" />
    <load-typesystem name="typesystem_gui_common.xml" generate="no" />
    <primitive-type name="double"/>
    <object-type name="QeDartBoard">
        <modify-function signature="paintEvent(QPaintEvent*)">
            <modify-argument index="1" invalidate-after-use="yes">
                <rename to="event"/>
            </modify-argument>
        </modify-function>
    </object-type>
</typesystem>

  • As you see here, the XML defines in line 2 a package called QeWidgetExamlple. This will be the name of the resulting Python package.

  • Lines 3 to 7 indicate the typesystem engine to load more typesystem definitions. These ones are the base set of Qt needed for widgets.

  • Then one primitive is declare in line 7, as it is used in slots and signals in the QeDartBoard class. If any other primitive is used, but it is not in the signature, then no definition for them is needed.

  • In line 9 an object is declared. This object is a C++ class named QeDartBoard. This should be the same class name as in the C++ code declaration. The resulting Python library will have a class in package QeWidgetExample named QeDartBoard.

  • Inside the object-type, several lines (10 to 14), give special instructions to typesetting. This indicates that the function, with signature “paintEvent(QPaintEvent*)” should be modified:

  • The first (index=1) argument gets invalidated after use, and it is renamed to event.

  • Lets remember that Event is a object generated in Python, but is being passed as argument to a C++ class. Since python keep references counters, we need to avoid leaking memory. This is how it is done in this case.

4. Code generation

In Qt’s examples, CMake is used. I will indicate here how to do this without a build system (and eventually, how to do it in WAF).

The Code generation is roughly this command, which invokes Shiboken2 with a bunch of options, and passes two mandatory arguments, the C++ header file, and the typesystem XML file:

$SHIBOKEN_CMD $SHIBOKEN_OPT ${SRC_DIR}/bindings.h ${SRC_DIR}/bindings.xml

The SHIBOKEN_CMD variable is path to the code generator. We use pyside2_config.py which is part of PySide2 package. It offers information about the installed libraries and includes, so that a build system can use those strings.

First we find out where the Python site-packages are located (pyside2 shiboken2 and shiboken2-generator are installed in this directory), and then we construct the path to Shiboken2 executable:

PYTHON_SP=$(python -c 'import site; print(site.getsitepackages()[0])')
SHIBOKEN_CMD="$(python $PYTHON_SP/PySide2/examples/utils/pyside2_config.py --shiboken2-generator-path)/shiboken2 "

The options are not supposed to change much, expect for the highlighted section (lines 8, 9, 10 and 11). Each of the -I entries is a path to the include files for the libraries mentioned in your source class to be binded.

  • Line 5 indicates where the includes for the original C++ widgets are.

  • Line 6 indicates where the bindings.h file is located

  • Line 7 $INTROOT/include are there for already installed files.

  • Lines 8 to 11 give access to QtWidgets, QtCore and QtGui classes.

  • Line 12 Also the PySide2/include headers files are needed

  • Lined 13 as well as the C++ STL headers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
SHIBOKEN_OPT="--generator-set=shiboken --enable-parent-ctor-heuristic "\
"--enable-return-value-heuristic "\
"--enable-pyside-extensions --use-isnull-as-nb_nonzero "\
--avoid-protected-hack "\
"-I${SRC_DIR}/../../widgets/src/include "\
"-I${SRC_DIR} "\
"-I$INTROOT/include "\
"-I/opt/Qt/5.14.0/gcc_64/include "\
"-I/opt/Qt/5.14.0/gcc_64/include/QtCore "\
"-I/opt/Qt/5.14.0/gcc_64/include/QtWidgets "\
"-I/opt/Qt/5.14.0/gcc_64/include/QtGui "\
"-I$PYTHON_SP/PySide2/include "\
"-I/opt/llvm/include/c++/v1/ "\
"-T$PYTHON_SP/PySide2/typesystems "\
"-T${SRC_DIR} "\
"--output-directory=${DEST_DIR}"

The ones in -T entries are special. -T indicates where the **t**ypesettings files are located, in case we need to “include” other libraries.

  • Line 14 indicates the location of all typesettings files from Qt. In this case, I do not want to specify how to bind a QWidget, QPen, QBrush classes, so I just indicate that my project uses typesettings files from Qt project.

  • Lined 15 indicate where our typesetting file is located.

Tip

If you need more example, we suggest to look at typesystem files from Qt:

As a result, this will generate a directory in DEST_DIR, that has 2 files for each binded class, and 2 more for each Python module indicated as target.

5. Compilation

The compilation of these are more or less straight C++ objects, that need to be linked together as a shared library.

We use WAF custom script to conduct compilation:

In lines 4 to 8, the configure method preparation during “waf configure” several variables and check for us. In this case, we tell WAF to find the generate script, and to declare the DEST_DIR and SRC_DIR variable. One is on the source, and the other in the build directory WAF uses for compilation products.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import os
import sys
from wtools.module import declare_custom
def configure(cnf):
  paths = [cnf.path.abspath(), os.path.join(cnf.path.abspath(), "bin")]
  cnf.find_program('generate', path_list=paths)
  cnf.env.DEST_DIR = cnf.path.find_or_declare('src').abspath()
  cnf.env.SRC_DIR =  cnf.path.find_node('src').abspath()


def build(bld):
  code_generation = bld(
    rule='${GENERATE} ${SRC_DIR} ${DEST_DIR}',
    source=['src/bindings.xml', 'src/bindings.h'],
    target=['src/QeWidgetExample/qewidgetexample_module_wrapper.cpp',
            'src/QeWidgetExample/qedartboard_wrapper.cpp'],
    name='bindings-generate'
  )
compilation = bld(
    features='pyext cxx cxxshlib wdep',
    includes=[
      '/opt/anaconda3/lib/python{}/site-packages/shiboken2/include'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/lib/python{}/site-packages/PySide2/include'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/include/python{}m'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/lib/python{}/site-packages/shiboken2_generator/include'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/lib/python{}/site-packages/PySide2/include/QtCore'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/lib/python{}/site-packages/PySide2/include/QtWidgets'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/lib/python{}/site-packages/PySide2/include/QtGui'.format(bld.env.PYTHON_VERSION),
      '/opt/anaconda3/lib/python{}/site-packages/PySide2/include/QtUiTools'.format(bld.env.PYTHON_VERSION)
    ],
    libpath=[
      '/opt/anaconda3/lib',
        '/opt/anaconda3/lib/python{}/site-packages/shiboken2'.format(bld.env.PYTHON_VERSION),
        '/opt/anaconda3/lib/python{}/site-packages/PySide2'.format(bld.env.PYTHON_VERSION),
    ],
    lib=[':libpython3.5m.so',':shiboken2.abi3.so', ':libpyside2.abi3.so.5.14',],
    source=['src/QeWidgetExample/qewidgetexample_module_wrapper.cpp',
            'src/QeWidgetExample/qedartboard_wrapper.cpp'],
    target='QeWidgetExample',
    use=['widgets','QT5CORE', 'QT5GUI', 'QT5WIDGETS', 'Qt5Designer',
         'Qt5Xml', 'Qt5UiPlugin'],
    name='bindings-compile'
  )
declare_custom(depends=['WidgetExampleWidgets'], provides=['bindings-compile'])

Lines 12 to 18, inside of the build method execute the “generation” script. It passes as argument the SRC_DIR and DEST_DIR variable from configuration. We use WAF source and target variables to indicate what it should expect as input, and output. This allows WAF to track what needs recompilation in case of changes, and the dependency tree.

Lines 19 to 43 to the compilation. This is quite large, so lets see its contents:

  • Line 20 indicates that features WAF will use to compile this target. In particular cxxshlib is of interest for us, as this is a shared library.

  • Includes (line 21), specify extra “-I” entries to the C++ compiler.

  • Lines 22, 23 and 25 includes Shiboken2 and PySide2 libraries

  • Line 24 include Python library. Since we are compiling a Python module, we need the Python library headers.

  • Lines 26 to 29 includes the necessary Qt headers, but from PySide2.

  • Lines 31 to 34 are special. LibPath extends locations of shared object libraries.

  • Line 36 makes use of the path above, and indicates extra libraries it needs to link the target againts.

  • Line 39 indicates the resulting name of the library. Since it is a shared library, it will be: libQeWidgetExample.so

  • Line 40 uses WDEP and PkgConfig to provide dependencies to WAF.

  • “widgets” target is the name of the directory in the WAF project structure that this module depends on. Not the name of the target. In case of nested modules, use a point separate notation (module.submodule.subsubmodule).

  • This module depends on “widgets”, as we are producing Python version of widgets defined in “widgets” module.

  • Line 44 is a general WAF module declaration where dependencies and targets provided are indicated.

6. Future Work

  • Integration with WAF:

  • Provide PySide2/Shiboken2 pkgconfig file.

Tip

Currently, you can find this in https://gitlab.eso.org/ahoffsta/cut/-/tree/master/examples/widget_example

7. FAQ

7.1. How do I use the widget?

The typesettings file (the XML in the Input section), includes the name of the Python package. Next, you need to know the name of the class, which is the same as in C++. A very simple application could be:

import sys
from PySide2.QtWidgets import QApplication
from QeWidgetExample import QeDartBoard

app = QApplication()
dartboard = QeDartBoard()
dartboard.show()
sys.exit(app.exec_())

7.2. If I connect a signal to one of the widget’s slots, it does nothing.

Most likely, the bindings.h file does not include the sentence to make visible the classifiers (signal, slots keywords in the C++ header).

#define QT_ANNOTATE_ACCESS_SPECIFIER(a) __attribute__((annotate(#a)))

Remember Shiboken2 needs to pass this instruction to the C++ preprocessor in order to generate the code.

Source: https://blog.basyskom.com/2019/using-shiboken2-to-create-python-bindings-for-a-qt-library/

7.3. PySide::Signal namespace not found

Compiler outputs an error like this one:

error: ‘PySide::Signal’ has not been declared

Most likely, you have not activated in Shiboken2 executable, the option that enables support for signal and slots. Shiboken2 is a fully capable binding generator, and Qt’s is one of the things it does. PySide2 support is not enabled by default. Check if your generation script includes this option for the Shiboken2 executable:

--enable-pyside-extensions

Source: https://github.com/ros-visualization/qt_gui_core/issues/142

7.4. If I connect a signal to one of the widget’s slots and trigger it, the application crashes with a Segmentation Fault.

Most likely, you have not activated in Shiboken2 executable, the option that enables support for signal and slots. Shiboken2 is a fully capable binding generator, and Qt’s is one of the things it does. PySide2 support is not enabled by default. Check if your generation script includes this option for the Shiboken2 executable:

--enable-pyside-extensions

7.5. A symbol is missing (a method), and it is from my widgets:

This happens when you generate the code for all the widgets you want to, but forget to include them in the final linking of the Python shared library that acts as a Python Package. Check in the wscript that does the compilation, if all <widget_class_name>_wrapper.cpp are listed as target for the generation task, and listed as sources in the compilation task.

7.6. Application crash on startup, indicates Qt_5_6 version is used, but it was compiled against Qt_5_14

DevEnv comes with PyQt installed (provided by anaconda) and it is used by multiple applications. But the compilation procedure will resolve against Qt5.14, if every dependency is declared.

In your project, you have to manually configure the dependencies, and then use them in widgets, and in bindings.

For example, in the project (first) wscript:

def configure(cnf):
    for pkg in 'Qt5Core Qt5Gui Qt5Widgets'.split():
        cnf.check_cfg(package=pkg, uselib_store=pkg)

And then in your wscript of the widget:

declare_qt5cshlib( target='widgetexample-widgets',
                   use='QT5CORE QT5GUI QT5WIDGETS'
)

These are just snipper of code. Take a look into the example in https://gitlab.eso.org/ahoffsta/cut/-/tree/master/examples/widget_example for complete wscripts.

7.7. My application crashes, in a section that is C++, how can I get a back trace

You need to start the python interpeter with GDB:

gdb
file python
run <path_to_python_script>

When the application crashes, type in the GDB console:

bt

Indices and tables