PageDump: Python extensions

Preview: Page dump of Python extensions

Date: Tue, 15 May 2007 21:02:50 +0000
Mime-Version: 1.0 (Produced by PhpWiki 1.3.14)
Message-Id:
<1314-1179262970+0000-MinGW%3APython%20extensions@oldwiki.mingw.org>
Content-Type: application/x-phpwiki;
  pagename=Python%20extensions;
  flags="";
  author=UnDefined;
  owner=KenStaton;
  version=10;
  lastmodified=1179262970;
  created=1179262970;
  author_id=UnDefined;
  markup=2;
 
summary=update%20for%20python%20%3E%3D%202.4%2C%20but%20leave%20and%20mark%20historical%20notes%20%28python%20%3C%3D%202.3%29%20as%20such;
  hits=50322;
  charset=iso-8859-1
Content-Transfer-Encoding: binary

!!!Overview

As of 2007-05-15, using Python 2.4.4 or 2.5.1 and the MinGW 5.1.3
installer installing "current" (mingw-runtime 3.12, w32api 3.9,
binutils 2.16.91-20060119-1, gcc-core 3.4.2-20040916-1), the ["minimal
example"|http://en.wikibooks.org/wiki/Python_Programming/Extending_with_C#A_minimal_example]
Python module can be built easily with MinGW following the same
instructions as for Linux, but invoking distutils with =python
setup.py build --compiler=mingw32=.

But if you are using a win32 binary distribution of Python 2.3 or
older (e.g. the one from www.python.org), then developing Python
extensions from a *mingw/MSYS* (or *cygwin*) environment requires a
few tricks that are not obvious. This is because the python
distribution does not come with an export library, so you have to
create one, and there's a couple of other gotchas too.

I spent a lot of time looking around for information on how to make
this work. The most cited source seems to be
http://sebsauvage.net/python/mingw.html but for some reason it didn't
work for me (just bad luck?). Several other pages exist on the web
that give some instructions, but I found that they are either out of
date, or wrong, or too difficult to understand. The Python.org
document, ["Installing Python Modules", section 6.2.2 |
http://www.python.org/doc/2.3.4/inst/tweak-flags.html] explains the
basics, and this is the only one that worked for me, at least with
MinGW 3.1.0-1 (MSYS 1.0.10 w/ gcc 3.2.3) and Python 2.3.3.

While looking for something else, I came across this patch request
made by [Connelly (python-Bugs-957198)
|http://mail.python.org/pipermail/python-bugs-list/2004-June/023351.html]
that summarizes nicely what to do, and I so wish I had found that
earlier, that I decided to create this page. I include the relevant
parts of the instructions mentioned in the patch, and I add a few
clarifications.

!!!Setup for Python 2.3 (or older)

You only need to do this once, every time you change your Python
interpreter (i.e. upgrade or re-install):

# If your mingw does NOT contain pexports (e.g., do ='which
pexports'= from msys command line), [download it |
http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip] and
extract =pexports.exe= to your Mingw =bin= directory.
# Find =pythonxx.dll=. On my machine, with Python 2.3 it is in
=/c/WINNT/system32=, as =python23.dll=. It has been mentioned in
several web pages that it could also be in =/yourPythonRoot/libs=.
# cd to =/yourPythonRoot/libs=. E.g. on my system I cd to
=/c/Python23/libs=.
# Assuming =pythonxx.dll= is in =/c/WINNT/system32=, do the following
(from =/yourPythonDir/libs=; also, adjust the version number to your
version of Python):

  <verbatim>
  pexports /c/WINNT/system32/python23.dll > python23.def
  dlltool --dllname python23.dll --def python23.def --output-lib
libpython23.a
  </verbatim>

!!!Building an extension

You are now ok to build python extensions. E.g. if your extension is
in =example.c=,

<verbatim>
    gcc -c  example.c  -I/c/Python23/include
    gcc -shared example.o  -L/c/Python23/libs -lpython23  -o
example.dll
</verbatim>

If you haven't done the above Basic Setup steps correctly, you could
get several kinds of errors:

# The link stage (=gcc -shared ...=) could fail, saying the python23
library was not found, is not the correct format, there are undefined
symbols;
# When you run python and import your module, you may get a ="Fatal
Python error: Interpreter not initialized (version mismatch?)"= and
the interpreter will exit.

If you use *SWIG*, remember that your dll extension must start with
an underscore, because swig wraps your dll with Python code to make it
more robust. E.g. the above example would become:

<verbatim>
    swig -python  example.i
    gcc -c example_wrap.c example.c  -I/c/Python23/include
    gcc -shared example.o example_wrap.o -L/c/Python23/libs
-lpython23  -o _example.dll
</verbatim>


!!!Creating an extension with Distutils

You must specify which compiler to use, so you would do:

<verbatim>
    python setup.py build --compiler=mingw32
</verbatim>

and then:

<verbatim>
    python setup.py install
</verbatim>

Note this [interesting post |
http://mail.python.org/pipermail/distutils-sig/2004-October/004194.html]
on the distutils-sig mailing list:

<verbatim>
    Thomas Heller wrote:
    > Scott David Daniels <Scott.Daniels at Acm.Org> writes:
    >>I am having a bit of trouble creating a windows distributable
    >>with a C module using MingW32.  While I can do:
     >>         python setup.py build --compile=mingw32
     >>         python setup.py bdist_wininst --skip-build
     >>I get an executable with no tag for the particular python
version.
    >
    > You could use command chaining, if that's the correct word for
that:
    > python setup.py build --compiler=mingw32 bdist_wininst

    This does the trick, both for using the mingw32 compiler, and
for
    tagging the result with the python version.  Thanks for the
help.

    -- Scott David Daniels
</verbatim>

If you are wondering what the =setup.py= file looks like, here is an
example for a C extension, YMMV:

<verbatim>
    # setup.py
    from distutils.core import setup, Extension
    setup (name = "_example",
        version = "1.0",
        maintainer = "Your Name",
        maintainer_email = "your.name@domain.org",
        description = "Sample Python C Extension",
        ext_modules = [Extension('_example',
                      extra_compile_args=['-O2'],
                      library_dirs=['/mingw/lib'],
                      sources=['example.c','example_wrap.c'])])
</verbatim>

And here is an example for C++, YMMV:

<verbatim>
    # setup.py
    from distutils.core import setup, Extension
    setup (name = "_example",
        version = "1.0",
        maintainer = "Your Name",
        maintainer_email = "your.name@domain.org",
        description = "Sample Python C++ Extension",
        ext_modules = [Extension('_example',
                       extra_compile_args=['-O2'],
                       library_dirs=['/mingw/lib'],
                       libraries=['supc++'],
                       sources=['example.cxx','example_wrap.cxx'])])
</verbatim>

!!!Extra Setup for Distutils in Python 2.3 (or older)

You may want to use *distutils* to distribute or even just build your
python extensions.  If you are using Python 2.3 or older and do NOT
have MSVC installed on your machine, then you need to patch
*distutils* because it thinks your platform is nt, and the nt compiler
defaults to MSVC (I haven't yet found how to say in =setup.py= that
you want the 'unix' compiler type).  So:

# Locate =msvccompiler.py= in your Python distribution. On my
machine, it is in =/c/Python23/Lib/distutils=.
# Open it
# Find and DELETE the following lines (around line 211):

  <verbatim>
  if len (self.__paths) == 0:
      raise DistutilsPlatformError, ("Python was built with version
%s of Visual Studio, "
                "and extensions need to be built with the same "
                "version of the compiler, but it isn't installed." %
self.__version)
  </verbatim>

You will notice that if you don't apply this patch, the above
exception is raised when you do =python setup.py install=, because the
condition is true when MSVC is not installed (the path doesn't contain
anything).

NOTE: (added by Chris Barker).
Rather than editing the distutils code, you can just do:

=python setup.py install --skip-build=

By skipping the build stage, you seem to skip the compiler check.
END NOTE

If you want to make *C++* extensions, you will likely have to create
=/mingw/bin/cc.exe=. You will know, because the setup.py build will
give an error about =cc.exe= not found. On my system, I have found
that I had to create it by copying from =/mingw/bin/dllwrap.exe=, but
the original Connelly post says to copy from =/mingw/bin/gcc.exe=, so
maybe it depends on the version of gcc or mingw you have. Naturally,
creating =cc.exe= only has to be done once, forever.


!!! Extra Setup for Swig/autotools

If you have configure scripts that use AM_PATH_PYTHON or PYTHON_DEVEL
(from Swig),
there's a bit more that you need to do if you want them working in
MinGW.

First, make sure there is nothing from cygwin in your $PATH variable.
Make a
/usr/local/bin (*) if you don't have one already and put that in your
$PATH. Make
a file called 'python' in /usr/local/bin that contains simply:

<verbatim>
#!/bin/sh
/f/Software/Python-242/python.exe "$@"
</verbatim>

replacing /f/Software/Python-242 with /yourPythonRoot/


On my machine, after doing this, AM_PATH_PYTHON was able to find the
script and extension directories
in /f/Software/Python-242/Lib/, but not the includes or the
libraries. So

<verbatim>
cp -r /yourPythonRoot/include /usr/local/include/python2.4
cp -r /yourPythonRoot/Lib /usr/local/lib/python2.4
cp -r /yourPythonRoot/libs /usr/local/lib/python2.4/config
</verbatim>

You'll also need a dot in the version spec for the library you
created above with dlltool, so (for example):

<verbatim>
cp /usr/local/lib/python2.4/config/libpython24.a
/usr/local/lib/python2.4/config/libpython2.4.a
</verbatim>

Then things should be peachy.

-- TrentApted 2005-10-27

!!! Python2.4 and MSVCR71

This seems to be widely discussed with no clear answer.%%%
One approach is to just follow the directions above.%%%
If this doesn't work, or bothers you, then check these pages:%%%
http://article.gmane.org/gmane.comp.python.general/378028/match=msvcr71
http://jove.prohosting.com/iwave/ipython/issues.html (section 4 -
Extending the official distribution)

If you run depends.exe (download it with Microsoft's platform SDK) on
the windows binary (python.exe):%%%
<verbatim>
Python.exe depends on: python24.dll, msvcr71.dll, and kernel32.dll
python24.dll depends on: kernel32.dll, user32.dll, advapi32.dll,
shell32.dll, and msvcr71.dll
shell32.dll depends on: msvcrt.dll (and others)
</verbatim>
If nothing else works, you can download Microsoft's toolkit and
follow these directions:
http://www.vrplumber.com/programming/mstoolkit/  (Note that the
author of this page gave up)

-- KenStaton 22Nov2005

!!! One Last Step...
This may help some, it worked for me (I'm using distutils) for
Python24xx and now for Python25rc1.
It requires the minimum setup and does not modify any Python files.
# Follow the instructions above under "Basic Setup" to create a
compatable library.
# Create a cfg file for distutils (distutils.cfg) eg:
/c/Python2x/Lib/distutils/distutils.cfg
containing:%%%
<verbatim>
[build]
compiler=mingw32
</verbatim>
Done. For whatever reason, using "--compiler=mingw32" never worked
for me.

-- ShaneYuile 2006-Sep-11