In version 0.3 of pymatbridge, an additional dependency was added for ZMQ. To install this version, you might therefore need to compile a messenger mex extension linking against a dll of zeromq. However, this turns out to be rather difficult to do in some cases.
Does anyone have examples of success stories installing pymatbridge from source on a machine running Windows?
Here's my setup and build instructions; apologies for the verbosity -- this is from a write-up I did for my lab notebook.
Specifications:
Windows 8.1 (64 bit)
Anaconda 1.9.1
python 2.7.6
MinGW
TDM GCC 4.8.1
First I installed MS Visual C++ & SDK 7.1. Possible reason for the install to fail quite early on is the presence of other Visual C++ compiler versions and/or SDKs -- before successfully installing SDK 7.1 I had to uninstall quite a few of them.
First, using the Windows SDK 7.1 Command Prompt, I
mex -setup
and selected the newly installed compiler. In my case
Select a compiler:
[1] Microsoft Software Development Kit (SDK) 7.1 in
C:\Program Files (x86)\Microsoft Visual Studio 10.0
[0] None
Compiler:
I elected to pick number 1.
Next I downloaded and installed a prebuilt 64 bit ZMQ 3.2.4, and from the ZMQ 3.2.4 install location, C:\Program Files\ZeroMQ 3.2.4\lib, I copied libzmq-v90-mt_3_2_4.lib, and libzmq-v90-mt_3_2_4.dll from C:\Program Files\ZeroMQ 3.2.4\bin to the messenger build directory, ~\python-matlab-bridge\messenger\src.
I also put zmq.h and zmq_utils.h into that directory but as of now am unsure if they are really necessary.
Then, in the Windows SDK 7.1 Command Prompt, I cded into the messenger\src directory and issued
mex messenger.c libzmq-v90-mt-3_2_4.lib -O -output messenger
to obtain a messenger.mexw64.
Finally, in the ~\python-matlab-bridge directory, using MinGW, I just
python setup.py install
This automatically put the built package into my Anaconda\Lib\site-packages along with an egg info. In Anaconda\Lib\site-packages\pymatbridge\matlab there is the messenger.mexw64 that was built earlier -- from ~\python-matlab-bridge\messenger\src.
I also copied to Anaconda\Lib\site-packages\pymatbridge\matlab all other .dlls, .libs and .hs for good measure.
Now in the iPython 2.7 console, or an iPython Notebook, I can
%load_ext pymatbridge
successfully.
As I use Visual Studio as my standard IDE I fiddled around and got a working build process.
I used:
VS 2013
Matlab 2013b x64
Python 3.3 x64
ZeroMQ 4.0.4.
If you use different Versions please make sure that you adjust the Path where necessary.
Build messenger using Visual Studio (2013)
First create a new Solution with an empty C++ Project.
Set up the configuration, e.g. x64
Release
Open up the Project Properties and set up the following:
Configuartion Properties -> General:
Target Name: messenger
Target Extension: .mexw64
Configuration Type: Dynamic Library (.dll)
Configuartion Properties -> VC++ Directories -> Include Directories:
MATLABPATH \extern\include
ZEROMQ PATH\include
Configuartion Properties -> VC++ Directories -> Library Directories:
MATLABPATH \extern\lib\win64\microsoft
ZEROMQ PATH \lib
Configuartion Properties -> Linker -> Input -> Additional Dependencies:
libmx.lib; libmex.lib; libmat.lib; libzmq-v120-mt-4_0_4.dll
Configuartion Properties -> Linker ->Command Line -> Additional Options:
/export:mexFunction
Add messenger.c as source file
Build the library
Copy it to: PYTHONPATH \Lib\site-packages\pymatbridge\matlab\
Please Note:
I got some incompatible error cause by the function names already present in windows.h.
The solution was to rename:
initialize
listen
respond
cleanup
to:
initializeMat
listenMat
respondMat
cleanupMat
Some of the other answers are a little bit outdated, and this can be done much more easily now (still won't be as easy on a Linux system :/). First of all, pymatbridge now includes a binary mex on Windows which is statically linked to libzmq, so you shouldn't actually have to compile anything to get it to work.
However, if you still want to do compile it from source, the process isn't too bad for a dynamically linked library.
1) Install zeromq from the website: http://zeromq.org/distro:microsoft-windows
2) Rename one of the lib/libzmq-*.lib files to libzmq.lib in the ZeroMQ installation directory
3) Add the ZeroMQ bin directory to your path.
4) Edit the messenger/mexw64/local.cfg file in messenger to point to the zeromq install directory (you will need to update both ZMQ_INC and ZMQ_LIB). Also ensure the MATLAB directory is correct.
5) Run make.py matlab in the messenger directory. This should build messenger.mexw64
A statically linked library is simple in practice, but because zeromq (as of now) does not provide a .lib for static linking with the windows installer, you will need to compile this yourself. After you have a static .lib, build messenger.mexw64 with make matlab --static
Related
I am running python 3.9.1 for windows 10 (x64). I have noticed the openssl dll files shipped with python (libcrypto-1_1.dll & libssl-1_1.dll) are only version 1.1.1.7. I would therefore like to replace these with the latest versions.
I have been able to successfully build openssl 1.1.1.9 in windows 10 following the instructions in the release. I am using Microsoft visual studio 2019 + strawberry perl + nasm. I ran the following config command using the x64 native tools command prompt for vs 2019:
perl Configure VC-WIN64A
I then run the nmake command. This produces the two dlls I need: libcrypto-1_1-x64.dll and libssl-1_1-x64.dll
I renamed both files to remove the "-x64" at the end of the name and then used them to replace the existing dlls in the python installation.
When I try to import the ssl library in python, I get the error:
ImportError: DLL load failed while importing _ssl: The specified module could not be found.
I then built openssl for x86, using the x86 native tools command prompt for vs 2019 and the command:
perl Configure VC-WIN32
I then ran nmake which produced: libcrypto-1_1.dll and libssl-1_1.dll
I replaced the dlls in the python installation with those. When I try and import the ssl library in python I get:
ImportError: DLL load failed while importing _ssl: %1 is not a valid Win32 application.
I am reasonably certain I need the x64 version of the dlls because (1) my python installation is x64 (2) when I used the visual studio dumpbin.exe tool on the python shipped dlls to inspect the headers they say "x64" in them (3) the x64 dlls are a similar size to the ones shipped with python and the x86 versions are smaller.
I was once able to do this following the above steps for an older version of python. That version shipped with openssl 1.1.3 (could be 4) and I built the dlls for 1.1.6 (could be 7) and replaced them and python had no issues. Is it possible that the default configuration scripts have changed over these versions such that the default windows x64 configuration no longer builds openssl in a way that works with python and I therefore need to use custom settings?
I think the info at section 1.3 of this page may be relevant, but if it is I am not sure how to implement that in the openssl build process.
I managed to solve this with some help.
The scripts used by CPython to build OpenSSL for the windows installers actually run the command:
Perl Configure VC-WIN64A-masm no-asm
instead of
Perl Configure VC-WIN64A
After running the correct command on the setup described in my original post I was able to swap out the dlls and python accepted them without issue.
ssl.OPENSSL_VERSION now outputs ‘OpenSSL 1.1.1i 8 Dec 2020’.
To preface: my code works as I expect when compiling and running on Linux. However, this library needs to be compiled for use on a Windows machine. I looked in to a couple different options, and decided that using Cygwin to compile for Windows seemed to be the correct choice. I'm using a setup.py file with the distutils.core library and compiling using python setup.py install. When compiling on Windows in Cygwin, it fails to find pthread.h, arpa/inet.h, netinet/in.h, and sys/socket.h. I was under the impression that Cygwin came prepackaged with these headers, which is why I chose to use it. The alternative to Cygwin is putting preprocessor commands everywhere and using Windows specific libraries such as winsock2.h, which I want to avoid if at all possible. Is it possible to compile for Windows using Cygwin? If so, what have I done wrong to cause Cygwin to not recognize these headers?
You need to install the proper headers
$ cygcheck -p usr/include/pthread.h
Found 9 matches for usr/include/pthread.h
cygwin-devel-3.0.7-1 - cygwin-devel: Core development files
..
cygwin-devel-3.1.6-1 - cygwin-devel: Core development files
...
so install the cygwin-devel package
To check all the shared libraries needed by the built dll, you can use cygcheck
$ cygcheck /usr/lib/python3.8/site-packages/Cython/Compiler/FlowControl.cpython-38-x86_64-cygwin.dll
D:\cygwin64\lib\python3.8\site-packages\Cython\Compiler\FlowControl.cpython-38-x86_64-cygwin.
dll
D:\cygwin64\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
D:\cygwin64\bin\libpython3.8.dll
D:\cygwin64\bin\cygintl-8.dll
D:\cygwin64\bin\cygiconv-2.dll
D:\cygwin64\bin\cyggcc_s-seh-1.dll
As was built with Cygwin Python, you need also to transfer the cygwin python...
Most important, I think, is to follow the instructions in the Python help or on the Python doc web site for "Extending and Embedding the Python Interpreter" for the version you are building the extension for. For windows, the build instructions identify the build environment used to create the binary package that you download from python.org, usually something like VS2013 or VS2017. (As an aside, I think the Community editions have everything you need, and I don't think you actually have to use the Visual Studio GUI when you build using nmake from the CMD.EXE terminal.)
To build in Cygwin for use in a Windows version of Python, you may need to install and then use the x86_64-w64-mingw32-gcc, etc., cygwin packages to cross-compile non-cygwin (i.e. pure windows) executables and DLLs from Cygwin.
Binary extensions must be built using the source tree for a specific Python major.minor version, and bitness. For windows, you will need to build multiple versions of the extension, one for each major.minor, bitness version of Python that will import it, e.g. 3.6, 3.7, 3.8, 3.9, 32-bit, 64-bit. The extension code may not require changes between versions, but it still needs to be compiled with the right compiler and linked against exactly the same shared libraries (in this case .DLL files) as used by the Python executable. For instance, it must use exactly the same version of Microsoft's C run time library DLL as the Python executable does. This is a bit more sensitive and restrictive than on Linux, where you can rebuild the python executable and your extension with the same toolchain from your distro more easily.
After installing Python 3.5.6 using a tarball (.gz file), I would like to run
C:\path\to\desired\venv\directory>c:\path\to\python\installation\Python-3.5.6\python -m venv venv
However, my Python-3.5.6 directory doesn't seem to include the necessary python.exe. After extracting Python from a tarball on Windows, what extra steps are required to produce the python executable?
Note: I already have Python 3.6 installed as the default.
That's not a "tarball installation". You downloaded the source code and extracted it manually into somewhere you shouldn't. This is roughly analogous to attempting to live in the blueprints of a house.
If for some reason you are determined to use 3.5.6, you will have to find an actual installer somewhere (there is no official one), or you will have to build it yourself. From PCbuild/readme.txt:
Quick Start Guide
Install Microsoft Visual Studio 2015, any edition.
Install Subversion, and make sure 'svn.exe' is on your PATH.
Run "build.bat -e" to build Python in 32-bit Release configuration.
(Optional, but recommended) Run the test suite with "rt.bat -q".
For more complete build instructions, see PCbuild/readme.txt.
I have installed PostgreSQL Server 9.6.0 and Python 3.4.2 on Windows 2012 R2 Server.
I copied plpython3.dll to C:/Program Files/PostgreSQL/9.6/lib/
The in PostgreSQL I try running this command: CREATE EXTENSION plpython3u;
And I receive this message:
ERROR: could not load library "C:/Program Files/PostgreSQL/9.6/lib/plpython3.dll": The specified module could not be found.
Under this folder: C:\Program Files\PostgreSQL\9.6\share\extension there are plpython3u files.
How can I get PostgreSQL to recognize this Python 3 extension?
Thanks!
Copy the python34.dll file to c:\windows\system32 and name the copy python33.dll
The create language plpython3u should then work without a problem.
Exactly the same situation I faced with Postgres 9.6 Windows 10.
PL/Python3U would not get through.
Worked around it:
Installed Python34 64bit Windows 10 version.
Copied Python34.dll to c:\windows\system32 as Python33.dll and it worked.
The information is in Makefile of source installation package
We need libpython as a shared library. In Python >=2.5, configure asks Python directly. But because this has been broken in Debian for a long time (http://bugs.debian.org/695979), and to support older Python versions, we see if there is a file that is named like a shared library as a fallback.
for python windows:
ifeq ($(PORTNAME), win32)
pytverstr=$(subst .,,${python_version})
PYTHONDLL=$(subst \,/,$(WINDIR))/system32/python${pytverstr}.dll
So the write answer is:
WINDIR is : C:\Windows
pytverstr is use in makefile has a parameter to define version of python
PYTHONDLL is the location of dll
To check version of my installation, I open plpython3.dll located in C:\Program Files\PostgreSQL\9.4\lib (change path with your environnement)
With Notepad++ and search PyUnicode_AsUTF8String
the python dll version is visible in last word (in my case python33.dll)
check your installation to choice the good installer of python
SELECT version();
PostgreSQL 9.4.15, compiled by Visual C++ build 1800, 64-bit
So I need install Python 33 in 64bit
edit 2020-10-02
There is also all of these informations in doc of binary ..pgsql\doc\installation-notes.html look at title Procedural Languages
edit 2021-06-11
After install the good version of Python on your system you need copy it to C:\Windows\System32
Replacing the version of python with an old name is not a good solution because you can have librairie it not work with this version. Do that if you know risks. So if you want a newest version of python for plpython, compile it or check edb compilation to check if it contains what you need. You can ask EDB for this information.
plpython3.dll in the official package is built against python 3.3, not python 3.4. What it expect is python33.dll in system32 folder. You need to install python 3.3 for your system.
Since py33 has been phased out, you may soon get frustrated, due to lack of pre-built binary package, lxml, pyzmq etc all needs to be built from source. If you need any binary module, make sure you have a correctly set up compiler.
This may be helpful, I have struggled a lot with this. For me only worked when I installed the right version of python and added paths to environment variables. I am not sure if python 3.4.0 would be the right version for PostgreSQL Server 9.6.0 but it worked fine with PostgreSQL Server 10.0.
Try version python-3.4.0.amd64 for windows 64bit or other versions from this Python 3.4.0 downloads Link
Environment variables to add:
C:\Python34\Scripts
C:\Python34\
I'm trying to use scapy on win32 python2.7
I've manage to compile all the other dependencies expect this one
can some help in the goal of reaching this executable ?
"dnet-1.12.win32-py2.7.exe"
(I promise to update the this question too and the scapy manual,
Running Scapy on Windows with Python 2.7)
Update:
I've managed to compile it with mingw32
I'm using vs2005, and I have to make some fixes to libdnet to actually work (look like last time they compiled it on windows it was with vs6.0
I'll try updating scapy manual... (and upload the executables to there)
Did you read the README file? Download the source tarball from http://prdownloads.sourceforge.net/libdnet/libdnet-1.11.tar.gz?download, extract it and see README:
Windows 2000/XP
---------------
For raw Ethernet sending, install the WinPcap driver and DLLs, and
extract their developer pack to a build directory:
http://winpcap.polito.it/install/default.htm
# XXX - unsupported for now
# For tunnel interface support, install the OpenVPN "TAP-Win32 Virtual
# Ethernet Adapter" component (their Windows installer will let you
# install it alone):
#
# http://openvpn.sourceforge.net/
For firewall support, install HSC's PktFilter service:
http://www.hsc.fr/ressources/outils/pktfilter/index.html.en
Most Windows developers should just use the libdnet developer's pack
(with MinGW and MSVC++ libraries) instead of building it themselves.
But for those who really want to know...
To build a MinGW native library (under Cygwin+MinGW):
./configure && make
To build a Microsoft Visual C++ native library and Python module
(again, under Cygwin+MinGW):
./configure
cd python && C:/Python23/python.exe setup.py build
cd ../src && lib /out:dnet.lib *.obj
As you most likely noticed you will need Cygwin+MinGW. I don't know whether you want the MinGW native lib or the Microsoft Visual C++ native lib. Also note that you have to use the C:/Python27/python.exe to build it.
If you try this, you can come up with a concrete problem I guess, which you most likely, but not 100% necessarly, better put into a new question and might better ask the mailing list of libdnet if you have compilation problems.
Dirk Loss provides an Windows installer for the dnet package for Python 2.7. Together with the Windows installer for the pypcap package for Python 2.7 I was able to run Scapy successfully on Python 2.7 on Windows.