I'm trying to get the HDBSCAN package to run on a Windows 7 machine with no C++ compiler. Installing a compiler is not an option, unfortunately.
I read that some packages have precompiled wheel files that require no compiler to install. The installation notes state "Binary wheels for a number of platforms are available thanks to the work of Ryan Helinski". However, there is no mention of where those can be found. My questions are then:
How do I obtain the .whl file for the HDBSCAN package?
Is it possible to simply compile+install on another machine and copy it? If so, what should the machine on which I compile have in common with the one on which the code must run? Can I do it on a Windows 10 machine, or does it have to be Win7 as well, do the same Windows updates need to have been run on both, etc?
Looks like there is no pre-built wheel distribution published by the maintainers of the project themselves on PyPI.
As already mentioned by other contributors, one could get such wheel distributions from a third party such as Christoph Gohlke's "Unofficial Windows Binaries for Python Extension Packages".
It is also of course possible to build such wheels yourself on one machine and then reuse it on another. As far as I know in the case of Windows, both machines need to have the same Python interpreter (minor) version as well as the same bitness (both 32 bits or both 64 bits). The exact version of the operating system should not matter (from Windows 10 to Windows 7 or the other way around should work).
I ended up getting this to work by compiling on another machine and copying the package from it. It was critical that the required packages were of the same version on both machines, so I simply set up a new conda environment which had the same package versions as the target machine, then intalled hdbscan with pip there, and copied.
I was worried about Windows version compatibility, but this worked even though I installed on Windows 10 and moved to a machine running Win7.
here is the site you can download the wheel file :https://www.lfd.uci.edu/~gohlke/pythonlibs/
for python 3.5>
or PyPi for older versions
run python -m pip install thefiledownloadedforyourpythonversion.whl
For the second question , yes you can but is rather complicated and you should avoid it when you can :)
I just checked. Python 3.8 is coded in C. You need a C compiler, not a C++ one. MINGW is one, and TinyCC Win32 is another one (a small one, producing quickly slow executables). Look also into this list of free C or C++ compilers.
A possibility (which could take several days of work) might be to use some WSL or some Linux emulator such as JSLinux (it runs in a web browser). Then you could (painfully) build a cross compiler (starting first with a tinycc-win32, then compiling an old cross GCC 3 compiler, then compiling with that a newer C++ GCC, etc....)
But the reality is that you should not be allowed to do this. Your real issue is non technical: why are you not allowed to install then use a C++ compiler, such as MinGW? Get permission (and resources) to install some....
Alternatively, consider installing some Linux distribution (ensure you are allowed to do so). Most of them have a recent Python and GCC...
Is it possible to simply compile+install on another machine and copy it?
This is called cross-compilation and is possible in general. The point is to be permitted to do that. You should find relevant cross-compilers for your situation.
If you are allowed to, you could even use a live Linux USB stick....
Some C compilers written in Python do exist.... You could use them to cross-compile tinycc for Win32. Then you should have a C compiler for Win32. You could then compile an old GCC, etc.... Qemu exist for windows. You could run a Linux with a cross-GCC compiler in Qemu.
Related
To preface: my code works as I expect when compiling and running on Linux. However, this library needs to be compiled for use on a Windows machine. I looked in to a couple different options, and decided that using Cygwin to compile for Windows seemed to be the correct choice. I'm using a setup.py file with the distutils.core library and compiling using python setup.py install. When compiling on Windows in Cygwin, it fails to find pthread.h, arpa/inet.h, netinet/in.h, and sys/socket.h. I was under the impression that Cygwin came prepackaged with these headers, which is why I chose to use it. The alternative to Cygwin is putting preprocessor commands everywhere and using Windows specific libraries such as winsock2.h, which I want to avoid if at all possible. Is it possible to compile for Windows using Cygwin? If so, what have I done wrong to cause Cygwin to not recognize these headers?
You need to install the proper headers
$ cygcheck -p usr/include/pthread.h
Found 9 matches for usr/include/pthread.h
cygwin-devel-3.0.7-1 - cygwin-devel: Core development files
..
cygwin-devel-3.1.6-1 - cygwin-devel: Core development files
...
so install the cygwin-devel package
To check all the shared libraries needed by the built dll, you can use cygcheck
$ cygcheck /usr/lib/python3.8/site-packages/Cython/Compiler/FlowControl.cpython-38-x86_64-cygwin.dll
D:\cygwin64\lib\python3.8\site-packages\Cython\Compiler\FlowControl.cpython-38-x86_64-cygwin.
dll
D:\cygwin64\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
D:\cygwin64\bin\libpython3.8.dll
D:\cygwin64\bin\cygintl-8.dll
D:\cygwin64\bin\cygiconv-2.dll
D:\cygwin64\bin\cyggcc_s-seh-1.dll
As was built with Cygwin Python, you need also to transfer the cygwin python...
Most important, I think, is to follow the instructions in the Python help or on the Python doc web site for "Extending and Embedding the Python Interpreter" for the version you are building the extension for. For windows, the build instructions identify the build environment used to create the binary package that you download from python.org, usually something like VS2013 or VS2017. (As an aside, I think the Community editions have everything you need, and I don't think you actually have to use the Visual Studio GUI when you build using nmake from the CMD.EXE terminal.)
To build in Cygwin for use in a Windows version of Python, you may need to install and then use the x86_64-w64-mingw32-gcc, etc., cygwin packages to cross-compile non-cygwin (i.e. pure windows) executables and DLLs from Cygwin.
Binary extensions must be built using the source tree for a specific Python major.minor version, and bitness. For windows, you will need to build multiple versions of the extension, one for each major.minor, bitness version of Python that will import it, e.g. 3.6, 3.7, 3.8, 3.9, 32-bit, 64-bit. The extension code may not require changes between versions, but it still needs to be compiled with the right compiler and linked against exactly the same shared libraries (in this case .DLL files) as used by the Python executable. For instance, it must use exactly the same version of Microsoft's C run time library DLL as the Python executable does. This is a bit more sensitive and restrictive than on Linux, where you can rebuild the python executable and your extension with the same toolchain from your distro more easily.
I have written several small scripts in python and build the binaries using pyinstaller.
When I would build them on my Ubuntu 16.04 machine -- they would run fine on the machine where I build them. But moving the file to a Centos / Redhat 7.4 machine would give me GCLIB and other .so version dependency errors.
Building the same binary on docker with the same version of Centos would not give these errors.
If I try running the binary compiled on Centos 7.4 on Centos 6.6 I would get errors again -- But building on Centos 6.6 it would work correctly with Centos 6.6
I have solved the problem using a lower version of Centos to build my binaries for now.
My specific question is -- In Python, is it a common approach of building the binaries on different OSs based on the target OS that it is intended for (assuming linux targets only) or what I am doing is a hack / bad way of solving this problem?
I am trying to understand how this problem is approached in the standard way.
As long as the binary produced by pyinstaller only depends on glibc, then it should be a valid approach to build it on the oldest system available, and it should work on future systems.
In general, glibc is designed to be backwards compatible, so that applications built against an older version of glibc will still run with a newer glibc, but not vice versa. It does this via symbol versioning, in which each symbol you link to can have a version associated with it, and any case in which a newer glibc has changed the ABI of some function, it will also have a compatibility routine with the old ABI exposed with the older symbol version, so that applications linked against the old one will dynamically link with the compatibility routine, while if you have an application linked against the newer symbol versions, there won't be the newer versions in an older glibc to dynamically link to.
While other libraries can also do this, not many library authors bother to, so newer versions may simply be incompatible, while the glibc developers generally try to preserve compatibility.
So yes, as long as the final binary links only to glibc, or to other libraries which follow a similar symbol versioning scheme to ensure that older binaries will still link properly to newer versions of the library, it is perfectly valid to build against an older version and then run it on newer versions of various Linux distros, and even generally across distros as well.
Unfortunately, there's no good way to get the linker to pick the older symbol versions if linking against a newer glibc, so frequently the easiest way to do this is within a Docker or other type of container containing an older distro that has the oldest glibc that you want to be compatible with.
I'm quit new in using python. The current version I'm using is 2.7. I need to employ function mncontour in minuit which requires the installation of natgrid as additional toolkit for matplotlib. I downloaded natgrid 0.2.1 with a file named setup.py in it. I ran this setup.py through python shell without reaching any error. But it seems that the installation was not succeed. Anyone has any idea how the installation can be done? Many thanks.
Liang
Could you please provide some more information on this topic.
what operating system you are working with ( some offer more support than others for python)
have you installed python headers, and a C/C++ compiler in your environment? ( numerical libraries might require native code to speed up the computation)
Have you tried a package manager for python ( such as easy or pip)? Both work on both windows and unixes, and usually download and install all the needed packages to make your module working.
A piece of the setup's output before setup.py finished would help us help you greatly.
Python for Windows [Nt - 7] is compiled with either cygwin or mingw, thus you not only need the python environment, but also said compiler, and python headers. If you want a more point and click install, then there's this professor at this university who maintains a good and up to date repository of scientific python modules, that depend on native extensions ( among which numpy, scipy, matplotlib).
http://www.lfd.uci.edu/~gohlke/pythonlibs/
Manuals to setup mingw and python :
https://docs.python.org/2/using/windows.html
MingW can be downloaded from here:
http://www.mingw.org/
Best option for installing natgrid is from conda
conda install -c jochym natgrid=0.2
I'm trying to compile and use the PyGRIB module. There is no binary distribution of the module, so I have compiled using Cygwin. I would really like to be able to use the module in my windows python installation -- I already have numpy, matplotlib, and a development environment setup for my windows installation. How do I do this?
It looks like the Cygwin install creates the following two files:
pygrib-1.9.3-py2.6.egg-info
pygrib.dll
in my c:\cygwin\lib\python2.6\site-packages directory.
I have tried copying these to: C:\Python27\Lib\site-packages but that doesn't seem to do the trick.
If I can't do this, can I get IPython in Cygwin? I haven't seen it in the setup utility.
Honestly, the easiest way to compile on windows when using python is to just use the free visual studios distribution. I've installed many different packages that way and never had an issue. Normally, the installation will place the path variable on your path, but you will need to verify that.
You need to make sure to use 2008 though, and not 2010.
You can retrieve it from here:
http://msdn.microsoft.com/en-us/express/future/bb421473
Do note, if you go this way, it will mean you will have to reinstall any other compiled python binary packages (numpy, scipy, etc)
That said, I notice you are downloading a 2.6egg to a 2.7 distro. Off the top of my head, I'm not certain that 2.6 and 2.7 were compiled using the same compiler, but I believe they were. In any event, that could be your problem, either the package doesn't support 2.7, or 2.6 doesn't compile with the same compiler as 2.7.
Short Question
Is there any way to control / guarantee the architecture (32bit vs 64bit) when building a pyinstaller executable?
Background
I migrated from py2exe to pyinstaller because of the lack of 64bit support along with a host of small things that I am having a hard time looking past. So on that note, I would prefer not to go back to it. I have developed two applications using Python 2.7 64bit and am having performance issues when running on them 32 bit machines.
The first is a simple wxPython GUI (version 2.9) and connects to a windows DLL file for a USB driver. This one seems pretty "safe" to run as 32 bit because there are no modules which are 64bit only. However this application when running on 32bit Windows XP has horrible performance issues when talking to the USB device.
The second application is much larger and I have not attempted to build and run yet because of the fear of architecture issues. This application has a number 64bit only modules (psycopg2 for one) used in it. I would like to stay away from trying to build this if it impossible to run as a 32bit executable.
Current Thoughts
I feel that this might be possible (if the modules have 32bit support) by running the build.py with Python forced in 32bit mode. Does this make any sense?
Update
I had several breakthroughs on the first program I was building. It turns out the performance issues was solely based on the speed of the two machines. My dev machine had enough power to poll the USB device fast enough and the much slower test platform (Windows XP) did not.
I fixed this issue by modifying the way I polled the USB port. Now that this was fixed, I could run the exe on both systems. A new problem had come up when trying to build the executable as a single file. When running pyinstaller's Build.py, it pulls in all of the required DLL's the app needs to run. This seemed to work great at first, but when I tried to run the single exe that I built on Windows 7 64bit, it would not run on Windows XP because the USB dongle's DLL was not recognized as a valid DLL.
In order to get the single exe to run on both systems, I first tried to remove the DLL from the .spec file (which appears to be a python script). It was convenient because I was able to modify the list of includes prior to the build command with ordinary python list modifiers. My hope was that if the DLL was not found in the exe's temp directory it would find it on the system PATH. While this approach might work, I could not get it to run without throwing lots of errors.
My second attempt was to build the application on the Windows XP machine (leaving the DLL embedded) in hope that the Win XP DLL would work in Windows 7. Success! This configuration works well; however I do strongly believe that this not the best solution as it depends solely on the older DLL running on a newer OS.
Pyinstaller produces a binary depending from the python you used to build it. So if you use python 2.7 64 bit it is not possible, as far as I know, to produce a 32 bit executable. This is because Pyinstaller archives all modules and their dependencies (dlls, pyds etc..) which are 64 bit due to the python install.
As already said it is better, because of cross compatibility issues, to build 32-bit binaries.
Probably you can specify more your question.
If you are building an application and it runs fine on 32-bit Windows, there is no need to create a 64-bit version. Just create a 32-bit version and run it on both architectures. What is what WOW64 is for.
If you need to use a library or feature which is 64-bit only, just build a 64-bit version. There is no point in building a 32-bit version if the feature is 64-bit only.
The only reason to build a 64-bit and 32-bit version both, is to take advantage of increased address space of 64-bit windows. I.e. if you intend to allocate more than 1 or 2 GB of memory. An example might be an image editing application, or a data manipulation application. Then you can run on 32-bit platforms within the constraints of the platform but edit larger images or larger quantities of data on 64-bit platforms.
IOW, for your case follow the suggestion of #Velociraptors and build in 32-bit python if you are building a 32-bit exe.
If you want to build a 32-bit application on a 64-bit system and hav only a 64-bit version of Python installed, you have to install also a 32-bit version of Python. Then you have to instal pyinstaller to that 32-bit Python version as follows:
path/to/32-bit/python -m pip install pyinstaller
After that you can run the 32-bit version of pyinstaller, and thus build a 32-bit application, like this:
path/to/32-bit/pyinstaller your-app.py
On Windows 10, pyinstaller.exe is located at C:\Users\<user>\AppData\Local\Programs\Python\Python<version>-32\Scripts\pyinstaller.exe.
As you can see, the architecture of the application you want to build depends on the Python and pyinstaller versions which you are going to use to build that application. This way you can also, for instance, build an application from a Python version 2.x if you have it installed on the system.