Where can I put the CUDA DLLs for Tensorflow? - python

On my work computer, I do not have permissions to install new software, edit PATH, etc. I have Python installed to local folder (and everything works, including pip to install and update packages. But, I cannot install the Cuda Computing Toolkit.
On another computer that I have those permissions, I have tensorflow installed, verified, and by using ProcessExplorer I know what 3 DLLs I need to have installed for tensorflow to work with my Python app.
Typically, when I have this problem, I have found it is acceptable to install DLLs in the local path relative to the binary loading/requiring them; for instance: I put the Visual Studio runtime DLL in the same folder as python.exe and I do not need to actually run the installer.
However, I cannot figure out where the cuda DLLs need to go relative to tensorflow for them to be loaded. I have searched the source with no luck. I have set breakpoints in a Python debugger with no luck. Can anybody point me to where the DLLs need to go, or at least where they are loaded?

Related

PyCharm cannot install packages

Short description: two computers in the same network, in the new one only those python scripts work that use native packages.
I have Pycharm in my old computer and it has worked fine. Now I got a new computer, installed the most recent version of Python and Pycharm, then opened one of my old projects. Both the old and the new computer are in the same network and the project is on a shared folder. So I did the following:
File - Open - selected the project. Got a message that there is no interpreter
Add local interpreter - selected the latest Python 311 exe. So location of the venv is the same as in the old computer (because it's a network folder) but Base interpreter is pointing to the C drive of my new computer.
PyCharm creates a virtual environment and the code runs fine.
I select another project which uses imported packages such as pandas. Again, same steps as above, add local interpreter. Venv is created.
I go to File - Setting - Project and see that pip, setuptools and wheel are listed as Packages. If I double click one of these, I can re-install and get a note that installation is succesful, so nothing seems to be wrong in the connection (after all, both the old and the new computer are in the same network.
I click the plus sign to add a new one, search pandas. Installation fails. Same thing if I try e.g. numpy.
Error message has lots of retrying, then "could not find the version that satisfies the requirement pandas (from versions: none", "not matching distribution found for pandas" (pip etc. have the latest versions).
After few hours of googling for solutions, I have tried the following:
Complety uninstall and reinstall python and PyCharm. Checked that PATH was included in the installation.
Tried launching pip command from shell
Changed http proxy to auto-detect
Typed 'import pandas' in PyCharm, then used the dropdown in the yellow bulb but there is no install option
Started a new project in the new computer, tried to install pandas
All failed. I'm surprised that changing computers is this difficult. Please let me know if there are other options than staying in the old computer...
If you want to use venv in the network, please use SSH interpreter. Pycharm supports this method. Shared folders are not a recommended usage, For pycharm, it will consider this as a local file. If the file map is not downloaded locally, it will make an error.
Another way is to reinstall the project environment on the new computer through requirement.txt. Reasonable use of requirements.txt can effectively avoid many project bugs caused by environment migration or different dependent versions. Before installing some scientific module such as pandas, it is recommended to install visual studio build tools, such as gcc ...
This took a while but here is what happened. Package installation did not work in project settings. Neither did it work when you select Python Packages tab at the bottom of the screen. The only thing that worked was to select the Terminal tab and manually install (pip install) there. We use a trusted repository but for other users, the easier package installation methods work. Not sure why they do not for me but at least there is this manual workaround.

How to deal with missing libraries on Linux? [No Admin privileges and PIP is blocked]

For my work I need to write a GUI using PySide6 for a remote system. The OS is RHEL 7.9 and I have neither admin privileges nor PIP working (blocked by admins), so i can't install anything by myself (and i'm not allowed to anyways).
The script runs perfectly on Windows and Fedora, but it doesn't work on RHEL 7.9:
Since the machine doesn't allow pip, I've included PySide6 in my virtual environment, but there are missing libraries in the system itself, like CXXABI_1.3.9 and GLIBC_2.33 that Shiboken6 needs.
It also didn't work in compiled form (with PyInstaller) because the GLIBC_2.29 is missing.
Naively I copied libstdc++.so.6 and libc.so.6 from a Fedora machine to RHEL and redirected the linking to the libraries with the LD_LIBRARY_PATH environment variable, but because of other dependencies it didn't work.
Is there a solution to make the script work cross-platform and independently?
This won't answer your question about the missing libraries, but I hope it helps solve your current issue with PySide.
I've had a similar problem before, you should always develop on the target platform to get comparable results.
This means that you theoretically have to write, compile and package your program on the RHEL machine. You also need to always develop on the older platform. Forward compatibility is not always guaranteed. I therefore suggest, that you install CentOS 7 in a virtual machine and if your program is not too complicated try to use PySide2 instead of PySide6.
One way may be to put the (binary) files in the user bin.
The path for that (at least on my system) is /home/.local/bin
Make sure this is in your $PATH variable, if it isn't already (it should be).
If these are just Python modules, you can just drag the source file into the aforementioned path (make sure they have a shebang so they run). If not, you may have to compile them and then put them there.
Explaination of local bin
Note: This is from my experience
The local bin (/home/.local/bin) is where you can put programs you want to run without installing them globally. It's similar to doing ./file.py, except you don't need to be in the same directory as the file (and you don't have to include ./) I assume, if you include a compiled version of the dependencies you want, it should work.

How do I access OpenCV source compiled package from my Python project venv on windows?

So I have been writing a Python program that utilizes OpenCV for Windows. It's mostly just a project to learn how to use Vision-based Machine Learning. I've gotten the project working with my CPU, and while it "functions" it is abysmally slow, so I wanted to try working with the GPU version instead. Unfortunately, there isn't a python package that has OpenCV with Cuda-enabled GPU functionality (from what I could tell).
So after researching, I found that in order to do what I wanted, I had to compile the OpenCV source code with Cmake. So I set out to do so with the help of this guide which seemed to work (it compiled at least). But now i'm running into the issue of:
I don't actually know how to import this newly built package into my project's venv. I've tried moving the cv2.cp37-win_amd64.pyd into the project's venv site packages, i've tried moving the entire build folder into the project and directly importing it, but neither actually worked... so I'm a little at a loss on what to do to get OpenCV with GPU enabled CUDA working in my project.
EDIT:
I followed the guide that Miki linked, including installing Anaconda and all that jazz. The package imports correctly when I use:
Following #Miki's suggestion, I went through the guide that they linked, following all of the processes, testing that OpenCV was in fact being built correctly with:
set path=%openCvBuild%\install\x64\vc16\bin;%path%
python -c "import cv2; print(f'OpenCV: {cv2.__version__} for python installed and working')"
But it still isn't showing up in the python import in conda, despite being in the env's site-packages folder.

Cannot Open Python. Error 0xc000007b

I have recently been learning Python 3. I have had no problems on my netbook (32bit Windows 7) I with creating small simple programs. When I installed it on my netbook I had no problems, but now that I've been getting into it I want to install it on my desktop and I have one.
My desktop is a 64 bit OS running Windows 7. Like I did with my netbook, I went to download Python and install the 64 bit edition.
After installing it, I went to open up Python CLI and GUI and no luck. It gave me this error.
"The application was unable to start correctly (0xc000007b). Click OK to close the application"
I also tried reinstalling, but not luck.
I suspect a missing .dll but I am not completely sure. Does anyone know what is keeping me from opening Python and how do I fix it?. I want to resume my work as soon as possible.
That is the NTSTATUS error code for STATUS_INVALID_IMAGE_FORMAT. Almost invariably that means that the loaded is attempting to link a 32 bit DLL into a 64 bit process, or vice versa.
If you want to attempt to debug this then the tool of choice is Dependency Walker. You open the python.exe executable file in Dependency Walker and use the profile menu to start it up with logging of the load events. Somewhere along the way there will be a failure and you'll be able to see which module is causing that failure. That will hopefully point the way towards a resolution.
However, attempting to debug this may be a waste of time. It will probably better to uninstall the existing installation, and try to re-install. Perhaps picking a different Python distribution.
I note that you have selected a 64 bit version. Now, support for 64 bit extension modules is not as strong on 64 bit Python as on 32 bit Python. It does exist, but they modules can be harder to come by and install. You might consider installing a 32 bit Python next time round.
Please make sure that Python3.dll is the same version and the same architecture (32/64) with python.exe. I fixed this error via replacing it with 32bit version.
you should delete the old path of python in "Environment Variables" and enjoy ;)
As janne-karila mentioned, and for Python 3.5, Microsoft Visual Studio 2015 redistributable (in this case the 64 bit version)
This probably won't answer your question, but my answer is for a very specific case when you "Refresh Windows without affecting Files" to fix your Windows 8.1 PC.
If you do that, You'd get this error when you try to run Python after the windows refresh.
To fix this error, simply re-install PY at the same folder where your python was install before you refreshed windows. If you try to install it any other folder like username/appdata/... folder the PY installer would be stuck in initialization.
You may also get another error saying "api-ms-win-crt-runtime-l1-1-0.dll is missing". To fix, go to the windows.old folder and get the dll file ith the above name and paste it in the PY folder. Also like others have suggested, make sure Microsoft Visual Studio 2015 redistributable is installed.
Encountering this error after a windows refresh is so rare, that no one else seemed to have experienced according to google searches, so I though I share it here for that rare unlucky person.
Apparently the error code stands for STATUS_INVALID_IMAGE_FORMAT (source), and it looks like it's not specific to Python. You can try reinstalling the program, rebooting, running CHKDSK /r (source). It might also have something to do with the Microsoft Visual C++ redistributable package, as Janne Karila and this thread suggest.
This stackoverflow question also looks like it has some useful information.
I too faced this problem. So I uninstalled python3 64-bit version and installed 32-bit version, after that i am able to launch python3 in windows 8 successfully. (i have a 64-bit laptop).
Simply, download the last version of Microsoft Visual C++ redistributable package ..
I had the same problem ..
I have the same error found. Using dependency walker, I found tons of DLLs missing. Rather going for copying all the DLLs I went with virtual environment. It will create another environment for Python.
First, install python normally. Copy, the Python.exe path. In my case, it is
C:\Python\python
It will be used to install Python libraries. Then, using command prompt, I used pip to download and install virtualenv (Virtual environment). To do that, change the directory to where pip.exe is, which is in Scripts folder. Now, run the following to install virtualenv:
C:\Python\python pip install virtualenv
It will install Virtual environment.
Now, create a folder to hold virtual environment and change directory to that folder and run to create the following to create an environment:
virtualenv --no-site-packages [environment name]
It will create the Environment.
Now, change subdirectory to Scripts in the [environment name] folder. There is a activate file run it, in my case it is like:
C:\Python\virtualenv\env\Scripts activate
env is the [environment name] here.
If you activate it, every command will look like:
(env) C:\Python\python
This ends here. Now, you can install any python library using this. Do whatever, you want using Python.
Also, to deactivate the environment same deactivate.bat is in the scripts folder.
For me the issue was, I had installed 32 bit version on a 64 bit machine. Hope it helps someone out there.
Like many have said, Windows cannot load python.exe itself or one of its dependent DLLs because of INVALID_IMAGE_FORMAT:
python.exe is of unsupported bitness. 32bit on 64bit Windows is supported. 64bit on 32bit system is not (solution: install 32bit python, bye).
python.exe or one of its DLLs is corrupted (probably not)
one of the dependent DLLs is not of the same bitness as the EXE:
python.exe (say, 32bit) requires python*.dll (python3.dll or python27.dll). It installs that into Windows System32/SysWOW64 folder. The DLL is of correct bitness (32bit).
Something else overwrites that with 64-bit version, or places a 64-bit version somewhere in the search PATH.
When you run python.exe (32bit), Windows searches for python*.dll, and the first one it founds is the 64bit one. 32bit EXEs cannot normally load 64bit DLLs.
Typical examples:
TortoiseHg installs its own copy of python (both .exe and .dll) into its own folder and adds itself to PATH. The bitness can be different (say, 64bit) and usually matches platform bitness.
Another installation of Python with a different bitness somewhere else overwrites the DLL (it shouldn't, they have different System32s, but who knows)
Diagnostics:
Dependency Walker marks 64bit DLLs with 64bit icons. Verify that icons for python.exe and python*.dll match. Right click -> "Show full paths" to see where the DLL is taken from.
Simple solutions:
Maybe you have another, functioning copy of Python somewhere else? Check it.
Reinstall Python, preferably in the platform version (64bit for 64bit OS, 32bit for 32bit) Yeah, that's what many are saying, but now you know why this is needed.
Remove offenders from PATH. While this maaaay fix your Python, this will break the offenders and some (TortoiseHg explorer plugins) need the PATH. So maybe compromise with them and choose #2.
In my case I was installing both the 32 and 64 bit versions of the python msi. They both install to the same folder, c:\python32. I thought msi's would prevent those kinds of conflicts, but no.

Manual installation SciPy, NumPy, MatPlotlib (Windows)

I want to install SciPy, NumPy and MatplotLib globally so that the libraries can be accessed from network computers. Basically it should be in some network drive without full installation of the libraries on each remote computer.
Now I copied the site-package directory from one working installation into the network drive, but not surprisingly I now get "DLL load failed" (multiarray) errors when trying to import scipy from remote computers. I suppose there are DLLs missing.
What would be the easiest solution to this problem? Does anyone know which DLLs I need to copy to all remote computers or can I even have these DLLs on the network drive too inside PYTHONPATH?
In the docs, see : http://docs.python.org/install/index.html
"Alternate installation: Windows (the prefix scheme)"
"Modifying Python’s Search Path"
"Custom Installation"
may help you.
There is also the PYTHONHOME var used during install.
But frankly, I'm afraid you will not succeed doing this because the packages are compiled during installation, and compilers depends on CPU, vary from one PC to another (one has the minGW, other has the intel, third one has the borland ... ) and I think python packages do not use as much place, so that it makes sense to install on each computer, more than trying to deal with ONE install for everyone.
Unless you make sure everybody has the same C and fortran compilers... but even so, it will not be easy, i'm afraid.

Categories