I am attempting to use the PyMultinest package. The full error text I am encountering is AttributeError:dlsym(RTLD_DEFAULT, run): symbol not found in the __getitem__() function in ctypes/__init__.py. I'll include more text and code below, but I am mostly trying to understand what this error is telling me - my Google Fu is apparently lacking, and the StackExchange questions I have seen relating to this error seem to be hyper-focused on solving a specific instance of this error. So - What is this error trying to tell me is wrong?
More context. I attempt to execute the PyMultinest (PMN) package as directed in the PMN documentation. PMN is, effectively, a Python wrapper for a C program. Running PMN requires a fair bit of setup code (several ancillary functions need to be defined, as well as a host of variables), which I'm not including here by default because it's ... a lot, but I can if needed. The PMN execution line I use is
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
This returns the error traceback
File "[redacted]", line 139, in <module>
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pymultinest/run.py", line 254, in run
lib.run(*args_converted)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, run): symbol not found
If it helps, I have determined that the name variable being passeed through self.__getitem__(), and thus into self._FuncPtr(), is run. Although, that might be obvious looking at the AttributeError message.
I am running Python 3.8 (as shown above) on a MacOS machine. Last summer, I was able to execute PMN on this machine using extremely similar code to that which I am using now. I am currently trying to optimize my code from last summer, which is why I'm surprised it isn't simply "working".
So far my attempts at fixing this have been mostly centered on reinstalling PMN. I have done a clean install (pip uninstall/pip install) of the PMN package, as well as following the PMN documentation to rebuild the C portion of the package. I have included the source directories of the C software in my Path variables - or at least, I tried to, I am assuming I was successful, but I'm not very familiar with Macs.
Ultimately, I just wish I understood what Python was telling me with this error better. It would help me direct my own attempts at solving the issue. I suspect it is saying "We don't know where to find this 'run' command you're asking for," in which case I need to figure out why my Path variable changes aren't working. Am I on the right path?
Some hints how to resolve this are given in these issues:
https://github.com/JohannesBuchner/PyMultiNest/issues/160
https://github.com/JohannesBuchner/PyMultiNest/issues/163
It is important to cleanly recompile (empty build folder) when the environment changes, so that cmake recognises changed libraries and compilers.
Check if you are running everything with python3 and not python (works as python 2.x is you have both version 2 and 3 installed)
Check is all the PyMultiNest python files are rewritten in python 3 style, e.g. $ print "something" becomes $ print("something")
My supervisor was accidentally running on python 2.7 and got the same error, so your error might as well be due to mismatch of python version usages.
Run command flutter pub cache clean and flutter clean and after that run flutter pub get now relaunch your emulator it worked for me.
Related
So, I've been a dev for about 3 years, primarily working in TypeScript and NodeJS. I'm trying to broaden my skillset by learning python (and eventually expanding my learning of Computer Vision, Machine learning, etc.) but I feel incredibly frustrated by trying to get Python to work consistently on my machine. Surely I'm doing something wrong, but I just can't really understand what it is.
I've run into these problems mostly when using ML packages (TensorFlow, Whisper, OpenCV (although I was eventually able to resolve this), so I don't know if it's related to M1 support of one of the common dependencies, etc.
My current understanding of python is that:
M1 support of python is version dependent at best.
venv is the only environment manager I should need to use
I should use pyenv to install python versions so as to not conflict with os-installed python (macos dependencies)
I'll use the latest project I'm working on as an example.
My Machine and Environment
Mac M1 Pro, MacOS Monterey 12.6
pyenv 2.3.9
Python 3.7.13
Fish Shell version 3.5.1
My general workflow to get a project started:
Create virtual environment using venv
python3 -m venv <some_environment_name>
Open created directory using VSCode, and activate the virtual environment
Here is where I encounter my first issue, which seems to be persistent.
➤ source /Users/paal/src/whisper_transcription/bin/activate.fish
functions: Function '_old_fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 18):
functions -c _old_fish_prompt fish_prompt
^
in function 'deactivate' with arguments 'nondestructive'
called on line 30 of file ~/src/whisper_transcription/bin/activate.fish
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
functions: Function 'fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 47):
functions -c fish_prompt _old_fish_prompt
^
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
fish: Unknown command: _old_fish_prompt
~/src/whisper_transcription/bin/activate.fish (line 71):
_old_fish_prompt
^
in function 'fish_prompt'
in command substitution
(whisper_transcription)
So, what I do to resolve this is add the following if statement to the fish.config file.
if type -q $program
_old_fish_prompt
end
Looking at GitHub issues, this seems to be a persistent issue for fish shell and this seems to at least temporarily resolve it.
Or, I just switch to ZSH.
OK, so with that resolved I move on. The environment is activated, I'm using ZSH now, and I can successfully run a python script that prints "hello world" to the console.
Then comes the nightmare of installing any packages. It seems like any project I start has some weird edge case of compatibility issues. Between M1 processors, Python Versions, builds not working correctly, etc.
for example..
import whisper
... # rest of file
This with any other code or even by itself throws the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
import whisper
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/__init__.py", line 12, in <module>
from .decoding import DecodingOptions, DecodingResult, decode, detect_language
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/decoding.py", line 514
if prefix := self.options.prefix:
^
SyntaxError: invalid syntax
This appears to be some problem with the Python Version. From what I understand, the := operator isn't valid syntax until Python 3.8. However, dependencies of whisper (PyTorch) only seems to be supported up to version 3.7.9
So, you can see, it seems like I just end up in these bizarre circular problems where some dependency of some package I want to use isn't supported by either the platform or the current python version, and they seem basically unsolvable (at least with my current knowledge of python)
Why is this seemingly so complicated? I'm clearly doing something wrong here, and obviously I'm out of my comfort and knowledge zone, but these issues feel very daunting and opaque, and difficult to actually troubleshoot this in any consistent or clear way.
Is there a resource that makes this stuff more clear? Is Python development on M1 chips just this broken at the moment? How can I get past these seemingly basic issues to start actually learning?
I'm not necessarily looking for a solution to this specific problem here, but if theres general advice about environment management and how to make things work somewhat reliably, I'm fine troubleshooting. I just feel like every time I start trying to learn, I end up in these rabbit holes that take hours and hours to fix and sometimes don't even really resolve things.
Venv File I have a project that utilizes YoloV5, OpenCV, as well as many other python libraries. After finally finishing work on the project, I wanted to share it with my teammates. However, I was unable to get my code to run anywhere other than my machine. There was a Venv file that was generated automatically somewhere along the way, either by my IDE or a jupyter notebook. I tried giving out this file to others, but it is not only massive (~1GB), but it also didn't work. I will consistently get syntax errors that do not exist in my environment. For example, I will get:
"SyntaxError: Non-ASCII character '\xf0' in file detect.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
"
If I remove that line, I get:
" File "detect.py", line 120
model.warmup(imgsz=(1 if pt else bs, 3, *imgsz)) # warmup
^
SyntaxError: invalid syntax
"
Mind you, none of these errors exist in my environment.
I have tried looking up various commands to somehow link the venv file to the environment or to the python file. I thought that, perhaps, python wasn't looking for the file, so it wasn't using it. I also tried installing other packages and installing all the dependencies manually. However, none of these things worked. I am a little lost on where to go from here, I have heard of tools like Conda, which apparently might help manage things like this, but I am wondering if anyone could give advice on how to proceed, or maybe if anyone else has personally used YoloV5 and encountered this problem themselves
So I am trying to write some python code that will do two things, that seem to be mutually exclusive on my machine. My PC's host operating system is windows and I run Kali-Linux in WSL2 when I need to test my code on Linux. My code's main function creates two separate multiprocessing.Process objects, assigning a different thread, starting them both one after the other and then calling for them both to be joined. The plan is to allow each to run a simple server application simultaneously on different ports. This does not work when running python3 in PowerShell, as it seems to require access to os.fork() which doesn't work in said environment. When I found this out I pivoted to running in WSL2 which worked fantastically, for a time. After a while of experimenting with some ideas I decided to take some of my code and spin it off into its own file, which I placed in its own 'Libs' folder. WSL2 however, was unable to import this new file, instead giving me the exception ModuleNotFoundError: No module named 'NetStuff'. I originally had added:
sys.path.append('./Libs')
as has worked for me in the past, however when I found that WSL2 was unable to find my module, I printed out sys.path and it revealed that rather than appending my $current_working_directory/Libs like I intended, I was just appending the literal string, which wasn't useful. I then decided to try:
sys.path.append(str(pathlib.Path().resolve()) + '/Libs')
which at the bare minimum shows up as I would expect in sys.path. This, however still didn't work, python was unable to find my module and would unceremoniously crash every time. This led me to try something else, I ran my code in python3 under PowerShell again, which had no issue importing my module, it did still crash due to lacking os.fork() but the import gave no issues. Confused and annoyed I opened my code in IDLE 3.9 which, for some inexplicable reason, was able to import the file, and seemingly use os.fork(). The only major issue with running in IDLE is that it is seemingly incapable of understanding ascii colour escape characters. Given that the goal is to run my code in bash, and ideally also PowerShell, I am not satisfied with this as a solution. I returned to trying to fix the issue in WSL2 by adding my module to /home/Noah/bin, and appending this directory to sys.path, but this has still not so much as given me a new symptom.
I am utterly at a loss at this point. none of the fixes I know off hand are working, and neither are the new ones I've found online. I can't tell if I'm just missing something fundamental about python or if I'm running into a bug, if it's the latter, i can't seem to find other people with the same issue. As a result of my confusion and frustration I am appealing to you, kind users of stackoverflow.
The following is the snippet that is causing me problems in WSL2:
path0 = ('/home/Noah/bin')
path1 = (str(pathlib.Path().resolve()) + '/Libs')
sys.path.append(path0)
sys.path.append(path1)
print(sys.path)
import NetStuff
The following is output of print(sys.path) in WSL2:
['/mnt/c/Users/Noah/VSCodeRepos/Python/BlackPack', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/home/noah/.local/lib/python3.9/site-packages', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/home/Noah/bin', '/mnt/c/Users/Noah/VSCodeRepos/Python/BlackPack/Libs']
The following is the error being thrown by WSL2:
Traceback (most recent call last):
File "/mnt/c/Users/Noah/VSCodeRepos/Python/BlackPack/BlackPackServer.py", line 21, in <module>
import NetStuff
ModuleNotFoundError: No module named 'NetStuff'
I am specifically hoping to fix the issue with WSL2 at the moment as I am fairly certain that getting the code to run on PowerShell is merely going to require rewriting my code so that it doesn't rely on os.fork(). Thank you for reading my problem, and if I left out any information that you would like to see just tell me and I'll add it in an edit!
Edit: I instantly realized that I should specify that my host machine is running windows 10.
I’m having trouble using python’s multiprocessing module. This is the first time I’ve tried using the module. I’ve tried simplifying my processing to the bare bones, but keep getting the same error. I’m using python 2.7.2, and Windows 7.
The script I’m trying to run is called learnmp.py, and the error message says that the problem is that it can't find module learnmp.
import multiprocessing
def doSomething():
"""worker function"""
print 'something'
return
if __name__ == '__main__':
jobs = []
for i in range(2):
p = multiprocessing.Process(target=doSomething)
jobs.append(p)
p.start()
The error is :
File “<string>”, line 1, in <module> File “C:\Python27\ArcGISx6410.1\lib\multiprocessing\forking.py”, line 373,
in main prepare(preparation_data) File “C:\Python27\ArcGISx6410.1\lib\multiprocessing\forking.py”, line 482,
in prepare file, path_name, etc = imp.find_module (main_name, dirs)
ImportError: No module named learnmp
What’s causing the error, and how can I solve it?
EDIT: I still don't know what was causing the error, but changing the file name eliminated it.
I know it's been a while, but I ran into this same error, also using the version of Python distributed with ArcGIS, and I've found a solution which at least worked in my case.
The problem that I had was that I was calling my program name, Test.py, as test.py. Note the difference in case.
c:\python27\arcgisx6410.2\python.exe c:\temp\test.py
c:\python27\arcgisx6410.2\python.exe c:\temp\Test.py
This isn't normally an issue if you're not using the multiprocessing library. However, when you write:
if __name__ == '__main__':
what appears to be happening is that the part of the program in main is being bound to the name of the python file. In my case that was test. However, there is no test, just Test. So although Windows will allow case-incorrect filenames in cmd, PowerShell, and in batch files, Python's multiprocessing library balks at this and throws a nasty series of errors.
Hopefully this helps someone.
Looks like you might be going down a rabbit-hole looking into multiprocessing. As the traceback shows, your python install is trying to look in the ArcGIS version of python before actually looking at your system install.
My guess is that the version of python that ships with ArcGIS is slightly customized for some reason or another and can't find your python script. The question then becomes:
Why is your Windows machine looking in ArcGIS for python?
Without looking at your machine at a slightly lower level I can't quite be sure, but if I had to guess, you probably added the ArcGIS directory to your PATH variable in front of the standard python directory, so it looks in ArcGIS first. If you move the ArcGIS path to the end of your PATH variable it should resolve the problem.
Changing your PATH variable: http://www.computerhope.com/issues/ch000549.htm
Microsoft Visual C++ 9.0 is required for some python modules to work in windows,so download below package it will work.
http://aka.ms/vcpython27
This package contains the compiler and set of system headers necessary for producing binary wheels for Python 2.7 packages.
I have written a sudoku solver using especially, but not only, tkinter and multiprocessing. I want to make it a standalone app, which I'm using py2app for.
I've run py2applet-3.3 --make-setup sudoku_app.py (that is what it's called when installed through macports), and it succeeded at that. However when building with python3.3 (python3.3 setup.py py2app), it returns a SyntaxError with an almost endless stack trace that ends like this:
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/modulegraph/modulegraph.py", line 954, in load_module
co = compile(contents, pathname, 'exec', 0, True)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/sympy/mpmath/libmp/exec_py2.py", line 2
exec string in globals, locals
^
SyntaxError: invalid syntax
When running the same setup file with python2.7, it does succeed to build, but there's a bunch of errors when trying to run the created .app. I've written my code in python3, although trying to maintain python2 compatibility through various try ... except - blocks and from future imports. At some point though, it got too complicated and I decided to just go with python3.
Why does py2app not work with python3 there?
Your application doesn’t work with py2app (yet) because I didn’t consider the way you do py3k support. In particular, modulegraph assumes that all modules in the dependency graph contain valid python code for the current python version.
Could you file a bug at my bitbucket page (https://bitbucket.org/ronaldoussoren/py2app) to ensure that I don’t forget to fix this?