We Keep Coding

AttributeError: dlsym(RTLD_DEFAULT, run): symbol not found - meaning

I am attempting to use the PyMultinest package. The full error text I am encountering is AttributeError:dlsym(RTLD_DEFAULT, run): symbol not found in the __getitem__() function in ctypes/__init__.py. I'll include more text and code below, but I am mostly trying to understand what this error is telling me - my Google Fu is apparently lacking, and the StackExchange questions I have seen relating to this error seem to be hyper-focused on solving a specific instance of this error. So - What is this error trying to tell me is wrong?
More context. I attempt to execute the PyMultinest (PMN) package as directed in the PMN documentation. PMN is, effectively, a Python wrapper for a C program. Running PMN requires a fair bit of setup code (several ancillary functions need to be defined, as well as a host of variables), which I'm not including here by default because it's ... a lot, but I can if needed. The PMN execution line I use is
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
This returns the error traceback
File "[redacted]", line 139, in <module>
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pymultinest/run.py", line 254, in run
lib.run(*args_converted)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, run): symbol not found
If it helps, I have determined that the name variable being passeed through self.__getitem__(), and thus into self._FuncPtr(), is run. Although, that might be obvious looking at the AttributeError message.
I am running Python 3.8 (as shown above) on a MacOS machine. Last summer, I was able to execute PMN on this machine using extremely similar code to that which I am using now. I am currently trying to optimize my code from last summer, which is why I'm surprised it isn't simply "working".
So far my attempts at fixing this have been mostly centered on reinstalling PMN. I have done a clean install (pip uninstall/pip install) of the PMN package, as well as following the PMN documentation to rebuild the C portion of the package. I have included the source directories of the C software in my Path variables - or at least, I tried to, I am assuming I was successful, but I'm not very familiar with Macs.
Ultimately, I just wish I understood what Python was telling me with this error better. It would help me direct my own attempts at solving the issue. I suspect it is saying "We don't know where to find this 'run' command you're asking for," in which case I need to figure out why my Path variable changes aren't working. Am I on the right path?

Some hints how to resolve this are given in these issues:
https://github.com/JohannesBuchner/PyMultiNest/issues/160
https://github.com/JohannesBuchner/PyMultiNest/issues/163
It is important to cleanly recompile (empty build folder) when the environment changes, so that cmake recognises changed libraries and compilers.

Check if you are running everything with python3 and not python (works as python 2.x is you have both version 2 and 3 installed)
Check is all the PyMultiNest python files are rewritten in python 3 style, e.g. $ print "something" becomes $ print("something")
My supervisor was accidentally running on python 2.7 and got the same error, so your error might as well be due to mismatch of python version usages.

Run command flutter pub cache clean and flutter clean and after that run flutter pub get now relaunch your emulator it worked for me.

Requiring PsychoPy fails readthedocs build due to memory consumption

I have a project (https://github.com/jfkominsky/PyHab) which is essentially an extension of PsychoPy (http://psychopy.org/). I want to use Sphinx and ReadTheDocs for code documentation. I use the autoclass features from Sphinx for my documentation and have most of it in the code itself, which works great on local builds. On ReadTheDocs, this means that I needed to add a requirements file with psychopy as a requirement. This causes the build to fail in the following way:
/home/docs/checkouts/readthedocs.org/user_builds/pyhab/envs/latest/bin/pip install --exists-action=w --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/pyhab/.cache/pip -rdocs/requirements.txt
Command killed due to excessive memory consumption
I'm very amateur at this. Is there a way to use a setup.py virtual environment to get psychopy to work? Is readthedocs just being too stingy with memory? I've noticed psychopy itself has some issues with RTD builds right now, unsure if that's related...
EDIT: I should add, I was using Cpython2 as my environment in RTD. When I use CPython3 instead I get a different failure:
I tried again with Py3 and got a different error:
ERROR: failed building wxWidgets
Traceback (most recent call last):
File "build.py", line 1321, in cmd_build_wx
wxbuild.main(wxDir(), build_options)
File "/tmp/pip-build-7a1lqn7v/wxPython/buildtools/build_wxwidgets.py", line 374, in main
"Error running configure")
File "/tmp/pip-build-7a1lqn7v/wxPython/buildtools/build_wxwidgets.py", line 85, in exitIfError
raise builder.BuildError(msg)
buildtools.builder.BuildError: Error running configure
Finished command: build_wx (0m10.842s)
Finished command: build (0m10.842s)
Command '"/home/docs/checkouts/readthedocs.org/user_builds/pyhab/envs/latest/bin/python" -u build.py build' failed with exit code 1.

I have devised a solution, at least for CPython2.x. The long and short of it, which RTD needs to advertise MUCH better, is that you can slip a bunch of 'mocks' into conf.py in order to bypass modules you don't want to actually import when compiling your docs. You need mock in your requirements file in 2.7 (presumably unittest.mock in 3.x)
The idea came from here: http://blog.rtwilson.com/how-to-make-your-sphinx-documentation-compile-with-readthedocs-when-youre-using-numpy-and-scipy/
Which connects back to this in the FAQ: http://read-the-docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
I realized that you could create mock modules for literally anything, and if you use MagicMock you can add attributes to them as needed. It took me several tries to get all the right mock modules (mostly because the RTD environment is hard to emulate locally and it would only show you the first module it failed to import in the error message), but eventually, I was able to convince RTD that it had everything it needed without exceeding its memory limits.
This solution is not psychopy-specific. This will work for anything that you have trouble pip installing on RTD.

How do I freeze Celery using py2exe and esky?

I'm trying to freeze a Python application and Celery, packaged up with esky for update capabilities. I need Celery to be frozen so that it has visibility into the modules from the first executable that will also be present in the library.zip file that py2exe creates.
The problem is that I cannot get Celery to freeze with esky. Here's a bare minimum, stripped down setup.py file that I'm using to try and find a solution around the problem:
from esky import bdist_esky
from distutils.core import setup
setup(name='ColdCelery',
scripts=['C:\\Python27\\Lib\\site-packages\\celery\\bin\\celery.py'],
options = {
'bdist_esky':{
'freezer_module': 'py2exe',
}
}
)
When I run the following command:
python setup.py bdist_esky
I get the following error:
running bdist_esky
running build_scripts
*** searching for required modules ***
error: c:\temp\tmpz5146o\scripts\celery.py: The process cannot access the file
because it is being used by another process
There is no running Python process on the machine that could be using Celery. I assume this is a conflict between py2exe and esky, but don't know how to overcome it.
I can freeze Celery using py2exe without referencing esky without a problem, but I will require being able to update this project in the future, so esky support is a must.

I had problems running py2exe and esky in the tutorial
I use cx freeze in my project and i haven't had any problems.
It also supports python3, linux and mac so i can recommend giving it a go.
http://cx-freeze.readthedocs.org/en/latest/

Ultimately, we did not find the reason for the failures that were occurring, but we did pick up on an important point and we changed our environments to get around the issue.
Our build machine did double duty for development. It's not the best idea of course and it was probably one of the many, many packages installed for that development that was causing trouble.
So we started with a bare bones Windows installation and installed only the minimum number of packages needed to actually perform the build. That seems to have done the trick.

I rediscovered our fix for this issue. The error message is misleading - after debugging I found that the actual error was caused by esky adding some lines to the top of celery.py during the packaging operation. These lines were going above a from __future__ import absolute_import, unicode_literals causing the following error:
SyntaxError: from __future__ imports must occur at the beginning of the file
To fix:
Edit C:\Python27\Lib\site-packages\celery\bin\celery.py and strip all comments and empty lines from the top of the file.

ImportError: Bad magic number, since OSX Lion

I'm getting this error every time I run any python file in Eclipse using PyDev:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/site.py", line 73, in <module>
__boot()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/site.py", line 2, in __boot
import sys, imp, os, os.path
ImportError: Bad magic number in /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.pyc
I'm using python 2.6. This problem does not occur when I run python from the terminal (2.7 or 2.6). The only substantial thing I've changed since everything last worked, is an update to OSX Lion from Snow Leopard.
Similar discussions to this seem to suggest some kind of removal of the .pyc file, because of some kind of mismatch between what was originally using the .pyc files (I'm not entirely sure what a magic number is...). But I was a bit cautious of the idea of deleting os.pyc from the Frameworks directory. When the only other file is an os.pyo file (not sure what the difference it), rather than an os.py.
I've installed all OSX Lion updates, Eclipse updates and PyDev updates.
This problem occurs even with code such as :
if __name__ == '__main__':
pass
Any help resolving this would be appreciated!

Upgrading Python to 2.7.1, running "Update Shell Profile" command file which is located in Python directory and changing the Python settings in Netbeans according to new installation worked for me.

Yeah, you'll need to regenerate all your *.pyc and *.pyo files from the *.py files.
How you do this depends on how they were generated in the first place. Some packaging of python (and it's add-ons), such as in some Linux distros, gets a little too clever for its own good and keeps the original *.py files somewhere else and have their own build system for generating and placing the *.pyc and/or *.pyo files. In a case like that, you have to use that build system to regenerate them from the original *.py files.
FYI, here are a couple links on *.pyo files. They are the optimized versions of compiled python modules.

On OS X Lion, you should have a os.py file. This is likely the root cause of your error. The os.pyc file was generated from a different version of python than you are running now. Normally, I imagine the python interpreter would just regenerate the file from os.py. But for whatever reason, your system does not have that file.
I suspect that this is a small data point in a larger set of issues and would, in general, recommend a reinstallation of your operating system.
For comparison, I'm running 10.7.1, and I have the following:
[2:23pm][wlynch#orange workout] ls /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.*
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.py
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.pyc
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.pyo
As an aside, the *.pyo file is an optimized version of the python bytecode.

Python ctypes MemoryError in fcgi process from PIL library

I'm trying to run Django on shared hosting (Bluehost). I'm using functionality that requires PIL. PIL imports and works from an interactive shell, but in my fcgi process it crashes with a MemoryError at from PIL import image. Any help on why it might be failing inside fcgi would be much appreciated.
__Environment Info__:
Python2.7
Local installs of libjpg, zlib, freetype, and lcms
Virtualenv:
Django 1.3, PIL, flup, etc.
__Stack Trace__:
File ".../feincms_thumbnail.py", line 3, in <module>
from PIL import Image
File ".../PIL/Image.py", line 45, in <module>
\__import__("FixTk")
File ".../python2.7/lib-tk/FixTk.py", line 15, in <module>
import ctypes
File ".../python2.7/ctypes/__init__.py", line 549, in <module>
CFUNCTYPE(c_int)(lambda: None)
__.fcgi__:
<!-- language: python -->
# setup paths
# set DJANGO_SETTINGS_MODULE in os.environ
from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")

I have temporarily fixed that error commenting the last line in this file $HOME/lib/python2.7/ctypes/__init__.py that is something like #CFUNCTYPE(c_int)(lambda: None).
That worked for me, but I don't know exactly what is the problem.
UPDATE
In python 2.7.3 the line number is :279 not the last as I said above.
UPDATE 2
Since the line may vary between minor versions, you should look for a chunk of code that looks something like this:
# XXX for whatever reasons, creating the first instance of a callback
# function is needed for the unittests on Win64 to succeed. This MAY
# be a compiler bug, since the problem occurs only when _ctypes is
# compiled with the MS SDK compiler. Or an uninitialized variable?
CFUNCTYPE(c_int)(lambda: None)

try running this command:
setsebool -P httpd_tmp_exec on
fixes things for me on CentOS. Taken from this post:
https://bugzilla.redhat.com/show_bug.cgi?id=645193

Just to expand on eos87's answer a bit, this does fix the problem for me as well, and judging by the comment before that line, it sounds like it was added as a workaround to a windows bug, but the workaround is apparently causing trouble of its own. Here's the bit at the end of __init__.py:
# XXX for whatever reasons, creating the first instance of a callback
# function is needed for the unittests on Win64 to succeed. This MAY
# be a compiler bug, since the problem occurs only when _ctypes is
# compiled with the MS SDK compiler. Or an uninitialized variable?
CFUNCTYPE(c_int)(lambda: None)
It looks like it's safe to remove.
FWIW, this issue showed up for me on a Centos 5.7 x64 box when using python 2.6 as installed (in parallel with python 2.4) from epel. The file was found here: /usr/lib64/python2.6/ctypes/__init__.py
Also note that the exception that shows up is a MemoryError which according to strace results from a segmentation fault immediately (though perhaps coincidentally) after a call to munmap; and it only shows up when running as FastCGI.