I have a project (https://github.com/jfkominsky/PyHab) which is essentially an extension of PsychoPy (http://psychopy.org/). I want to use Sphinx and ReadTheDocs for code documentation. I use the autoclass features from Sphinx for my documentation and have most of it in the code itself, which works great on local builds. On ReadTheDocs, this means that I needed to add a requirements file with psychopy as a requirement. This causes the build to fail in the following way:
/home/docs/checkouts/readthedocs.org/user_builds/pyhab/envs/latest/bin/pip install --exists-action=w --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/pyhab/.cache/pip -rdocs/requirements.txt
Command killed due to excessive memory consumption
I'm very amateur at this. Is there a way to use a setup.py virtual environment to get psychopy to work? Is readthedocs just being too stingy with memory? I've noticed psychopy itself has some issues with RTD builds right now, unsure if that's related...
EDIT: I should add, I was using Cpython2 as my environment in RTD. When I use CPython3 instead I get a different failure:
I tried again with Py3 and got a different error:
ERROR: failed building wxWidgets
Traceback (most recent call last):
File "build.py", line 1321, in cmd_build_wx
wxbuild.main(wxDir(), build_options)
File "/tmp/pip-build-7a1lqn7v/wxPython/buildtools/build_wxwidgets.py", line 374, in main
"Error running configure")
File "/tmp/pip-build-7a1lqn7v/wxPython/buildtools/build_wxwidgets.py", line 85, in exitIfError
raise builder.BuildError(msg)
buildtools.builder.BuildError: Error running configure
Finished command: build_wx (0m10.842s)
Finished command: build (0m10.842s)
Command '"/home/docs/checkouts/readthedocs.org/user_builds/pyhab/envs/latest/bin/python" -u build.py build' failed with exit code 1.
I have devised a solution, at least for CPython2.x. The long and short of it, which RTD needs to advertise MUCH better, is that you can slip a bunch of 'mocks' into conf.py in order to bypass modules you don't want to actually import when compiling your docs. You need mock in your requirements file in 2.7 (presumably unittest.mock in 3.x)
The idea came from here: http://blog.rtwilson.com/how-to-make-your-sphinx-documentation-compile-with-readthedocs-when-youre-using-numpy-and-scipy/
Which connects back to this in the FAQ: http://read-the-docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
I realized that you could create mock modules for literally anything, and if you use MagicMock you can add attributes to them as needed. It took me several tries to get all the right mock modules (mostly because the RTD environment is hard to emulate locally and it would only show you the first module it failed to import in the error message), but eventually, I was able to convince RTD that it had everything it needed without exceeding its memory limits.
This solution is not psychopy-specific. This will work for anything that you have trouble pip installing on RTD.
Related
So, I've been a dev for about 3 years, primarily working in TypeScript and NodeJS. I'm trying to broaden my skillset by learning python (and eventually expanding my learning of Computer Vision, Machine learning, etc.) but I feel incredibly frustrated by trying to get Python to work consistently on my machine. Surely I'm doing something wrong, but I just can't really understand what it is.
I've run into these problems mostly when using ML packages (TensorFlow, Whisper, OpenCV (although I was eventually able to resolve this), so I don't know if it's related to M1 support of one of the common dependencies, etc.
My current understanding of python is that:
M1 support of python is version dependent at best.
venv is the only environment manager I should need to use
I should use pyenv to install python versions so as to not conflict with os-installed python (macos dependencies)
I'll use the latest project I'm working on as an example.
My Machine and Environment
Mac M1 Pro, MacOS Monterey 12.6
pyenv 2.3.9
Python 3.7.13
Fish Shell version 3.5.1
My general workflow to get a project started:
Create virtual environment using venv
python3 -m venv <some_environment_name>
Open created directory using VSCode, and activate the virtual environment
Here is where I encounter my first issue, which seems to be persistent.
➤ source /Users/paal/src/whisper_transcription/bin/activate.fish
functions: Function '_old_fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 18):
functions -c _old_fish_prompt fish_prompt
^
in function 'deactivate' with arguments 'nondestructive'
called on line 30 of file ~/src/whisper_transcription/bin/activate.fish
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
functions: Function 'fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 47):
functions -c fish_prompt _old_fish_prompt
^
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
fish: Unknown command: _old_fish_prompt
~/src/whisper_transcription/bin/activate.fish (line 71):
_old_fish_prompt
^
in function 'fish_prompt'
in command substitution
(whisper_transcription)
So, what I do to resolve this is add the following if statement to the fish.config file.
if type -q $program
_old_fish_prompt
end
Looking at GitHub issues, this seems to be a persistent issue for fish shell and this seems to at least temporarily resolve it.
Or, I just switch to ZSH.
OK, so with that resolved I move on. The environment is activated, I'm using ZSH now, and I can successfully run a python script that prints "hello world" to the console.
Then comes the nightmare of installing any packages. It seems like any project I start has some weird edge case of compatibility issues. Between M1 processors, Python Versions, builds not working correctly, etc.
for example..
import whisper
... # rest of file
This with any other code or even by itself throws the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
import whisper
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/__init__.py", line 12, in <module>
from .decoding import DecodingOptions, DecodingResult, decode, detect_language
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/decoding.py", line 514
if prefix := self.options.prefix:
^
SyntaxError: invalid syntax
This appears to be some problem with the Python Version. From what I understand, the := operator isn't valid syntax until Python 3.8. However, dependencies of whisper (PyTorch) only seems to be supported up to version 3.7.9
So, you can see, it seems like I just end up in these bizarre circular problems where some dependency of some package I want to use isn't supported by either the platform or the current python version, and they seem basically unsolvable (at least with my current knowledge of python)
Why is this seemingly so complicated? I'm clearly doing something wrong here, and obviously I'm out of my comfort and knowledge zone, but these issues feel very daunting and opaque, and difficult to actually troubleshoot this in any consistent or clear way.
Is there a resource that makes this stuff more clear? Is Python development on M1 chips just this broken at the moment? How can I get past these seemingly basic issues to start actually learning?
I'm not necessarily looking for a solution to this specific problem here, but if theres general advice about environment management and how to make things work somewhat reliably, I'm fine troubleshooting. I just feel like every time I start trying to learn, I end up in these rabbit holes that take hours and hours to fix and sometimes don't even really resolve things.
I am attempting to use the PyMultinest package. The full error text I am encountering is AttributeError:dlsym(RTLD_DEFAULT, run): symbol not found in the __getitem__() function in ctypes/__init__.py. I'll include more text and code below, but I am mostly trying to understand what this error is telling me - my Google Fu is apparently lacking, and the StackExchange questions I have seen relating to this error seem to be hyper-focused on solving a specific instance of this error. So - What is this error trying to tell me is wrong?
More context. I attempt to execute the PyMultinest (PMN) package as directed in the PMN documentation. PMN is, effectively, a Python wrapper for a C program. Running PMN requires a fair bit of setup code (several ancillary functions need to be defined, as well as a host of variables), which I'm not including here by default because it's ... a lot, but I can if needed. The PMN execution line I use is
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
This returns the error traceback
File "[redacted]", line 139, in <module>
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pymultinest/run.py", line 254, in run
lib.run(*args_converted)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, run): symbol not found
If it helps, I have determined that the name variable being passeed through self.__getitem__(), and thus into self._FuncPtr(), is run. Although, that might be obvious looking at the AttributeError message.
I am running Python 3.8 (as shown above) on a MacOS machine. Last summer, I was able to execute PMN on this machine using extremely similar code to that which I am using now. I am currently trying to optimize my code from last summer, which is why I'm surprised it isn't simply "working".
So far my attempts at fixing this have been mostly centered on reinstalling PMN. I have done a clean install (pip uninstall/pip install) of the PMN package, as well as following the PMN documentation to rebuild the C portion of the package. I have included the source directories of the C software in my Path variables - or at least, I tried to, I am assuming I was successful, but I'm not very familiar with Macs.
Ultimately, I just wish I understood what Python was telling me with this error better. It would help me direct my own attempts at solving the issue. I suspect it is saying "We don't know where to find this 'run' command you're asking for," in which case I need to figure out why my Path variable changes aren't working. Am I on the right path?
Some hints how to resolve this are given in these issues:
https://github.com/JohannesBuchner/PyMultiNest/issues/160
https://github.com/JohannesBuchner/PyMultiNest/issues/163
It is important to cleanly recompile (empty build folder) when the environment changes, so that cmake recognises changed libraries and compilers.
Check if you are running everything with python3 and not python (works as python 2.x is you have both version 2 and 3 installed)
Check is all the PyMultiNest python files are rewritten in python 3 style, e.g. $ print "something" becomes $ print("something")
My supervisor was accidentally running on python 2.7 and got the same error, so your error might as well be due to mismatch of python version usages.
Run command flutter pub cache clean and flutter clean and after that run flutter pub get now relaunch your emulator it worked for me.
I'm trying to freeze a Python application and Celery, packaged up with esky for update capabilities. I need Celery to be frozen so that it has visibility into the modules from the first executable that will also be present in the library.zip file that py2exe creates.
The problem is that I cannot get Celery to freeze with esky. Here's a bare minimum, stripped down setup.py file that I'm using to try and find a solution around the problem:
from esky import bdist_esky
from distutils.core import setup
setup(name='ColdCelery',
scripts=['C:\\Python27\\Lib\\site-packages\\celery\\bin\\celery.py'],
options = {
'bdist_esky':{
'freezer_module': 'py2exe',
}
}
)
When I run the following command:
python setup.py bdist_esky
I get the following error:
running bdist_esky
running build_scripts
*** searching for required modules ***
error: c:\temp\tmpz5146o\scripts\celery.py: The process cannot access the file
because it is being used by another process
There is no running Python process on the machine that could be using Celery. I assume this is a conflict between py2exe and esky, but don't know how to overcome it.
I can freeze Celery using py2exe without referencing esky without a problem, but I will require being able to update this project in the future, so esky support is a must.
I had problems running py2exe and esky in the tutorial
I use cx freeze in my project and i haven't had any problems.
It also supports python3, linux and mac so i can recommend giving it a go.
http://cx-freeze.readthedocs.org/en/latest/
Ultimately, we did not find the reason for the failures that were occurring, but we did pick up on an important point and we changed our environments to get around the issue.
Our build machine did double duty for development. It's not the best idea of course and it was probably one of the many, many packages installed for that development that was causing trouble.
So we started with a bare bones Windows installation and installed only the minimum number of packages needed to actually perform the build. That seems to have done the trick.
I rediscovered our fix for this issue. The error message is misleading - after debugging I found that the actual error was caused by esky adding some lines to the top of celery.py during the packaging operation. These lines were going above a from __future__ import absolute_import, unicode_literals causing the following error:
SyntaxError: from __future__ imports must occur at the beginning of the file
To fix:
Edit C:\Python27\Lib\site-packages\celery\bin\celery.py and strip all comments and empty lines from the top of the file.
I have written a sudoku solver using especially, but not only, tkinter and multiprocessing. I want to make it a standalone app, which I'm using py2app for.
I've run py2applet-3.3 --make-setup sudoku_app.py (that is what it's called when installed through macports), and it succeeded at that. However when building with python3.3 (python3.3 setup.py py2app), it returns a SyntaxError with an almost endless stack trace that ends like this:
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/modulegraph/modulegraph.py", line 954, in load_module
co = compile(contents, pathname, 'exec', 0, True)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/sympy/mpmath/libmp/exec_py2.py", line 2
exec string in globals, locals
^
SyntaxError: invalid syntax
When running the same setup file with python2.7, it does succeed to build, but there's a bunch of errors when trying to run the created .app. I've written my code in python3, although trying to maintain python2 compatibility through various try ... except - blocks and from future imports. At some point though, it got too complicated and I decided to just go with python3.
Why does py2app not work with python3 there?
Your application doesn’t work with py2app (yet) because I didn’t consider the way you do py3k support. In particular, modulegraph assumes that all modules in the dependency graph contain valid python code for the current python version.
Could you file a bug at my bitbucket page (https://bitbucket.org/ronaldoussoren/py2app) to ensure that I don’t forget to fix this?
I'm trying to run Django on shared hosting (Bluehost). I'm using functionality that requires PIL. PIL imports and works from an interactive shell, but in my fcgi process it crashes with a MemoryError at from PIL import image. Any help on why it might be failing inside fcgi would be much appreciated.
__Environment Info__:
Python2.7
Local installs of libjpg, zlib, freetype, and lcms
Virtualenv:
Django 1.3, PIL, flup, etc.
__Stack Trace__:
File ".../feincms_thumbnail.py", line 3, in <module>
from PIL import Image
File ".../PIL/Image.py", line 45, in <module>
\__import__("FixTk")
File ".../python2.7/lib-tk/FixTk.py", line 15, in <module>
import ctypes
File ".../python2.7/ctypes/__init__.py", line 549, in <module>
CFUNCTYPE(c_int)(lambda: None)
__.fcgi__:
<!-- language: python -->
# setup paths
# set DJANGO_SETTINGS_MODULE in os.environ
from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")
I have temporarily fixed that error commenting the last line in this file $HOME/lib/python2.7/ctypes/__init__.py that is something like #CFUNCTYPE(c_int)(lambda: None).
That worked for me, but I don't know exactly what is the problem.
UPDATE
In python 2.7.3 the line number is :279 not the last as I said above.
UPDATE 2
Since the line may vary between minor versions, you should look for a chunk of code that looks something like this:
# XXX for whatever reasons, creating the first instance of a callback
# function is needed for the unittests on Win64 to succeed. This MAY
# be a compiler bug, since the problem occurs only when _ctypes is
# compiled with the MS SDK compiler. Or an uninitialized variable?
CFUNCTYPE(c_int)(lambda: None)
try running this command:
setsebool -P httpd_tmp_exec on
fixes things for me on CentOS. Taken from this post:
https://bugzilla.redhat.com/show_bug.cgi?id=645193
Just to expand on eos87's answer a bit, this does fix the problem for me as well, and judging by the comment before that line, it sounds like it was added as a workaround to a windows bug, but the workaround is apparently causing trouble of its own. Here's the bit at the end of __init__.py:
# XXX for whatever reasons, creating the first instance of a callback
# function is needed for the unittests on Win64 to succeed. This MAY
# be a compiler bug, since the problem occurs only when _ctypes is
# compiled with the MS SDK compiler. Or an uninitialized variable?
CFUNCTYPE(c_int)(lambda: None)
It looks like it's safe to remove.
FWIW, this issue showed up for me on a Centos 5.7 x64 box when using python 2.6 as installed (in parallel with python 2.4) from epel. The file was found here: /usr/lib64/python2.6/ctypes/__init__.py
Also note that the exception that shows up is a MemoryError which according to strace results from a segmentation fault immediately (though perhaps coincidentally) after a call to munmap; and it only shows up when running as FastCGI.