So, I've been a dev for about 3 years, primarily working in TypeScript and NodeJS. I'm trying to broaden my skillset by learning python (and eventually expanding my learning of Computer Vision, Machine learning, etc.) but I feel incredibly frustrated by trying to get Python to work consistently on my machine. Surely I'm doing something wrong, but I just can't really understand what it is.
I've run into these problems mostly when using ML packages (TensorFlow, Whisper, OpenCV (although I was eventually able to resolve this), so I don't know if it's related to M1 support of one of the common dependencies, etc.
My current understanding of python is that:
M1 support of python is version dependent at best.
venv is the only environment manager I should need to use
I should use pyenv to install python versions so as to not conflict with os-installed python (macos dependencies)
I'll use the latest project I'm working on as an example.
My Machine and Environment
Mac M1 Pro, MacOS Monterey 12.6
pyenv 2.3.9
Python 3.7.13
Fish Shell version 3.5.1
My general workflow to get a project started:
Create virtual environment using venv
python3 -m venv <some_environment_name>
Open created directory using VSCode, and activate the virtual environment
Here is where I encounter my first issue, which seems to be persistent.
➤ source /Users/paal/src/whisper_transcription/bin/activate.fish
functions: Function '_old_fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 18):
functions -c _old_fish_prompt fish_prompt
^
in function 'deactivate' with arguments 'nondestructive'
called on line 30 of file ~/src/whisper_transcription/bin/activate.fish
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
functions: Function 'fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 47):
functions -c fish_prompt _old_fish_prompt
^
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
fish: Unknown command: _old_fish_prompt
~/src/whisper_transcription/bin/activate.fish (line 71):
_old_fish_prompt
^
in function 'fish_prompt'
in command substitution
(whisper_transcription)
So, what I do to resolve this is add the following if statement to the fish.config file.
if type -q $program
_old_fish_prompt
end
Looking at GitHub issues, this seems to be a persistent issue for fish shell and this seems to at least temporarily resolve it.
Or, I just switch to ZSH.
OK, so with that resolved I move on. The environment is activated, I'm using ZSH now, and I can successfully run a python script that prints "hello world" to the console.
Then comes the nightmare of installing any packages. It seems like any project I start has some weird edge case of compatibility issues. Between M1 processors, Python Versions, builds not working correctly, etc.
for example..
import whisper
... # rest of file
This with any other code or even by itself throws the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
import whisper
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/__init__.py", line 12, in <module>
from .decoding import DecodingOptions, DecodingResult, decode, detect_language
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/decoding.py", line 514
if prefix := self.options.prefix:
^
SyntaxError: invalid syntax
This appears to be some problem with the Python Version. From what I understand, the := operator isn't valid syntax until Python 3.8. However, dependencies of whisper (PyTorch) only seems to be supported up to version 3.7.9
So, you can see, it seems like I just end up in these bizarre circular problems where some dependency of some package I want to use isn't supported by either the platform or the current python version, and they seem basically unsolvable (at least with my current knowledge of python)
Why is this seemingly so complicated? I'm clearly doing something wrong here, and obviously I'm out of my comfort and knowledge zone, but these issues feel very daunting and opaque, and difficult to actually troubleshoot this in any consistent or clear way.
Is there a resource that makes this stuff more clear? Is Python development on M1 chips just this broken at the moment? How can I get past these seemingly basic issues to start actually learning?
I'm not necessarily looking for a solution to this specific problem here, but if theres general advice about environment management and how to make things work somewhat reliably, I'm fine troubleshooting. I just feel like every time I start trying to learn, I end up in these rabbit holes that take hours and hours to fix and sometimes don't even really resolve things.
Related
Venv File I have a project that utilizes YoloV5, OpenCV, as well as many other python libraries. After finally finishing work on the project, I wanted to share it with my teammates. However, I was unable to get my code to run anywhere other than my machine. There was a Venv file that was generated automatically somewhere along the way, either by my IDE or a jupyter notebook. I tried giving out this file to others, but it is not only massive (~1GB), but it also didn't work. I will consistently get syntax errors that do not exist in my environment. For example, I will get:
"SyntaxError: Non-ASCII character '\xf0' in file detect.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
"
If I remove that line, I get:
" File "detect.py", line 120
model.warmup(imgsz=(1 if pt else bs, 3, *imgsz)) # warmup
^
SyntaxError: invalid syntax
"
Mind you, none of these errors exist in my environment.
I have tried looking up various commands to somehow link the venv file to the environment or to the python file. I thought that, perhaps, python wasn't looking for the file, so it wasn't using it. I also tried installing other packages and installing all the dependencies manually. However, none of these things worked. I am a little lost on where to go from here, I have heard of tools like Conda, which apparently might help manage things like this, but I am wondering if anyone could give advice on how to proceed, or maybe if anyone else has personally used YoloV5 and encountered this problem themselves
I am attempting to use the PyMultinest package. The full error text I am encountering is AttributeError:dlsym(RTLD_DEFAULT, run): symbol not found in the __getitem__() function in ctypes/__init__.py. I'll include more text and code below, but I am mostly trying to understand what this error is telling me - my Google Fu is apparently lacking, and the StackExchange questions I have seen relating to this error seem to be hyper-focused on solving a specific instance of this error. So - What is this error trying to tell me is wrong?
More context. I attempt to execute the PyMultinest (PMN) package as directed in the PMN documentation. PMN is, effectively, a Python wrapper for a C program. Running PMN requires a fair bit of setup code (several ancillary functions need to be defined, as well as a host of variables), which I'm not including here by default because it's ... a lot, but I can if needed. The PMN execution line I use is
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
This returns the error traceback
File "[redacted]", line 139, in <module>
pmn.run(Loglike, Prior, ndims, n_live_points=1000, n_params=n_params, outputfiles_basename='./'+ProjectName+'/temp_', resume=False, verbose=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pymultinest/run.py", line 254, in run
lib.run(*args_converted)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
func = self.__getitem__(name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 386, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, run): symbol not found
If it helps, I have determined that the name variable being passeed through self.__getitem__(), and thus into self._FuncPtr(), is run. Although, that might be obvious looking at the AttributeError message.
I am running Python 3.8 (as shown above) on a MacOS machine. Last summer, I was able to execute PMN on this machine using extremely similar code to that which I am using now. I am currently trying to optimize my code from last summer, which is why I'm surprised it isn't simply "working".
So far my attempts at fixing this have been mostly centered on reinstalling PMN. I have done a clean install (pip uninstall/pip install) of the PMN package, as well as following the PMN documentation to rebuild the C portion of the package. I have included the source directories of the C software in my Path variables - or at least, I tried to, I am assuming I was successful, but I'm not very familiar with Macs.
Ultimately, I just wish I understood what Python was telling me with this error better. It would help me direct my own attempts at solving the issue. I suspect it is saying "We don't know where to find this 'run' command you're asking for," in which case I need to figure out why my Path variable changes aren't working. Am I on the right path?
Some hints how to resolve this are given in these issues:
https://github.com/JohannesBuchner/PyMultiNest/issues/160
https://github.com/JohannesBuchner/PyMultiNest/issues/163
It is important to cleanly recompile (empty build folder) when the environment changes, so that cmake recognises changed libraries and compilers.
Check if you are running everything with python3 and not python (works as python 2.x is you have both version 2 and 3 installed)
Check is all the PyMultiNest python files are rewritten in python 3 style, e.g. $ print "something" becomes $ print("something")
My supervisor was accidentally running on python 2.7 and got the same error, so your error might as well be due to mismatch of python version usages.
Run command flutter pub cache clean and flutter clean and after that run flutter pub get now relaunch your emulator it worked for me.
Just working my way through a (very good) book call Test Driven Development using Python.
This makes use of Python3.4 by the way. By the way, I am running in a Windows 7 OS.
I've got all the stuff working using a simple text editor and running from the command line... in the course of which in particular I used "pip install" to install Django and Selenium, as per book's instructions.
This created folders "selenium" and "django" under ...\Python34\Lib\site-packages\ ... so I added these to the PythonPath for my Eclipse/PyDev project.
With the correct interpreter selected I then tried to run a file which runs fine on the command line: "> python3 functional_tests.py"... but I get
File "D:\apps\Python34\lib\site-packages\django\http\__init__.py", line 1, in <module>
from django.http.cookie import SimpleCookie, parse_cookie
File "D:\apps\Python34\lib\site-packages\django\http\cookie.py", line 5, in <module>
from django.utils.six.moves import http_cookies
ImportError: cannot import name 'http_cookies'
... to me this looks like a dependency thing... as though "pip install" handles dependency matters in a way just including a single folder doesn't.
Question boils down to this: what's the "proper" way to install a python module using PyDev?
several days later
wow... nothing? Nothing! I suppose this must mean that you either have to add dependencies manually or use something like Ant, Maven or Gradle within Eclipse itself. These latter are not my strong areas, even outside an IDE. Would still be nice to have an answer from a PyDev expert!
Well, pip install should work for PyDev (it should automatically recognize the dependency)...
I.e.: in your use case, the only folder that should be in the PYTHONPATH is D:\apps\Python34\lib\site-packages (and pip should install packages to that folder -- make sure you don't add extra folders for "D:\apps\Python34\lib\site-packages\django" nor anything else inside the site-packages to the PYTHONPATH).
If it's still not working, please check if the module django.utils.six.moves.http_cookies is indeed where you expect it to be. Also, you can print the PYTHONPATH being used in runtime with:
import sys
print('\n'.join(sorted(sys.path)))
To check if that's really what you expect.
So I downloaded source for Python3.4 and used VisualStudio 2010 to build a Python executable for my Windows7 machine. Ultimately, I want to use this to embed Python support in my application.
The first time I tried to execute my app, PyInitialise ends up aborting, and presents the error message:
"Py_Initialize: unable to load the file system codec"
So now I'm thinking, D'oh, you never installed your Python build after building it. After much googling, I find myself running msi.py. When I do, I see:
File "msi.py", line 934
raise ValueError, "Could not find "+srcdir+"/../"+pat
^
SyntaxError: invalid syntax
This looks to me like I'm trying to run pre-Python3.X code. But I'm so sure this MSI tool came with my Python3.4 source distribution. That said, I've been staring at this for so long things are starting to blend to a blur.
Any clues about where to look or what to try next?
Should I be worrying about MSI? Is there some other answer to my codec problem?
I'm getting this error every time I run any python file in Eclipse using PyDev:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/site.py", line 73, in <module>
__boot()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/site.py", line 2, in __boot
import sys, imp, os, os.path
ImportError: Bad magic number in /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.pyc
I'm using python 2.6. This problem does not occur when I run python from the terminal (2.7 or 2.6). The only substantial thing I've changed since everything last worked, is an update to OSX Lion from Snow Leopard.
Similar discussions to this seem to suggest some kind of removal of the .pyc file, because of some kind of mismatch between what was originally using the .pyc files (I'm not entirely sure what a magic number is...). But I was a bit cautious of the idea of deleting os.pyc from the Frameworks directory. When the only other file is an os.pyo file (not sure what the difference it), rather than an os.py.
I've installed all OSX Lion updates, Eclipse updates and PyDev updates.
This problem occurs even with code such as :
if __name__ == '__main__':
pass
Any help resolving this would be appreciated!
Upgrading Python to 2.7.1, running "Update Shell Profile" command file which is located in Python directory and changing the Python settings in Netbeans according to new installation worked for me.
Yeah, you'll need to regenerate all your *.pyc and *.pyo files from the *.py files.
How you do this depends on how they were generated in the first place. Some packaging of python (and it's add-ons), such as in some Linux distros, gets a little too clever for its own good and keeps the original *.py files somewhere else and have their own build system for generating and placing the *.pyc and/or *.pyo files. In a case like that, you have to use that build system to regenerate them from the original *.py files.
FYI, here are a couple links on *.pyo files. They are the optimized versions of compiled python modules.
On OS X Lion, you should have a os.py file. This is likely the root cause of your error. The os.pyc file was generated from a different version of python than you are running now. Normally, I imagine the python interpreter would just regenerate the file from os.py. But for whatever reason, your system does not have that file.
I suspect that this is a small data point in a larger set of issues and would, in general, recommend a reinstallation of your operating system.
For comparison, I'm running 10.7.1, and I have the following:
[2:23pm][wlynch#orange workout] ls /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.*
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.py
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.pyc
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/os.pyo
As an aside, the *.pyo file is an optimized version of the python bytecode.