We Keep Coding

Python Environment Setup seems complicated and unsolvable

So, I've been a dev for about 3 years, primarily working in TypeScript and NodeJS. I'm trying to broaden my skillset by learning python (and eventually expanding my learning of Computer Vision, Machine learning, etc.) but I feel incredibly frustrated by trying to get Python to work consistently on my machine. Surely I'm doing something wrong, but I just can't really understand what it is.
I've run into these problems mostly when using ML packages (TensorFlow, Whisper, OpenCV (although I was eventually able to resolve this), so I don't know if it's related to M1 support of one of the common dependencies, etc.
My current understanding of python is that:
M1 support of python is version dependent at best.
venv is the only environment manager I should need to use
I should use pyenv to install python versions so as to not conflict with os-installed python (macos dependencies)
I'll use the latest project I'm working on as an example.
My Machine and Environment
Mac M1 Pro, MacOS Monterey 12.6
pyenv 2.3.9
Python 3.7.13
Fish Shell version 3.5.1
My general workflow to get a project started:
Create virtual environment using venv
python3 -m venv <some_environment_name>
Open created directory using VSCode, and activate the virtual environment
Here is where I encounter my first issue, which seems to be persistent.
➤ source /Users/paal/src/whisper_transcription/bin/activate.fish
functions: Function '_old_fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 18):
functions -c _old_fish_prompt fish_prompt
^
in function 'deactivate' with arguments 'nondestructive'
called on line 30 of file ~/src/whisper_transcription/bin/activate.fish
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
functions: Function 'fish_prompt' does not exist
~/src/whisper_transcription/bin/activate.fish (line 47):
functions -c fish_prompt _old_fish_prompt
^
from sourcing file ~/src/whisper_transcription/bin/activate.fish
(Type 'help functions' for related documentation)
fish: Unknown command: _old_fish_prompt
~/src/whisper_transcription/bin/activate.fish (line 71):
_old_fish_prompt
^
in function 'fish_prompt'
in command substitution
(whisper_transcription)
So, what I do to resolve this is add the following if statement to the fish.config file.
if type -q $program
_old_fish_prompt
end
Looking at GitHub issues, this seems to be a persistent issue for fish shell and this seems to at least temporarily resolve it.
Or, I just switch to ZSH.
OK, so with that resolved I move on. The environment is activated, I'm using ZSH now, and I can successfully run a python script that prints "hello world" to the console.
Then comes the nightmare of installing any packages. It seems like any project I start has some weird edge case of compatibility issues. Between M1 processors, Python Versions, builds not working correctly, etc.
for example..
import whisper
... # rest of file
This with any other code or even by itself throws the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
import whisper
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/__init__.py", line 12, in <module>
from .decoding import DecodingOptions, DecodingResult, decode, detect_language
File "/Users/paal/src/whisper_transcription/lib/python3.7/site-packages/whisper/decoding.py", line 514
if prefix := self.options.prefix:
^
SyntaxError: invalid syntax
This appears to be some problem with the Python Version. From what I understand, the := operator isn't valid syntax until Python 3.8. However, dependencies of whisper (PyTorch) only seems to be supported up to version 3.7.9
So, you can see, it seems like I just end up in these bizarre circular problems where some dependency of some package I want to use isn't supported by either the platform or the current python version, and they seem basically unsolvable (at least with my current knowledge of python)
Why is this seemingly so complicated? I'm clearly doing something wrong here, and obviously I'm out of my comfort and knowledge zone, but these issues feel very daunting and opaque, and difficult to actually troubleshoot this in any consistent or clear way.
Is there a resource that makes this stuff more clear? Is Python development on M1 chips just this broken at the moment? How can I get past these seemingly basic issues to start actually learning?
I'm not necessarily looking for a solution to this specific problem here, but if theres general advice about environment management and how to make things work somewhat reliably, I'm fine troubleshooting. I just feel like every time I start trying to learn, I end up in these rabbit holes that take hours and hours to fix and sometimes don't even really resolve things.

How to reuse modules across Python installs?

I am trying to import a python (2.7.5) module but I'm not sure if I am going at it in the right way. I usually work in Jupyter Notebook (in a seperate Conda env) to keeps things organized per project. Now I am trying to import a module called otbApplication which are Python bindings for a GIS program called Orfeo Toolbox. The thing is, Orfeo Toolbox (together with QGIS) comes with its own Python install (and subsequent paths) and even its own CMD prompt (assuming you use OSGEO4W). If I use this CMD prompt to start Python and import otbApplication, it works fine. But I want to install more packages and just work within my own 'usual' environment (Jupyter Notebook) in this case.
How should you normally reuse modules between Python installations?
I already tried placing a .pth txt file containing the path to the module in one of the sys.path locations of a different Python installation but it wouldn't even find it. I tried to force it by hosting a notebook (with the same Python install) in the same folder as the module and then importing it. I got the following error which makes me question if I am going at this the wrong way:
Python 2.7.5 |Continuum Analytics, Inc.| (default, Jul 1 2013, 12:37:52)
[MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import otbApplication
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "otbApplication.py", line 28, in <module>
_otbApplication = swig_import_helper()
File "otbApplication.py", line 24, in swig_import_helper
_mod = imp.load_module('_otbApplication', fp, pathname, description)
ImportError: DLL load failed: The specified module could not be found.
This means that forcing the same paths to my new installation is not enough, what am I missing? Apologies for the long story (or the probable butchering of some of the terms).
Kind regards,
Jasper

You've got the right idea with the .pth file, but in order to get it to work you need to check some prerequisites. Obviously, the syntax needs to be good (just give the directory locations separated by line breaks). A common issue is that folks don't put the .pth files in the correct directory (usually though not necessarily \Lib\site-packages). I bet that if you check these you'll be okay.
**Also: as noted in the comments be aware that 32-bit python isn't going to like a 64-bit DLL and verse-visa, so ensure that you're running the right version of python when you try to access those libs.

Python-magic has OSError: [WinError 193] error while running in 32-bit version of IDLE

I have been trying to install the module python-magic for a few hours, and I've been encountering some problems. I am using the 32-bit version of Python 3.5.2 with 64-bit Windows 7.
First, I used the command "pip install python-magic". I downloaded magic1.dll, regex2.dll, and zlib1.dll, and magic from the Files for Windows project, and I copied those four files to
C:\Program Files (x86)\Python35-32\Lib\site-packages\python_magic-0.4.12-py3.5.egg-info.
I added the aforementioned directory to the beginning of PATH in my Windows environment variables. Then I opened IDLE's Shell and typed "import magic" and got the response OSError: [WinError 126] The specified module could not be found.
I read that the "magic" file should have the extension .dll, so I renamed it. This led to the pop-up warning by Windows "C:\Program Files (x86)\Python35-32\Lib\site-packages\magic.dll is either not designed to run on Windows or contains an error" and by Python, "OSError: [WinError 193] %1 is not a valid Win32 application". I read that the latter error is commonly encountered while running it in a 64-bit environment, but I made sure to run it in 32-bit IDLE and only have the 32-bit version of Python installed.
Following the advice of previous StackOverflow posts, I tried copying cygmagic-1.dll, cygwin1.dll, and cygz.dll to C:\Windows\System32, to the same folder as magic.dll, and I also tried renaming cygmagic-1.dll as magic1.dll, but that didn't have any effect. I know other places say you're not supposed to mix Cygwin Python and Windows Python, but I tried it without the involvement of these files, and it didn't work then, either.
I tried renaming magic.dll to magic.exe, and that allowed "import magic" and magic.Magic(magic_file=r'C:\Program Files (x86)\Python35-32\Lib\site-packages\python_magic-0.4.12-py3.5.egg-info\magic.exe') with the response "<magic.Magic object at 0x02EA0A70>". When I tried testing with magic.from_file(r'C:\Program Files (x86)\Python35-32\Lib\site-packages\README.txt'), though, I got the error magic.MagicException: b'could not find any magic files! I figured that renaming it to magic.exe had to be wrong, but that it was worth a try.
After I gave up on python-magic, someone recommended an older project. I downloaded it and put the pymagic folder in my site-packages directory. When I tried to import pymagic.pymagic, it told me that the module StringIO doesn't exist, and the recommender told me it was because StringIO is from Python2. I changed all mentions of StringIO to io and tried the command pymagic.pymagic.identify_file(r'E:\Pictures\picture.jpg')
This generated the error TypeError: startswith first arg must be bytes or a tuple of bytes, not str. I'm not involved enough with Python's os, io, etc. modules to know how to make modifications to get this to work. Can anyone make any recommendations on how to get python-magic or pymagic working, or any other module for identifying a file based on its header? I know this question has been asked a lot, but the previous answers didn't work out for me.

Did you call the 'magic' data file magic, and leave it in the same folder as magic1.dll?
Following your instructions I was able to reproduce the same error as you. Using Sysinternals Process Monitor, I could see that the reason for your first error appeared to be that Python was trying to load the the magic data file as if it were the library.
I then renamed the magic data file to magic_data, restarted IDLE, and it worked. I could then use magic to identify a file:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import magic
>>> fn = r'C:\Python34\Lib\site-packages\python_magic-0.4.12-py3.4.egg-info\magic_data'
>>> m = magic.Magic(magic_file=fn)
>>> m.from_file(r'C:\Python34\Lib\site-packages\python_magic-0.4.12-py3.4.egg-info\zlib1.dll')
'PE32 executable for MS Windows (DLL) (console) Intel 80386 32-bit'
(I'm using a different version of Python (3.4), and a different version of Windows (10) to you, but I don't think these matter too much.)

Is it possible to make my Windows Python installation aware of extra libraries it can use when running inside a Cygwin terminal?

I would like to use the following Python package, which requires ncurses: https://pypi.python.org/pypi/blessings
I have the Windows version of Anaconda Python installed, and designated as my Python distribution.
I have also have a Cygwin installation. When I run python -i within the provided Cygwin terminal, Anaconda Python starts up -- great!
If I then try import blessings, I'll get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "A:\anaconda\lib\site-packages\blessings\__init__.py", line 5, in <module>
import curses
File "A:\anaconda\lib\curses\__init__.py", line 15, in <module>
from _curses import *
ImportError: No module named _curses
Alright, so what's happening is that Anaconda Python is understandably not picking up Cygwin's curses.
Now, does it even make sense to try and somehow make Anaconda aware of ncurses, only within the context of usage in a Cygwin terminal? I suspect it isn't, and that I am missing something conceptually?

You won't be able to use Cygwin Python's curses module with CPython unfortunately. Modules for the two Pythons are incompatible for the following reasons (and more):
Windows Python directly calls the Win32 API (via either the Visual Studio 2008, 2010 or 2015 C-Runtimes) while Cygwin Python links to the Cygwin POSIX API which is layered on top of the Visual Studio 6.0 C-Runtime. Mixing different C-Runtimes in a single process is a very bad idea: http://siomsystems.com/mixing-visual-studio-versions/, never mind further abstracted through a POSIX API.
Regardless of the C-Runtime differences, Cygwin implements the LP64 model while Windows implements the LLP64 model which would make the 64-bit size of long different between each, so some structures could be differently sized. They would likely be anyway due to #ifdefs in the code.
Is there any reason you can't use colorama?
The documentation for blessings suggests that should work.

Installing Swampy for Python 3

I am currently running Python 3.2 on my computer and need to install Swampy for a book I am reading. Reading many pages and pages on a tutorial has left me further confused. I have downloaded 'swampy1.4'. I am trying to do this by following the set of instructions on this page- http://www.instructables.com/id/How-to-install-Python-packages-on-Windows-7/.
When trying to change directories to simplejson2.6.1 (I have an updated version of this software to the page) I am getting this error- 'The system could not find the specified path.'
Could anyone tell me where I am going wrong? It would be much appreciated.

To get swampy:
Doing a pip install won't install swampy for python 3.
This method should work:
Download the source code from here. Unzip the file to the directory you want. You're going to have to remember this directory.
Next, create a swampy.txt file in the following directory:
C:/Python32/Lib/site-packages
This assumes that you installed python in C:/Python32. You should modify this depending on where you installed python.
Remember the directory in which you unzipped the source code? Type the full path to the source code folder, not the directory you unzipped it into, into swampy.txt. After this, change the extension of the text file you just created from .txt to .pth. What this does is adds the source code to the search path of python.
You should be good to go now.

Quote from Swampy Installation Instructions:
You should see something like this:
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
>>>
If so, you have Python. If the version number starts with 2, you have
Python 2, which is the version of Python used in Think Python. If the
version number starts with 3, you have Python 3. You will not be able
to use Swampy with Python 3.
I suppose you want Swampy just to learn Python. In this case I would recommend you the official Python 3 tutorial.
If you want to continue reading the book don't be afraid of installing Python 2.7.3. Multiple versions of Python can coexist even on Windows assuming that you correctly setup the PATH variable.
There is also a newer version of Swampy (2.1.1).

You can now actually use swampy from source with Python 3.2. Please see http://www.greenteapress.com/thinkpython/swampy/install.html. It clearly says:
Swampy for Python 3 is not available as a package. But the source code is available in a zip file: Swampy source for Python 3: swampy-2.1.python3.zip
I tried to use it under Windows Vista following the instructions on the web page, and at least importing TurtleWorld worked just fine.

The link is correct but the explanation is vague even for an experienced Windows developer. It assumes too much knowledge of the Python installation process IMO.
e.g. "The simplest way to use this code is to unzip it in your home directory, cd into the unzipped directory and work there."
What is meant by 'home directory'? Then there is a reference to an 'unzipped directory', which I presume means the home directory. The change of name is confusing.
Nevertheless, say one unzips to C:\Python33\lib\swampy-2.1, and works from there. Whatever this means? I can only presume it means save your code in the swampy 'home directory'. It is not best practice to save your python code in a library directory. I use \dev\python\test\ but then
Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:18:40) [MSC v.1600 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import swampy
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import swampy
ImportError: No module named 'swampy'
However, if the swampy directory is simply renamed to swampy (from swampy-2.1) then all is OK!
>>> import swampy
>>> from swampy.Gui import *
>>> g=Gui()
>>> g.title('Swampy.GUI')
>>> g.mainloop()
FYI this is my path (my dev drive is E: rather C:)
E:\Python33\Lib>path
PATH=E:\Python33\;E:\WINDOWS\system32;E:\WINDOWS;E:\WINDOWS\System32\Wbem;E:\Program Files\Microsoft SQL Server\100\Tool
s\Binn\;E:\Program Files\Microsoft SQL Server\100\DTS\Binn\;E:\WINDOWS\system32\WindowsPowerShell\v1.0;E:\Program Files\
Microsoft\Web Platform Installer\;E:\Program Files\Microchip\xc8\v1.21\bin;E:\Program Files\GtkSharp\2.12\bin
and I don't have a PYTHONPATH environment variable as suggested by other posts.