We Keep Coding

ModuleNotFoundError: No module named 'encodings' while running python in iOS

I am trying to integrate python in iOS. I tried same thing as mentioned here - https://github.com/beeware/Python-Apple-support/tree/3.9
Here is my python script in Xcode projects
func RunPythonScript() -> PythonObject {
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
setenv("PYTHONPATH", path, 1)
setenv("PYTHONHOME", path, 1)
}
let sys = Python.import("sys")
sys.path.append("/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/PAS_10_11_v3/")
let file = Python.import("pythonscript")
let response = file.hello_world()
print(response)
return response
}
It builds successfully but when I call python program it end up saying
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.base_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.base_exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.platlibdir = 'lib'
sys.executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.path = [
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python39.zip',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python3.9',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000108fd4600 (most recent call first):
<no Python frame>

I just got this working after days of trying! Using Python 3.11.
For anyone wondering, we're using a version of Python patched for iOS (from Beeware's Python Apple Support), and PythonKit (Swift pkg) to embed Python in an iOS app using Xcode. The goal (for me at least) is to do my UI in SwiftUI, but use my own Python logic on some of the app's data.
OP, I don't know what all of your problems might be (there could be several), but I am immediately noticing at least one thing wrong. You're looking on your mac for Python when your goal is to put Python itself inside of your app. You should be looking in the app for the directory containing the Python standard library ('python-stdlib'), as well as the 'lib-dynload' subdir, provided to you by Beeware's Python Apple Support repo. You want to set PYTHONPATH and PYTHONHOME to this combo of paths.
This post is intended to supplement to "How to embed a Python interpreter in an iOS app - presented by Łukasz Langa." on YT.
In the video the OP linked to, Lukasz's 'python-stdlib' has a different name and a very different directory structure than the latest version of Py for iOS provides. What you'll need to figure out is the path to the 'python-stdlib' dir, AND [probably] its subdir, 'lib-dynload'. Counterintuitively, you do need to specify both, even though the latter is a subdir of the former.
Your line:
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
Should read something like:
if let libPath = Bundle.main.path(forResource: "python-stdlib", ofType: nil),
let libPath2 = Bundle.main.path(forResource: "python-stdlib/lib-dynload", ofType: nil) {
let mergedPaths = "\(libPath):\(libPath2)"
...where 'python-stdlib' is the PATH of the python libraries directory--not just the directory name. (If that's confusing to a newbie, this is the same: let mergedPaths = libPath + ":" + libPath2.
To get an idea of the path, go to Product > Show Build Folder in Finder, then find Products/Debug-iphonesimulator, right click .app (the greyed / X'd out icon) > Show package contents.
Assuming you have properly copied this stuff into the project, you should be able to find 'python-stdlib' inside the .app. For me, it was right in the root of the .app (since that's where I effectively put it, by adding it to the GROUP (not dir!) called 'Resources'. (Because Resources is a group, whatever is in there will be in the root level of the .app, NOT in a dir called Resources. It seems there can be groups that are backed by folders, and groups that are not. This is an important distinction.)
Now, set PYTHONPATH and PYTHONHOME to mergedPaths like you were doing:
setenv("PYTHONHOME", mergedPaths, 1)
setenv("PYTHONPATH", mergedPaths, 1)
If you don't include the path to 'lib-dynload', then this will only work in the simulator. Why would it work at all? Lovely question... 🤷🏽
IMPORTANT: The script.py file Lukasz created for his custom Python code must go into the 'python-stdlib' directory for this to work. I imagine you could stick it anywhere, as long as you append its new path to mergedPaths, as can be seen in a project generated by Beeware's Briefcase. To append that path to the merged paths, you're just concatenating strings with a ":" between each path. I have not tested this.
COPYING THE LIBRARY PROPERLY
Here's something that screwed me up for a while:
When you copy over Python.xcframework and the std-lib, make sure you're creating folders and not groups (in the prompt after you drag and drop), else you will end up with hundreds of errors due to the flattening of directories resulting from creating groups and not dirs (think of every main.py from the lib ending up in the same folder--no bueno). Make sure your prompt looks exactly like this:
screenshot of said prompt ☝🏽
SIGNING THE PYTHON LIBRARY
Follow 'The Manual Way' > Step 6 # Python Apple Support > USAGE.md.
All I had to change was this line segment:
"$CODESIGNING_FOLDER_PATH/Contents/Resources/python-stdlib/lib-dynload"
Due to my directory structure ('python-stdlib' in root of .app), I changed it to:
"$CODESIGNING_FOLDER_PATH/python-stdlib/lib-dynload"
...omitting the extra directories in the path to 'lib-dynload'.
ALMOST FORGOT--module.modulemap!
Assuming you have created 'module.modulemap', and copied it to all three locations (in the Xcode project and in both Headers dirs in the framework), you'll want it to read like this:
module Python {
umbrella header "Python.h"
export *
link "python-stdlib"
}
...where Lukasz's read link Python, ours must read link python-stdlib, as above. I wouldn't be surprised this is actually the path to 'python-stdlib'. Again, mine is right in the root of the .app, so under that assumption, the name is also the path, here. I have not tested this theory.
NOT NECESSARILY RELEVANT TO THE SOLUTION BUT MAYBE ENTERTAINING
This part is pretty amazing. I've had this working in the simulator only for the last several days. For most of that time, on the device, it would import Random, but not Math (which was being called from Random). Then I figured how to point the app to both 'python-stdlib' and 'python-stdlib/lib-dynload'. Got a new error -- code signature invalid for '.../math.cpython-311-iphoneos.so' (that Python math module!) in 'lib-dynload'! Wow! It was finally SEEING the math module.
PARTIAL CONSOLE OUTPUT:
...(code signature in <45B34416-425D-3E01-BC39-CB7A8C170A0A> '/private/var/containers/Bundle/Application/1465B572-3399-4B76-B017-4EE168637AF5/SIXTH_try.app/python-stdlib/lib-dynload/math.cpython-311-iphoneos.so' not valid for use in process: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.)
Here's the amazing part... YESTERDAY, I told Github to tell me about any changes to Beeware's Python Apple Support. Last night at 7pm, I get a notification that dude updated the USAGE.md to add instructions for code signing! Ha! What are the chances that this would happen on exactly the day that I need it to happen. Anyway, I followed those instructions, ran on my phone, and WHUHBAM! MY IPHONE IS USING THE RANDOM MODULE TO GENERATE INTS UPON A BUTTON TAP! THE WHOLE GUI WAS MADE IN SWIFTUI. HOLY MOTHER#(#)!#$ IT'S WORKING! (I realize enthusiasm and emotion are not always welcome on SO, but I'm going to celebrate, and no one will stop me!)

How does the Python subprocess module find executables to run? Can I change the location?

On Linux systems, subprocess.run(["some_command"]) appears to scrape /usr/bin for some_command by default and thus, subprocess accepts every command that also has a respective binary in this directory.
I have a use case now, where I have binaries in multiple directories and I want to execute some of them with the subprocess module. Obviously, I can do something like:
subprocess.run(["/absolute/path/to/first_binary"])
subprocess.run(["/another/absolute/path/to/another/second_binary"])
But I was wondering if there exists anything like:
subprocess.set_bin_dirs(["/usr/bin/", "/absolute/path/to/", "/another/absolute/path/to/another/"])
subprocess.run(["first_binary"])
subprocess.run(["second_binary"])

This has nothing to do with Python and the subprocess module. This is a core idea about how Unix-like(and other) operating systems work, when you do some_command, that command is looked up on your PATH. So you must add that directories of these binaries to the PATH, otherwise, you need to use the full path when you try to execute a command, e.g. /usr/bin/some_command
To do this in Python with the subprocess module, you could do something like:
import os
env_copy = os.environ.copy()
current_path = current_env['PATH']
env_copy["PATH"] = os.pathsep.join([
"/usr/bin/",
"/absolute/path/to/",
"/another/absolute/path/to/another/",
current_path
])
subprocess.run(["some_command"], env=env_copy)
Note, paths are looked up in order, so the order in which you append it matters if there are possible name-collisions. Note, you don't actually have to add the current PATH, but you might. All of this is up to you.

Setting CWDir as a variable for MS-Dos environment

Can someone help me with this please?
I am trying to compile a program in this case programmed in python that I can run in win9Xdos, that I can call/start from a 9xDos batchfile, that will find the Current working Dir & by that I mean identify the cwd (current working directory) from where the python program and batchfile are executed. The python program will be called getcwd.py which I am hoping someone will outline what I need to do to convert to EXE/COM file. there is a program called Py2EXE but not sure if this will compile for Ms-dos file. Anyways heres my simple code thus far. Can someone tell me if I am on the right track please? Oh by the way what I am trying to do is find the CWD & inject the resultant path into a variable that can be read from 9Xdos. The current Var would be %cwd%
# importing os module
import os
# some websites say use: del cwd (to remove variable if it exists)
cwd = none
cwd = os.getcwd()
print(cwd)
The print line may need interchanging with code below, not sure help needed:
print(type(path))
# <class 'str'>
would the above code work, say in the root e.g. C:\ with & work in obtaining the CWD variable & if correct how would I proceed to compile it to an exe/com file? do I need to take into account LFN's & Spaces between possible paths e.g C:\Program Files & possible backslashes etc?

Your code isn't perfect but it is on the right track. All you need is this:
import os
if __name__ == '__main__':
print(os.getcwd())
There is no need for an auxiliary variable, and I don't know what websites are recommending that you delete the variable before creating it. Trying to delete a nonexistent Python variable is a runtime error. So I would stay away from those websites.
But your question is about setting an environment variable. Calling print() won't do that. All it will do is echo the current working directory to the console. There is no way to change the environment of a running process that will affect the parent process. This is not a Python restriction nor a Windows restriction: it is quite general. The OS sets up the environment of the process when it creates the process. You can make changes to the environment (using os.environ[env-var]) but those changes will only be visible inside that Python process and will not be visible to the environment of the batch file that runs the Python program. To do that, you need to pass the value back to the calling process.
One way to do that is like this:
In Python:
import os
if __name__ == '__main__':
print(f"set CWDIR={os.getcwd()}", file=open("mycd.bat","w"))
I haven't had a Python 1.5.2 environment for 15 years, so I can't test this, but I think the equivalent would have been
if __name__ == '__main__':
print >> open("mycd.bat","w"), "set CWDIR=%s" % (os.getcwd(),)
In a cmd.exe console:
call mycd.bat
Though if your Win9XDos doesn't provide %cd% (which, as far as I recall, was available in MS-DOS 5, or maybe further back still) there is no way of telling if it supports call either. (Are you maybe running command.com instead of cmd.exe? That would explain why things that should be there are missing).
I used pyinstaller to create a 64-bit .exe and that resulted in a file of about 6MB. Now, 32-bit executables are smaller, but it might be that the resulting executable is still too big to load.
So I think the Python route may turn out to be more trouble than it is worth.

Python Scripts + Use of folders and paths

This time I need help with python and paths manipulations. In first place i will will show you the structure im using on this set of apps:
MainFolder:
Folder1.
Subfolder1.
Subfolder2.
Folder2.
The folder I have the scripts in, is Folder 2. But i need the path to Main Folder(1 back from Folder2). But the method i'm using, isn't quite reliable.
Currently i'm using os.getcwd(), but if I lunch a shell trought an Excel Macros, the path breaks. From time to time the code brakes(often in loops that use paths).
I need to lunch said scripts trought Excel or at least CMD. Because this will be for People who knows just enough about computers to make it every day. And it need to operate on everyone machines.
PS: The scripts works just fine, but they do need to be in a separate folder from the files they are working on.
UPDATE 1 AS REQUESTED:
I've made a class and this is the class
class mypath:
def Revpath(self):
CWD = os.getcwd()
Revpaths = CWD[:-14]
return Revpaths
def Saldos(self):
CWD = os.getcwd()
Revpaths = CWD[:-14]
Cuadraje = Revpaths+"Stock\\Saldos"
return Cuadraje
def Banco(self,IDkind):
CWD = os.getcwd()
Revpaths = CWD[:-14]
Stocks = Revpaths+"Stock\\kind\\"+IDkind
return Stocks
mp = mypath()
Then I just assign the returned value to a Var. One is used on a CSV writter(Wich happens to be the most common miss path). And the other to set a download folder, for a firefox profile. Each script uses the same Class, and the logic is 100% the same on every script. But i only can show you this much, because the code on itself is messy and have sensitive data in it.
UPDATE 2: SOLVED
Solved this by replacing os.getcwd() for os.path.realpath(__file__)
Because of language (First Lanaguage is Spanish), I've assumed that the Current Working Directory was the one with the script, instead it returned the PyCharm Settings folders(I'm still clueless about this, cus' i'm launching the scripts trough a Shell from a excel macross button).
Also i've updated the code on the class i've presented above, for stability in my applications.

py2exe and the file system

I have a Python app. It loads config files (and various other files) by
doing stuff such as:
_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CONFIG_DIR = os.path.join(_path, 'conf')
This works fine. However, when I package the app with py2exe, bad things happen:
File "proj\config.pyc", line 8, in <module>
WindowsError: [Error 3] The system cannot find the path specified: 'C:\\proj\
\dist\\library.zip\\conf'
Obviously that's an invalid path... What's a more robust way of doing this? I don't
want to specify absolute paths in the program because it could be placed in different
folders. Should I just say "if it says the folder name is 'library.zip', then go
one more level down to the 'dist' folder"?
Note that I have pretty nested directory hierarchies... for example, I have
a module gui.utils.images, stored in "gui/utils/images.py", and it uses its path
to access "gui/images/ok.png", for example. Right now the py2exe version
would try to access "proj/dist/library.zip/gui/images/ok.png", or something,
which just won't work.

The usual approach to doing this sort of thing (if I understand properly) is this:
check sys.frozen, which py2exe contrives to set, using something like getattr(sys, 'frozen', '')
if that's not set, use the usual method since you're running from source
if it's set, check sys.executable since that will point you to where the .exe is (instead of to where python.exe is, which is what it normally points to). Use something like os.path.dirname(sys.executable) and then join that with your relative paths to find subfolders etc
Example:
frozen = getattr(sys, 'frozen', '')
if not frozen:
# not frozen: in regular python interpreter
approot = os.path.dirname(__file__)
elif frozen in ('dll', 'console_exe', 'windows_exe'):
# py2exe:
approot = os.path.dirname(sys.executable)
elif frozen in ('macosx_app',):
# py2app:
# Notes on how to find stuff on MAC, by an expert (Bob Ippolito):
# http://mail.python.org/pipermail/pythonmac-sig/2004-November/012121.html
approot = os.environ['RESOURCEPATH']
Or some variant thereof... the latter one handles the use of py2app. You could extend this for other builders, if needed.

What do you think about using relative paths for all of the included files? I guess it should be possible to use sys.path.append(".." + os.path.sep + "images") for your example about ok.png, then you could just open("ok.png", "rb"). Using relative paths should fix the issues with the library.zip file that's generated by py2exe, at least that's what it does for me.

Use os.path.dirname(os.path.abspath(sys.argv[0])) from a py2exe app, it'll work the same way from the python script (so you can test without creating the exe every time) and from the exe.
This can be much better than using relative paths because you don't have to worry about where your app (.py or .exe) is running from.

Here's my solution (tested on Windows, Linux and Mac). It also works if the application or Python script is started via a symbolic link.
# Get if frozen
is_frozen = bool( getattr(sys, 'frozen', None) )
# Get path variable
path = sys.path if is_frozen else sys.argv
# Get nonempty first element or raise error
if path and path[0]:
apppath = path[0]
elif is_frozen():
raise RuntimeError('Cannot determine app path because sys.path[0] is empty.')
else:
raise RuntimeError('Cannot determine app path in interpreter mode.')
# Make absolute (eliminate symbolic links)
apppath = os.path.abspath( os.path.realpath(apppath) )
# Split and return
appdir, appname = os.path.split(apppath)

I use the following trick in windows.
When the program is frozen in a py2exe 'executable' (i.e my_application.exe), dirname(__file__) returns the folder where your main python script is running.
Oddly enough, that folder is not the folder containing my_application.exe but my_application.exe itself.
Note that my_application.exe is not a binary file but a compressed folder (you can unzip it) that contains python libs and your compiled scripts.
That's why the os.path.dirname of your __file__ is in fact my_application.exe.
Thus, this code gives you the root directory for your configuration files or images (in case of a frozen application, the folder containing the py2exe executable, otherwise the folder from where your python script is running):
dirname = os.path.dirname(__file__)
if dirname.endswith('.exe'):
dirname = os.path.split(dirname)[0]
Obviously you can use more general, portable methods to detect if the file is frozen as other answers show instead of the endswith method.