We Keep Coding

ModuleNotFoundError: No module named 'encodings' while running python in iOS

I am trying to integrate python in iOS. I tried same thing as mentioned here - https://github.com/beeware/Python-Apple-support/tree/3.9
Here is my python script in Xcode projects
func RunPythonScript() -> PythonObject {
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
setenv("PYTHONPATH", path, 1)
setenv("PYTHONHOME", path, 1)
}
let sys = Python.import("sys")
sys.path.append("/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/PAS_10_11_v3/")
let file = Python.import("pythonscript")
let response = file.hello_world()
print(response)
return response
}
It builds successfully but when I call python program it end up saying
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.base_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.base_exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.platlibdir = 'lib'
sys.executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.path = [
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python39.zip',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python3.9',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000108fd4600 (most recent call first):
<no Python frame>

I just got this working after days of trying! Using Python 3.11.
For anyone wondering, we're using a version of Python patched for iOS (from Beeware's Python Apple Support), and PythonKit (Swift pkg) to embed Python in an iOS app using Xcode. The goal (for me at least) is to do my UI in SwiftUI, but use my own Python logic on some of the app's data.
OP, I don't know what all of your problems might be (there could be several), but I am immediately noticing at least one thing wrong. You're looking on your mac for Python when your goal is to put Python itself inside of your app. You should be looking in the app for the directory containing the Python standard library ('python-stdlib'), as well as the 'lib-dynload' subdir, provided to you by Beeware's Python Apple Support repo. You want to set PYTHONPATH and PYTHONHOME to this combo of paths.
This post is intended to supplement to "How to embed a Python interpreter in an iOS app - presented by Łukasz Langa." on YT.
In the video the OP linked to, Lukasz's 'python-stdlib' has a different name and a very different directory structure than the latest version of Py for iOS provides. What you'll need to figure out is the path to the 'python-stdlib' dir, AND [probably] its subdir, 'lib-dynload'. Counterintuitively, you do need to specify both, even though the latter is a subdir of the former.
Your line:
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
Should read something like:
if let libPath = Bundle.main.path(forResource: "python-stdlib", ofType: nil),
let libPath2 = Bundle.main.path(forResource: "python-stdlib/lib-dynload", ofType: nil) {
let mergedPaths = "\(libPath):\(libPath2)"
...where 'python-stdlib' is the PATH of the python libraries directory--not just the directory name. (If that's confusing to a newbie, this is the same: let mergedPaths = libPath + ":" + libPath2.
To get an idea of the path, go to Product > Show Build Folder in Finder, then find Products/Debug-iphonesimulator, right click .app (the greyed / X'd out icon) > Show package contents.
Assuming you have properly copied this stuff into the project, you should be able to find 'python-stdlib' inside the .app. For me, it was right in the root of the .app (since that's where I effectively put it, by adding it to the GROUP (not dir!) called 'Resources'. (Because Resources is a group, whatever is in there will be in the root level of the .app, NOT in a dir called Resources. It seems there can be groups that are backed by folders, and groups that are not. This is an important distinction.)
Now, set PYTHONPATH and PYTHONHOME to mergedPaths like you were doing:
setenv("PYTHONHOME", mergedPaths, 1)
setenv("PYTHONPATH", mergedPaths, 1)
If you don't include the path to 'lib-dynload', then this will only work in the simulator. Why would it work at all? Lovely question... 🤷🏽
IMPORTANT: The script.py file Lukasz created for his custom Python code must go into the 'python-stdlib' directory for this to work. I imagine you could stick it anywhere, as long as you append its new path to mergedPaths, as can be seen in a project generated by Beeware's Briefcase. To append that path to the merged paths, you're just concatenating strings with a ":" between each path. I have not tested this.
COPYING THE LIBRARY PROPERLY
Here's something that screwed me up for a while:
When you copy over Python.xcframework and the std-lib, make sure you're creating folders and not groups (in the prompt after you drag and drop), else you will end up with hundreds of errors due to the flattening of directories resulting from creating groups and not dirs (think of every main.py from the lib ending up in the same folder--no bueno). Make sure your prompt looks exactly like this:
screenshot of said prompt ☝🏽
SIGNING THE PYTHON LIBRARY
Follow 'The Manual Way' > Step 6 # Python Apple Support > USAGE.md.
All I had to change was this line segment:
"$CODESIGNING_FOLDER_PATH/Contents/Resources/python-stdlib/lib-dynload"
Due to my directory structure ('python-stdlib' in root of .app), I changed it to:
"$CODESIGNING_FOLDER_PATH/python-stdlib/lib-dynload"
...omitting the extra directories in the path to 'lib-dynload'.
ALMOST FORGOT--module.modulemap!
Assuming you have created 'module.modulemap', and copied it to all three locations (in the Xcode project and in both Headers dirs in the framework), you'll want it to read like this:
module Python {
umbrella header "Python.h"
export *
link "python-stdlib"
}
...where Lukasz's read link Python, ours must read link python-stdlib, as above. I wouldn't be surprised this is actually the path to 'python-stdlib'. Again, mine is right in the root of the .app, so under that assumption, the name is also the path, here. I have not tested this theory.
NOT NECESSARILY RELEVANT TO THE SOLUTION BUT MAYBE ENTERTAINING
This part is pretty amazing. I've had this working in the simulator only for the last several days. For most of that time, on the device, it would import Random, but not Math (which was being called from Random). Then I figured how to point the app to both 'python-stdlib' and 'python-stdlib/lib-dynload'. Got a new error -- code signature invalid for '.../math.cpython-311-iphoneos.so' (that Python math module!) in 'lib-dynload'! Wow! It was finally SEEING the math module.
PARTIAL CONSOLE OUTPUT:
...(code signature in <45B34416-425D-3E01-BC39-CB7A8C170A0A> '/private/var/containers/Bundle/Application/1465B572-3399-4B76-B017-4EE168637AF5/SIXTH_try.app/python-stdlib/lib-dynload/math.cpython-311-iphoneos.so' not valid for use in process: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.)
Here's the amazing part... YESTERDAY, I told Github to tell me about any changes to Beeware's Python Apple Support. Last night at 7pm, I get a notification that dude updated the USAGE.md to add instructions for code signing! Ha! What are the chances that this would happen on exactly the day that I need it to happen. Anyway, I followed those instructions, ran on my phone, and WHUHBAM! MY IPHONE IS USING THE RANDOM MODULE TO GENERATE INTS UPON A BUTTON TAP! THE WHOLE GUI WAS MADE IN SWIFTUI. HOLY MOTHER#(#)!#$ IT'S WORKING! (I realize enthusiasm and emotion are not always welcome on SO, but I'm going to celebrate, and no one will stop me!)

Is it possible to display file size in a directory served using http.server in python?

I've served a directory using
python -m http.server
It works well, but it only shows file names. Is it possible to show created/modified dates and file size, like you see in ftp servers?
I looked through the documentation for the module but couldn't find anything related to it.
Thanks!

http.server is meant for dead-simple use cases, and to serve as sample code.1 That's why the docs link right to the source.
That means that, by design, it doesn't have a lot of configuration settings; instead, you configure it by reading the source and choosing what methods you want to override, then building a subclass that does that.
In this case, what you want to override is list_directory. You can see how the base-class version works, and write your own version that does other stuff—either use scandir instead of listdir, or just call stat on each file, and then work out how you want to cram the results into the custom-built HTML.
Since there's little point in doing this except as a learning exercise, I won't give you complete code, but here's a skeleton:
class StattyServer(http.server.HTTPServer):
def list_directory(self, path):
try:
dirents = os.scandir(path)
except OSError:
# blah blah blah
# etc. up to the end of the header-creating bit
for dirent in dirents:
fullname = dirent.path
displayname = linkname = dirent.name
st = dirent.stat()
# pull stuff out of st
# build a table row to append to r
1. Although really, it's sample code for an obsolete and clunky way of building servers, so maybe that should be "to serve as sample code to understand legacy code that you probably won't ever need to look at but just in case…".

PyInstaller: Executable to access user-specified Sourcecode

I'm using pyinstaller to distribute my code as executable within my team as most of them are not coding/scripting people and do not have Python Interpreter installed.
For some advanced usage of my tool, I want to make it possible for the user to implement a small custom function to adjust functionality slightly (for the few experienced people). Hence I want to let them input a python file which defines a function with a fixed name and a string as return.
Is that possible?
I mean the py-file could be drag/dropped for example, and I'd tell them that their user-defined function needs to have a certain name, e.g. "analyze()" - is it now possible to import that from the drag/dropped pythonfile within my PyInstaller Script and use it as this?
I know, it certainly will not be safe/secure and they could do evil things, delete files and so one... But that are things which we don#t care at this point, please no discussions about it. Thanks!

To answer my own question: yes it does actually work to import a module/function from a given path/pythonfile at runtime (that I knew already) even in PyInstaller (that was new for me).
I used this for my Py2.7 program:
f = r'C:\path\to\userdefined\filewithfunction.py'
if os.path.exists(f):
import imp
userdefined = imp.load_source('', f) # Only Python 2.x, for 3.x see: https://stackoverflow.com/a/67692/701049
print userdefined # just a debugging print
userdefined.imported() # here you should use try/catch; or check whether the function with the desired name really exists in the object "userdefined". This is only a small demo as example how to import, so didnt do it here.
filewithfunction.py:
--------------------
def imported():
print 'yes it worked :-)'
As written in the comments of the example code, you'll need a slightly different approach in Python 3.x. See this link: https://stackoverflow.com/a/67692/701049

Python create empty file - Unix

I want to create empty file using Python script in Unix environment. Could see different ways mentioned of achieving the same. What are the benefits/pitfalls of one over the other.
os.system('touch abc')
open('abc','a').close()
open('abc','a')
subprocess.call(['touch','abc'])

Well, for a start, the ones that rely on touch are not portable. They won't work under standard Windows, for example, without the installation of CygWin, GNUWin32, or some other package providing a touch utility..
They also involve the creation of a separate process for doing the work, something that's totally unnecessary in this case.
Of the four, I would probably use open('abc','a').close() if the intent is to try and just create the file if it doesn't exist. In my opinion, that makes the intent clear.
But, if you're trying to create an empty file, I'd probably be using the w write mode rather than the a append mode.
In addition, you probably also want to catch the exception if, for example, you cannot actually create the file.

TLDR: use
open('abc','a').close()
(or 'w' instead of 'a' if the intent is to truncate the file if it already exists).
Invoking a separate process to do something Python can do itself is wasteful, and non-portable to platforms where the external command is not available. (Additionally, os.system uses two processes -- one more for a shell to parse the command line -- and is being deprecated in favor of subprocess.)
Not closing an open filehandle when you're done with it is bad practice, and could cause resource depletion in a larger program (you run out of filehandles if you open more and more files and never close them).

To create an empty file on Unix in Python:
import os
try:
os.close(os.open('abc', os.O_WRONLY | os.O_CREAT | os.O_EXCL |
getattr(os, "O_CLOEXEC", 0) |
os.O_NONBLOCK | os.O_NOCTTY))
except OSError:
pass # decide what to consider an error in your case and reraise
# 1. is it an error if 'abc' entry already exists?
# 2. is it an error if 'abc' is a directory or a symlink to a directory?
# 3. is it an error if 'abc' is a named pipe?
# 4. it is probably an error if the parent directory is not writable
# or the filesystem is read-only (can't create a file)
Or more portable variant:
try:
open('abc', 'ab', 0).close()
except OSError:
pass # see the comment above
Without the explicit .close() call, non-reference-counting Python implementations such as Pypy, Jython may delay closing the file until garbage collection is run (it may exhaust available file descriptors for your process).
The latter example may stuck on FIFO and follows symlinks. On my system, it is equivalent to:
from os import *
open("abc", O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, 0666)
In addition, touch command updates the access and modification times of existing files to the current time.

In more recent Python 3 variants, we have Path.touch() from pathlib. This will create an empty file if it doesn't exist, and update the mtime if it does, in the same way as your example os.system('touch abc'), but it's much more portable:
from pathlib import Path
abc = Path('abc')
abc.touch()

Use different config files - decision should be made via command-line argument

I have a script, that uses a config file called config.py. Actually this is rather a configuration module then. Anyways: the configuration-module contains a lot of parameters and dictionaries and lists of dictionaries and so on.
In the script today it is used like this
import config
def main():
myParameter = config.myParameter
Now I have another application scenario for this script that uses a related config ('config_advanced.py', but the parameters and dictionaries have different values.
My goal is now, to chose the name of the used config-modul as a passed command-line argument:
myScript.py -configuration config_advanced.py
Since the configuration-module is in the same folder than the main script, I guess I have to rename the passed configuration file to 'config.py' first. Afterwards I can perform import config. Otherwise, if I used `import config_advanced, I wouldn't be able to use a call like
config.myParameter
in the main script.
Another possibility could be, to put the configuration-modules in subfolders and keep the name config.py. The passed command-line-argument will have to contain the subfolder then.
Either way I won't be able to perform the import at the top of the main file, since I have to do the argument parsing first. This isn't a technically problem, but someone said that this it at least bad pratice.
What do you think?
What is a better way to do the trick with not much effort?
Thanks a lot
Edit:
One working solution has been
import sys fullpath = "d:\\python\\scripts\\projectA\\configurationFiles\\"
sys.path.append(fullpath)
config = __import__('config_advanced')
Without syspath it does NOT work, so those following tries won't work:
config = __import__('d:\\python\\scripts\\projectA\\configurationFiles\\config_advanced')
config = __import__('d:\\python\\scripts\\projectA\\configurationFiles\\config_advanced.py')

Another possibility that's similar to what you suggest in the question, but which doesn't need you to hide things in subfolders, is to put config_advanced.py and config_plain.py in the same folder as the main script and then dynamically make config.py a link to the actual config file you want to use.
However, martineau's suggestion is much simpler.
OTOH, georg brings up a very valid point, especially if this script isn't just for your own personal use. While using Python itself for the config data is flexible and powerful, it's perhaps a little too powerful. Config data should just be data, not live executable code. If you make a minor mistake when modifying config data you could cause havoc if it's in an executable file. And if a malicious user gets to it, there's no limit to the damage they could cause.
Bad data in a plain old data file will at worst cause a ValueError if it does something weird that your config parsing code isn't suspecting. But bad data in a live Python file could throw all sorts of nasty errors. Or even worse, it could do something evil in complete silence...
In reply to your comments, here's some code to illustrate the first point:
#! /usr/bin/env python
import os
config_file = "config.py"
def link_config(mode):
if os.path.exists(config_file):
os.remove(config_file)
config_name = "config_%s.py" % mode
os.symlink(config_name, config_file)
#.... parse command line to determine config_mode string, then do
link_config(config_mode)
#Now import the newly-linked config file
import config
If config_mode == "plain" the above code will cause config_plain.py to be imported as 'config'
and if config_mode == "advanced" it will cause config_advanced.py to be imported as 'config'
But as I said before, martineau's method is much simpler. And IIRC, os.symlink may not work on non-unix systems.
...
As for your second point, check out the docs for the json module