We Keep Coding

ModuleNotFoundError: No module named 'encodings' while running python in iOS

I am trying to integrate python in iOS. I tried same thing as mentioned here - https://github.com/beeware/Python-Apple-support/tree/3.9
Here is my python script in Xcode projects
func RunPythonScript() -> PythonObject {
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
setenv("PYTHONPATH", path, 1)
setenv("PYTHONHOME", path, 1)
}
let sys = Python.import("sys")
sys.path.append("/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/PAS_10_11_v3/")
let file = Python.import("pythonscript")
let response = file.hello_world()
print(response)
return response
}
It builds successfully but when I call python program it end up saying
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.base_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.base_exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.platlibdir = 'lib'
sys.executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.path = [
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python39.zip',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python3.9',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000108fd4600 (most recent call first):
<no Python frame>

I just got this working after days of trying! Using Python 3.11.
For anyone wondering, we're using a version of Python patched for iOS (from Beeware's Python Apple Support), and PythonKit (Swift pkg) to embed Python in an iOS app using Xcode. The goal (for me at least) is to do my UI in SwiftUI, but use my own Python logic on some of the app's data.
OP, I don't know what all of your problems might be (there could be several), but I am immediately noticing at least one thing wrong. You're looking on your mac for Python when your goal is to put Python itself inside of your app. You should be looking in the app for the directory containing the Python standard library ('python-stdlib'), as well as the 'lib-dynload' subdir, provided to you by Beeware's Python Apple Support repo. You want to set PYTHONPATH and PYTHONHOME to this combo of paths.
This post is intended to supplement to "How to embed a Python interpreter in an iOS app - presented by Łukasz Langa." on YT.
In the video the OP linked to, Lukasz's 'python-stdlib' has a different name and a very different directory structure than the latest version of Py for iOS provides. What you'll need to figure out is the path to the 'python-stdlib' dir, AND [probably] its subdir, 'lib-dynload'. Counterintuitively, you do need to specify both, even though the latter is a subdir of the former.
Your line:
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
Should read something like:
if let libPath = Bundle.main.path(forResource: "python-stdlib", ofType: nil),
let libPath2 = Bundle.main.path(forResource: "python-stdlib/lib-dynload", ofType: nil) {
let mergedPaths = "\(libPath):\(libPath2)"
...where 'python-stdlib' is the PATH of the python libraries directory--not just the directory name. (If that's confusing to a newbie, this is the same: let mergedPaths = libPath + ":" + libPath2.
To get an idea of the path, go to Product > Show Build Folder in Finder, then find Products/Debug-iphonesimulator, right click .app (the greyed / X'd out icon) > Show package contents.
Assuming you have properly copied this stuff into the project, you should be able to find 'python-stdlib' inside the .app. For me, it was right in the root of the .app (since that's where I effectively put it, by adding it to the GROUP (not dir!) called 'Resources'. (Because Resources is a group, whatever is in there will be in the root level of the .app, NOT in a dir called Resources. It seems there can be groups that are backed by folders, and groups that are not. This is an important distinction.)
Now, set PYTHONPATH and PYTHONHOME to mergedPaths like you were doing:
setenv("PYTHONHOME", mergedPaths, 1)
setenv("PYTHONPATH", mergedPaths, 1)
If you don't include the path to 'lib-dynload', then this will only work in the simulator. Why would it work at all? Lovely question... 🤷🏽
IMPORTANT: The script.py file Lukasz created for his custom Python code must go into the 'python-stdlib' directory for this to work. I imagine you could stick it anywhere, as long as you append its new path to mergedPaths, as can be seen in a project generated by Beeware's Briefcase. To append that path to the merged paths, you're just concatenating strings with a ":" between each path. I have not tested this.
COPYING THE LIBRARY PROPERLY
Here's something that screwed me up for a while:
When you copy over Python.xcframework and the std-lib, make sure you're creating folders and not groups (in the prompt after you drag and drop), else you will end up with hundreds of errors due to the flattening of directories resulting from creating groups and not dirs (think of every main.py from the lib ending up in the same folder--no bueno). Make sure your prompt looks exactly like this:
screenshot of said prompt ☝🏽
SIGNING THE PYTHON LIBRARY
Follow 'The Manual Way' > Step 6 # Python Apple Support > USAGE.md.
All I had to change was this line segment:
"$CODESIGNING_FOLDER_PATH/Contents/Resources/python-stdlib/lib-dynload"
Due to my directory structure ('python-stdlib' in root of .app), I changed it to:
"$CODESIGNING_FOLDER_PATH/python-stdlib/lib-dynload"
...omitting the extra directories in the path to 'lib-dynload'.
ALMOST FORGOT--module.modulemap!
Assuming you have created 'module.modulemap', and copied it to all three locations (in the Xcode project and in both Headers dirs in the framework), you'll want it to read like this:
module Python {
umbrella header "Python.h"
export *
link "python-stdlib"
}
...where Lukasz's read link Python, ours must read link python-stdlib, as above. I wouldn't be surprised this is actually the path to 'python-stdlib'. Again, mine is right in the root of the .app, so under that assumption, the name is also the path, here. I have not tested this theory.
NOT NECESSARILY RELEVANT TO THE SOLUTION BUT MAYBE ENTERTAINING
This part is pretty amazing. I've had this working in the simulator only for the last several days. For most of that time, on the device, it would import Random, but not Math (which was being called from Random). Then I figured how to point the app to both 'python-stdlib' and 'python-stdlib/lib-dynload'. Got a new error -- code signature invalid for '.../math.cpython-311-iphoneos.so' (that Python math module!) in 'lib-dynload'! Wow! It was finally SEEING the math module.
PARTIAL CONSOLE OUTPUT:
...(code signature in <45B34416-425D-3E01-BC39-CB7A8C170A0A> '/private/var/containers/Bundle/Application/1465B572-3399-4B76-B017-4EE168637AF5/SIXTH_try.app/python-stdlib/lib-dynload/math.cpython-311-iphoneos.so' not valid for use in process: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.)
Here's the amazing part... YESTERDAY, I told Github to tell me about any changes to Beeware's Python Apple Support. Last night at 7pm, I get a notification that dude updated the USAGE.md to add instructions for code signing! Ha! What are the chances that this would happen on exactly the day that I need it to happen. Anyway, I followed those instructions, ran on my phone, and WHUHBAM! MY IPHONE IS USING THE RANDOM MODULE TO GENERATE INTS UPON A BUTTON TAP! THE WHOLE GUI WAS MADE IN SWIFTUI. HOLY MOTHER#(#)!#$ IT'S WORKING! (I realize enthusiasm and emotion are not always welcome on SO, but I'm going to celebrate, and no one will stop me!)

Executing Python Script from Windows Forms .NET

I am fairly new to Python and .NET in general, but decided to ask more competent people, since I have been struggling with the issue of executing python script from Windows Forms.
The basic idea of my project is a desktop applicaton and the overall logic would be to read from a couple of selected check boxes, pass the values of those selections to my python script, from there I generate an excell table based on those results, and display this table back into the Windows Forms application.
Creating the table and managing to display it in the Desktop App is already done, but I am having serious issues with the communication between the two platforms, when it came to executing the script itself.
I have tried using IronPython and it worked perfectly, untill the fact that I found that Iron Python does not support CPython packages, like Pandas, which is build on numpy, and numpy apparantly is one of those packages. I looked over a lot of articles about this issue and the answers did not seem promising and most of the suggestions were to use pythonnet.
I tried to implement pythonnet, following numerous articles and all I managed to do, besides creating a bigger mess, is nothing as a result.
Finally, I decided to use C# Process class, but did not succeed also.
Would appreciate if there are any comments and suggestions on how to remedy this issue.
Python version: 3.7
Windows 10 (64 bit)
.NET Framework 4.7.2
Here is some of my code attempts in Windows Forms:
Implementation with the usage of the Process Class
Issue here is that I am not able to run this script due to the error messages that it cannot find the packages for the python script
var processStartInfo = new ProcessStartInfo
{
Arguments = "C:\\Users\\Dobromir\\PycharmProjects\\pythonProject\\main.py",
FileName = "C:\\Python27\\python.exe",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
};
Process.Start(processStartInfo);
Implementation using IronPython (which was working before the usage of pandas package)
Issue here is the CPython packages limitation and errors.
For IronPython I had to downgrade to Python 2.7 in order to work with it. For the rest of the examples I am using Python 3.7
ScriptEngine pythonEngine = Python.CreateEngine();
var searchPaths = pythonEngine.GetSearchPaths();
searchPaths.Add(#"C:\Python27\Lib");
searchPaths.Add(#"C:\Users\Dobromir\PycharmProjects\pythonProject\venv\Lib\site-packages");
pythonEngine.SetSearchPaths(searchPaths);
List<String> argv = new List<String>();
argv.Add("Some Value1");
argv.Add("Some Value2");
ScriptSource pythonScript = pythonEngine.CreateScriptSourceFromFile("C:\\Users\\Dobromir\\PycharmProjects\\pythonProject\\main.py");
pythonEngine.GetSysModule().SetVariable("argv", argv);
pythonEngine.SetSearchPaths(searchPaths);
ScriptScope scope = pythonEngine.CreateScope();
pythonScript.Execute(scope);
Implementation of pythonnet
The issue that I got here is on the line using Py.GIL(). I believe it is having trouble finding the python files, and also tried giving the python37.dll in the variable pathToPython.
I received the error that Python.Runtime, Version=2.5.2.0, Culture=neutral....missmatch"
string pathToPython = #"C:\Users\Dobromir\AppData\Local\Programs\Python\Python37";
string path = pathToPython + "; " + Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);
Console.WriteLine(path);
var lib = new[]
{
#"C:\\Users\\Dobromir\\PycharmProjects\\App37\\main.py",
Path.Combine(pathToPython, "Lib"),
Path.Combine(pathToPython, "DLLs")
};
string paths = string.Join("; ", lib);
Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);
using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
{
try
{
Console.WriteLine("I am working");
}
catch (PythonException error)
{
Console.WriteLine("Error occured: ", error.Message);
}
}
I Also tried creating a bash script to execute the python script and got the no module found error as well
I know that these are not the best implementations out there, but do the job.
My question is if someone has any idea on how to make this simple operation work I would be very grateful, thank you for your time and understanding
P.S - Apologies for the long post, wanted to write what I have tried before asking for help, but if someone is more interested I will provide additional information.

I did a project like this recently; a couple of things I would suggest to make it easy:
Confirm that the instance of python set in your env variables (WIN+R, sysdm.cpl, Advanced, env variables) is that of the instance of python you wish to use (do this for your python search path too!)
Remove any lines attempting to set these in code; and instead handle errors if they are not found
Then, when you call you script from within your program; it only needs to look like this:
var processStartInfo = new ProcessStartInfo
{
Arguments = "main.py",
FileName = "Python",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
};
Process.Start(processStartInfo);

After some struggle, I found a solution to fit my needs.
Firstly, I completely removed python 2.7 and installed back 3.10.
I tried running the script file inside the shell command line and got the same error that the modules could not be found. What I did is try to import these modules and it gave an error, specifically for bs4 that I am using packages for python 2.x instead of 3.x packages.
After futher investigation I discovered that the packages that I have for my script are treated as "local" packages, meaning I installed them from the IDE (PyCharm) and they work for that project only I guess.
I also found that to "globally" access these packages I had to install them through the command line using the pip3 install <package_name>. After doing this the problem was gone and was left with running the script from the Windows Forms.
NOTE: I did not manage to start the script using python.exe, so I used bash script for the job.
Here is my code and I hope it helps someone down the line...
Code in C#
string myApp = string.Format("{0} {1}", #"C:\testing1.sh", "Hello");
var processStartInfo = new ProcessStartInfo
{
Arguments = myApp,
FileName = "C:\\Program Files\\Git\\git-bash.exe",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = false
};
Process.Start(processStartInfo)
Code in Bash Script File
#!/bin/bash
arg1="$1"
python C:/Users/Dobromir/PycharmProjects/testing/main.py "$arg1"
Inside the Python file I am using sys.argv[] and accessing the arguments.
NOTE: Passing arguments from the bash script to the python script, in this case, you will receive 2 arguments - first one is the path to the python file and the second is the variable arg1.
Another important thing to mention is you need to have comas around the $1 - this is the property that is being send from the C# file, else it will show as empty.
Articles that were useful:
Installed BeautifulSoup but still get no module named bs4
Passing arguments to Python from Shell Script
https://unix.stackexchange.com/questions/31414/how-can-i-pass-a-command-line-argument-into-a-shell-script
https://gist.github.com/creativcoder/6df4d349447ff1416e68
Thank you to everyone who contributed and tried to help, I managed to learned new things with your suggestions!

simplest possible python cffi example

I am trying to call a c function or c program from python using cffi, but I find most examples too complex for me to easily learn from. One of the best examples I have found is this example and I have copied that and made a slightly simpler version that I want to post here.
I would greatly appreciate any feedback on this example or I am hoping someone could make an even shorter and simpler example of using cffi in python for API out of line and inline and maybe even a super simple ABI example. I am impressed with how easy this is to call a simple adding function, a lot easier than calling c from Lua.
So it seems to work, but I am still not really sure why and not sure how well my more complex real example will work because I don't really know what I am doing. Incidentally this was the very first python program I have written or run. My 'hello world'. My purpose is to call a 1000 line c program I wrote from a python script to be run with the openlightspeed web server.

I am using Ubuntu 19.10. First I installed Pip with apt install python-pip. Then I installed cffi with pip install cffi which also pulled in pycparser. Then I made 4 files in my home directory: add.c, add.h, build.py, and add.py which are as follows:
add.c:
#include "add.h" //if you don't have a header file you will get a warning
int addme(int a, int b)
{
return (a + b);
}
add.h
int addme(int a, int b);
build.py:
from cffi import FFI
ffibuilder = FFI()
ffibuilder.cdef("int addme(int a, int b);")
ffibuilder.set_source("pyadd",'#include "add.h"',sources=["add.c"])
ffibuilder.compile()
add.py:
from pyadd.lib import addme
print(addme(2,6))
I entered python build.py which created pyadd.so which I was then able to import into my python script. Then I entered python add.py which returned 8. The first argument in set_source() is the name of the shared object that you will then import in your python script. Both "from pyadd.lib import addme" and "from pyadd import lib" seem to work but with the latter you have to refer to addme as "lib.addme". This must just be a python thing.

iOS 6: libpython2.7.a initialization import error

Stuck on initialization Embedded Python on iOS.
I took build script from Kivy project.
It produced libpython2.7.a, Python27.zip and includes.
So, my app unpacks zip to /Documents/lib/python2.7
Trying to set up Python:
dirPaths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory,
NSUserDomainMask, YES);
docsDir = [dirPaths objectAtIndex:0];
Py_SetPythonHome((char *)[docsDir UTF8String]);
Py_SetProgramName("");
Py_Initialize();
Falls on:
ImportError: No module named site
As I understand, it cant locate Python home path.
But if I fix site by:
Py_NoSiteFlag=1;
PyRun_SimpleString("import sys as s\nprint(s.path)");
It outputs valid pathes:
['/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python27.zip', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/plat-darwin', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/plat-mac', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/plat-mac/lib-scriptpackages', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/lib-tk', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/lib-old', '/var/mobile/Applications/1BC015FC-2F7A-41C0-8F3A-70A22510C3A3/Documents/lib/python2.7/lib-dynload']
But no one import works.
What I must fix to make Python correctly work with modules?

Already fixed it by myself.
My mistakes were:
- Need to place all python /lib file in base program files as directory (not group)
- Add this directory in build phases "Upload Resources" list
- In build phase put "Upload resources" stage upper than "Compile program". Don't know why, but it helps.

Run a particular process without using sudo

I understand my question is not that clear so let me try explaining it here.
What I am trying to do is suspend my computer after completing a certain task.
My script:
#my logic or task(Downloading a file)#
cmd = shlex.split("sudo pm-suspend")
>>> subprocess.call(cmd)
[sudo] password for noob:
As you can see it asks for my password but the problem is I will not be sitting on my system while the script is getting executed.
So, my question is how to handle this problem or what other alternative way exist to tackle this problem.
Python version is 2.7.3.

Assuming your pm-suspend is at /usr/sbin/pm-suspend
Make your sudoers entry in /etc/sudoers:
<username> ALL=(ALL) NOPASSWD: /usr/sbin/pm-suspend

You can use the SetUID bit to cause a script to run with the permissions of the owner of the file. You can allow a specific file to always run as root by changing its owner to root and setting its SetUID bit. To set the SetUID bit:
chmod 4755 <filename>
masks: 4000 is SetUID, 0700 is owner rwx, 0050 and 0005 are group and world rx.
It is imperative that you make sure that users other than the owner cannot edit this file, because if they can, they will be able to run arbitrary commands as your user, which is a security risk.
To be effective as you need it, you must also set the file's owner to root:
sudo chown root <filename>
In this case, <filename> should be whatever script you intend to run. It must be executable - if it is not, i.e. you are trying to run a python program not set up to be executed standalone, you will need to use a wrapper that launches it.
More information: http://en.wikipedia.org/wiki/Setuid
Be careful, there are a number of security risks associated with using the SetUID bit. Post further comments if you need clarification.
A commenter has pointed out that in all likelihood, this will not work for shell scripts. You will instead need to use a wrapper that will call your process from a compiled language such as C or C++.
/* setuid_wrapper.cpp */
#include <unistd.h>
int main(int c, char * v[])
{
// the program to execute
// replace this with the program you want to call.
const char * executable = "/bin/false";
// arguments to pass to program
// MUST be null terminated, MUST start with executable path
const char * arguments[] = {executable, "arg1", "arg2", "etc...", NULL};
execv(executable, arguments);
perror("execv: ");
return 1;
}
compile with:
g++ -o setuid_wrapper setuid_wrapper.cpp
Follow the directions earlier to change its owner to root and set the SetUID bit, and configure your system to run it instead of your script when needed.