We Keep Coding

Python subprocess.run can't find /bin/sh in chroot

(First off, apologies for the roughness of this question's writing-- would love any constructive feedback.)
Ok what I'm doing is a bit involved-- I'm trying to make a Python script that executes Bash scripts that each compile a component of a Linux From Scratch (LFS) system. I'm following the LFS 11.2 book pretty closely (but not 100%, although I've been very careful to check where my deviations break things. If you're familiar with LFS, this is a deviation that breaks things).
Basically, my script builds a bunch of tools (bash, tar, xz, make, gcc, binutils) with a cross compiler, and tells their build systems to install them into a directory lfs/temp-tools. Then the script calls os.chroot('lfs') to chroot into the lfs directory, and immediately resets all the environment variables (most importantly PATH) with:
os.environ = {"PATH" : "/usr/bin:/usr/sbin:/temp-tools/bin", ...other trivial stuff like HOME...}
But after the chroot, my calls of
subprocess.run([f"{build_script_path} >{log_file_path} 2>&1"], shell=True)
are failing with FileNotFoundError: [Errno 2] No such file or directory: '/bin/sh', even though
bin/sh in the chroot directory is a sym link to bash
there's a perfectly good copy of bash in /temp-tools/bin
calling print(os.environ) after the python chroot shows /temp-tools/bin is in PATH
I thought maybe subprocess.run is stuck using the old environment variables, before I reset them upon entering the chroot, but adding env=os.environ to subprocess.run does not help. :/ I'm stuck for now
For context if it helps, here is where the subprocess.run call gets made:
def vanilla_build(target_name, src_dir_name=None):
def f():
nonlocal src_dir_name
if src_dir_name == None:
src_dir_name = target_name
tarball_path = find_tarball(src_dir_name)
src_dir_path = tarball_path.split(".tar")[0]
if "tcl" in src_dir_path:
src_dir_path = src_dir_path.rsplit("-",1)[0]
snap1 = lfs_dir_snapshot()
os.chdir(os.environ["LFS"] + "srcs/")
subprocess.run(["tar", "-xf", tarball_path], check=True, env=os.environ)
os.chdir(src_dir_path)
build_script_path = f"{os.environ['LFS']}root/build-scripts/{target_name.replace('_','-')}.sh"
log_file_path = f"{os.environ['LFS']}logs/{target_name}"
####### The main call #######
proc = subprocess.run([f"{build_script_path} >{log_file_path} 2>&1"],
shell=True, env=os.environ)
subprocess.run(["rm", "-rf", src_dir_path], check=True)
if proc.returncode != 0:
red_print(build_script_path + " failed!")
return
tracked_file_record_path = f"{os.environ['LFS']}logs/tracked/{target_name}"
with open(tracked_file_record_path, 'w') as f:
new_files = lfs_dir_snapshot() - snap1
f.writelines('\n'.join(new_files))
f.__name__ = "build_" + target_name
return f
And how I enter the chroot:
def enter_chroot():
os.chdir(os.environ["LFS"])
os.chroot(os.environ["LFS"])
os.environ = {"HOME" : "/root",
"TERM" : os.environ["TERM"],
"PATH" : "/usr/bin:/usr/sbin:/temp-tools/bin",
"LFS" : '/'}
Thank you! In the meantime I'm going to chop away as much code as possible to isolate the problem to either understand whatever I'm not getting or rewrite this question to be less context specific

Access Tools from .venv

What is the best way to access python tools from within a script?
For my example I want to use msgfmt.py and pygettext from the Tools/i18n package(?).
On a Linux system probably no issue, since they are already on the PATH, but under Windows I have to call them with python as interpreter, so setting the directory on the path doesn't work like in Linux.
So is calling os.system('my_pygettext_command') actually the right attempt or should I do this with a different approach, like importing? If importing would be correct, how can I access them, if they are only installed on the system installation and not in venv

Researching a bit further into the topic, I got really mind gobbled:
Maybe I am doing this totally wrong under windows, but my line ratio (linux/windows) to call a python tool from inside a venv: 1/34. I did not fully tested the final call under linux yet, but this ratio is only for getting the subprocess command.
This is my momentary solution and I am open for better approaches:
windows-utils
import sys
from typing import Dict
def stdout_get_command_to_dict(string: str):
lines = [s for s in string.split("\n") if s]
# remove header
lines = lines[2:]
stdout_dict = {}
for idx, line in enumerate(lines):
# Reduce Spaces
while " " in line:
line = line.replace(" ", " ")
line_as_list = line.split()
stdout_dict[idx] = {
"Version": line_as_list[2][:5],
"Source": line_as_list[3],
"Venv": line_as_list[3].find("venv") > 0,
}
return stdout_dict
def get_system_py_path(stdout_dict: Dict[int, Dict[str, str]]):
major = sys.version_info.major
minor = sys.version_info.minor
micro = sys.version_info.micro
env_version = f"{major}.{minor}.{micro}"
for key in stdout_dict.keys():
if stdout_dict[key]["Version"] == env_version and not stdout_dict[key]["Venv"]:
return stdout_dict[key]["Source"]
and the script:
if platform.system() == "Windows":
cmd = ["powershell", "get-command", "python", "-totalCount", "4"]
processed_cmd = subprocess.run(cmd, shell=True, capture_output=True, text=True)
stdout_as_dict = stdout_get_command_to_dict(processed_cmd.stdout)
sys_python_path = Path(get_system_py_path(stdout_as_dict)).parent
tool = sys_python_path / "Tools/i18n/msgfmt.py"
tool_cmd = f"python {tool}"
elif platform.system() == "Linux":
tool_cmd = "msgfmt"

How to program hotstrings in python like in autohotkey

I want to make hotstrings in python that converts one word when typed into another after some processing, since AHK is very limiting when it comes to determining which word to type. Right now, I am using a hotstring in ahk that runs code on the command line that runs a python script with the word that I typed as arguments. Then I use pyautogui to type the word. However, this is very slow and does not work when typing at speed. I'm looking for a way to do this all with python and without ahk, but I have not found a way to do hotstrings in python. For example, every time I type the word "test" it replaces it with "testing." Thanks for your help. I'm running the latest version of Python and Windows 10 if that is useful to anyone by the way.

(if you want to process it as each letter is typed(t,te,tes,test), you should edit your question)
I call my SymPy functions using ahk hotkeys. I register the python script as a COM server and load it using ahk.
I do not notice any latency.
you'll need pywin32, but don't download using pip install pywin32
download from https://github.com/mhammond/pywin32/releases
OR ELSE IT WON'T WORK for AutoHotkeyU64.exe, it will only work for AutoHotkeyU32.exe.
make sure to download amd64, (I downloaded pywin32-300.win-amd64-py3.8.exe)
here's why: how to register a 64bit python COM server
toUppercase COM server.py
class BasicServer:
# list of all method names exposed to COM
_public_methods_ = ["toUppercase"]
#staticmethod
def toUppercase(string):
return string.upper()
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("Error: need to supply arg (""--register"" or ""--unregister"")")
sys.exit(1)
else:
import win32com.server.register
import win32com.server.exception
# this server's CLSID
# NEVER copy the following ID
# Use "print(pythoncom.CreateGuid())" to make a new one.
myClsid="{C70F3BF7-2947-4F87-B31E-9F5B8B13D24F}"
# this server's (user-friendly) program ID
myProgID="Python.stringUppercaser"
import ctypes
def make_sure_is_admin():
try:
if ctypes.windll.shell32.IsUserAnAdmin():
return
except:
pass
exit("YOU MUST RUN THIS AS ADMIN")
if sys.argv[1] == "--register":
make_sure_is_admin()
import pythoncom
import os.path
realPath = os.path.realpath(__file__)
dirName = os.path.dirname(realPath)
nameOfThisFile = os.path.basename(realPath)
nameNoExt = os.path.splitext(nameOfThisFile)[0]
# stuff will be written here
# HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\${myClsid}
# HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{C70F3BF7-2947-4F87-B31E-9F5B8B13D24F}
# and here
# HKEY_LOCAL_MACHINE\SOFTWARE\Classes\${myProgID}
# HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Python.stringUppercaser
win32com.server.register.RegisterServer(
clsid=myClsid,
# I guess this is {fileNameNoExt}.{className}
pythonInstString=nameNoExt + ".BasicServer", #toUppercase COM server.BasicServer
progID=myProgID,
# optional description
desc="return uppercased string",
#we only want the registry key LocalServer32
#we DO NOT WANT InProcServer32: pythoncom39.dll, NO NO NO
clsctx=pythoncom.CLSCTX_LOCAL_SERVER,
#this is needed if this file isn't in PYTHONPATH: it tells regedit which directory this file is located
#this will write HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{C70F3BF7-2947-4F87-B31E-9F5B8B13D24F}\PythonCOMPath : dirName
addnPath=dirName,
)
print("Registered COM server.")
# don't use UseCommandLine(), as it will write InProcServer32: pythoncom39.dll
# win32com.server.register.UseCommandLine(BasicServer)
elif sys.argv[1] == "--unregister":
make_sure_is_admin()
print("Starting to unregister...")
win32com.server.register.UnregisterServer(myClsid, myProgID)
print("Unregistered COM server.")
else:
print("Error: arg not recognized")
you first need to register the python COM server:
first, get your own CLSID: just use a python shell.
import pythoncom
print(pythoncom.CreateGuid())
then, set myClsid to that output
to register:
python "toUppercase COM server.py" --register
to unregister:
python "toUppercase COM server.py" --unregister
hotstring python toUppercase.ahk
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
#SingleInstance, force
SendMode Input ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir% ; Ensures a consistent starting directory.
SetBatchLines, -1
#KeyHistory 0
ListLines Off
#Persistent
#MaxThreadsPerHotkey 4
pythonComServer:=ComObjCreate("Python.stringUppercaser")
; OR
; pythonComServer:=ComObjCreate("{C70F3BF7-2947-4F87-B31E-9F5B8B13D24F}") ;use your own CLSID
; * do not wait for string to end
; C case sensitive
:*:hello world::
savedHotstring:=A_ThisHotkey
;theActualHotstring=savedHotstring[second colon:end of string]
theActualHotstring:=SubStr(savedHotstring, InStr(savedHotstring, ":",, 2) + 1)
send, % pythonComServer.toUppercase(theActualHotstring)
return
f3::Exitapp
you can test the speed of hotstring hello world, it's very fast for me.
Edit def toUppercase(string): to your liking

how to pass a python variable into a bash script function

I have searched and searched and searched but have not found and answer to my particular situation, or if I had I don't know how to implement it. I'm trying to create a script using python and bash to automate my project creation process. Whereas I would run the testing.py "as it is called now because I'm testing" with 1 command line argument which would be the name of the project folder, then it would ask "where I would like to create the project" with two options for paths to store the project and store the path using an if statement into the path variable. then I would like to pass that variable in the bash script to navigate to it and create the project directory there. here is what I have so far:
so the command that I would run is like:
python testing.py newProject
in the .test.sh
#!/bin/bash
function testing() {
cd
desktop
python testing.py $1
}
and in the testing.py
import sys
import os
school = "/Users/albert/Open/Microverse/"
personal = "/Users/albert/Open/code/Projects/"
path = " "
userInput = input("What type of project would you like to create? Personal or School? ")
if userInput.lower() == "personal":
path = personal
elif userInput.lower() == "school":
path = school
def testing():
folderName = str(sys.argv[1])
os.makedirs(path + str(folderName))
print(folderName)
if __name__ == "__main__":
testing()

Use "$1" instead of $1 as shown below
python testing.py "$1"

You have to run your bash function like this
testing $1
The whole script
#!/bin/bash
function testing() {
cd
desktop
python test.py $1
}
testing $1
And change input() to raw_input() in your python part. Input() function returns will actually evaluate the input string and try to run it as Python code. And if i try to input Personal this give me an error
What type of project would you like to create? Personal or School? Personal
Traceback (most recent call last):
File "test.py", line 9, in <module>
userInput = input("What type of project would you like to create? Personal or School? ")
File "<string>", line 1, in <module>
NameError: name 'Personal' is not defined
And if you input personal than userInput will be /Users/albert/Open/code/Projects/ wich is not wat you want right?

I'm guessing desktop is not actually a valid command, and that you really want
testing () {
cd ~/Desktop
python test.py "$1"
]
A better design might be to not force all projects to be created inside your desktop folder, though.
testing () {
python ~/Desktop/test.py "$1"
}
will let you run the script in any directory.
A still better design would be to give the script a proper shebang, mark it executable, call it testing, and save it in your PATH; then you don't need a shell function at all.
A different and IMHO more usable approach would be to allow the user to specify "personal" or "school" as part of the command line, instead of forcing the script to use interactive input. Requiring interactive I/O makes the script harder to use as a building block in further automation, and robs the user of the ability to use the shell's history and completion features.
The following refactoring hard-codes a very crude option parser, and uses a dict to map the project type to a specific folder name.
#!/usr/bin/env python3
import sys
import os
paths = {
"school": "/Users/albert/Open/Microverse/"
"personal": "/Users/albert/Open/code/Projects/"
}
def makeproject(type, folder):
os.makedirs(os.path.join(path, folder))
def gettype():
while True:
userInput = input("What type of project would you like to create? Personal or School? ")
if userInput in paths:
return paths[userInput]
print("Not a valid option: {} -- try again".format(userInput))
def testing():
idx = 1
if sys.argv[idx] == "--school":
type = paths["school"]
idx = 1
elif sys.argv[idx] == "--personal":
type = paths["personal"
idx = 2
else:
type = gettype()
folderName = sys.argv[idx]
makeproject(type, folderName)
print(folderName)
if __name__ == "__main__":
testing()
Notice that sys.argv and the value returned by input are already strings; there is no need to call str() on them.

Python failing to create symlinks

I'm trying to use python to make symbolic links. When I try to use os.symlink I get a WinError 5 "Access is denied" error, but if I try to create the same symlink using mklink, it works fine. (I'm running this as an administrator, and have the 'Create symbolic links' privilege.)
I then to use subprocess.call to call mklink directly with the proper arguments and such, and it runs without any errors, yet when it's finished the symlinks have not been created.
When I try to use Popen to run the command I get a 'system cannot find the file specified' error even though the directory is there.
This is on Windows Server 2008.
def createStudentFolders(studentList):
grad_year = input("Please enter the year of graduation for the class of students being created (i.e. 2016): ")
for student in studentList:
student = student.replace(" ","") # Gets rid of the white space in the name
studentPath = 'cpwd/{}'.format(student)
try:
os.mkdir(studentPath)
except:
pass
print(studentPath)
proc = Popen("icacls {} /inheritance:e /grant STUDENT\{}:(OI)(CI)F /T".format(studentPath, student))
classFolder = "students/ClassOf{}".format(grad_year)
try:
os.mkdir(classFolder)
except:
pass
os.symlink('{}/{}'.format(classFolder, student), studentPath, target_is_directory=True)
print("Done")
def main():
newStudentsLoc = "newStudents.txt"
studentList = readStudentsToList(newStudentsLoc)
createStudentFolders(studentList)
main()
input("Pausing")
File resides in Z:\
inside that are
cpwd\<student folders>
students\ClassOf<Year>\<symlinks to student folders go here>
I've tried using absolute paths as well with no luck.