We Keep Coding

Python subprocess.run can't find /bin/sh in chroot

(First off, apologies for the roughness of this question's writing-- would love any constructive feedback.)
Ok what I'm doing is a bit involved-- I'm trying to make a Python script that executes Bash scripts that each compile a component of a Linux From Scratch (LFS) system. I'm following the LFS 11.2 book pretty closely (but not 100%, although I've been very careful to check where my deviations break things. If you're familiar with LFS, this is a deviation that breaks things).
Basically, my script builds a bunch of tools (bash, tar, xz, make, gcc, binutils) with a cross compiler, and tells their build systems to install them into a directory lfs/temp-tools. Then the script calls os.chroot('lfs') to chroot into the lfs directory, and immediately resets all the environment variables (most importantly PATH) with:
os.environ = {"PATH" : "/usr/bin:/usr/sbin:/temp-tools/bin", ...other trivial stuff like HOME...}
But after the chroot, my calls of
subprocess.run([f"{build_script_path} >{log_file_path} 2>&1"], shell=True)
are failing with FileNotFoundError: [Errno 2] No such file or directory: '/bin/sh', even though
bin/sh in the chroot directory is a sym link to bash
there's a perfectly good copy of bash in /temp-tools/bin
calling print(os.environ) after the python chroot shows /temp-tools/bin is in PATH
I thought maybe subprocess.run is stuck using the old environment variables, before I reset them upon entering the chroot, but adding env=os.environ to subprocess.run does not help. :/ I'm stuck for now
For context if it helps, here is where the subprocess.run call gets made:
def vanilla_build(target_name, src_dir_name=None):
def f():
nonlocal src_dir_name
if src_dir_name == None:
src_dir_name = target_name
tarball_path = find_tarball(src_dir_name)
src_dir_path = tarball_path.split(".tar")[0]
if "tcl" in src_dir_path:
src_dir_path = src_dir_path.rsplit("-",1)[0]
snap1 = lfs_dir_snapshot()
os.chdir(os.environ["LFS"] + "srcs/")
subprocess.run(["tar", "-xf", tarball_path], check=True, env=os.environ)
os.chdir(src_dir_path)
build_script_path = f"{os.environ['LFS']}root/build-scripts/{target_name.replace('_','-')}.sh"
log_file_path = f"{os.environ['LFS']}logs/{target_name}"
####### The main call #######
proc = subprocess.run([f"{build_script_path} >{log_file_path} 2>&1"],
shell=True, env=os.environ)
subprocess.run(["rm", "-rf", src_dir_path], check=True)
if proc.returncode != 0:
red_print(build_script_path + " failed!")
return
tracked_file_record_path = f"{os.environ['LFS']}logs/tracked/{target_name}"
with open(tracked_file_record_path, 'w') as f:
new_files = lfs_dir_snapshot() - snap1
f.writelines('\n'.join(new_files))
f.__name__ = "build_" + target_name
return f
And how I enter the chroot:
def enter_chroot():
os.chdir(os.environ["LFS"])
os.chroot(os.environ["LFS"])
os.environ = {"HOME" : "/root",
"TERM" : os.environ["TERM"],
"PATH" : "/usr/bin:/usr/sbin:/temp-tools/bin",
"LFS" : '/'}
Thank you! In the meantime I'm going to chop away as much code as possible to isolate the problem to either understand whatever I'm not getting or rewrite this question to be less context specific

Running xlwings on a server

Solved the problem by making sure that python27.dll and win32api into the PYTHON_WIN directory. The two files with an arrow were missing in the server folder (P:/)
I am using udf module of xlwings and setting xlwings.bas configuration into a server. (I have my personal disk C:/ however other users need to run this macro and the file must be stored in P:/)
I have python, xlwings and other modules installed in P:/, but I get some strange error about importing os lib. Does any one have ever passed through this ?
Thanks in advance.
I got this from python:
"Python process exited before it was possible to create the interface object.
Command: P:\Python 27\python.exe -B -c ""import sys, os;sys.path.extend(os.path.normcase(os.path.expandvars(r'P:\Risco\Python Scripts')).split(';'));import xlwings.server; xlwings.server.serve('{2bb649ad-487e-48d5-ab31-24adaeefca59}')""
Working Dir: "
Xlwings.bas is set like this:
PYTHON_WIN = "P:\Python 27\python.exe"
PYTHON_MAC = ""
PYTHON_FROZEN = ThisWorkbook.Path & "\build\exe.win32-2.7"
PYTHONPATH = "P:\Risco\Python Scripts"
UDF_MODULES = "FetchPL"
UDF_DEBUG_SERVER = False
LOG_FILE = ""
SHOW_LOG = True
OPTIMIZED_CONNECTION = False

How do I put python script on path?

I know this question has been asked before. I've done just about all the suggestions, but I'm still having an issue.
I have a script file_rename_script.py that I have made executable, and I've tested away from the path to make sure that it's working as expected, all is well. However, when I move that script to my path, I get the following:
/Users/Nick/bin/file_rename_script.py: line 1: import: command not found
from: can't read /var/mail/datetime
/Users/Nick/bin/file_rename_script.py: line 4: syntax error near unexpected token `('
/Users/Nick/bin/file_rename_script.py: line 4: `current_directory = os.listdir(os.getcwd())'
Here is what the file looks like:
#!/usr/bin/env python3
import os, time
from datetime import datetime
current_directory = os.listdir(os.getcwd())
files = [f for f in os.listdir(os.getcwd()) if os.path.isfile(f)]
for file in files:
file_name, file_extension = os.path.splitext(file)
create_date = datetime.fromtimestamp(os.path.getctime(file)).strftime('%Y_%m_%d')
prompt_user = "What would you like to name " + file + " to? "
user_response = input(prompt_user)
if user_response:
os.rename(file, create_date + "-" + user_response.replace(" ", "_") + file_extension)
print("Renamed your file to", create_date + "-" + user_response.replace(" ", "_") + file_extension)
print()
else:
print(file, "was not changed")
print()
Why is it that I can't run this when it's on my path? Also note, I've had the shebang set as #!/usr/local/bin/python3 as well with no success.
Ideally I'd like to keep this file somewhere else, and I'd like to symlink it as an alias on my path.
What gives?

Errors indicate that she-bang is not working, and the shell is trying to run your python code (/Users/Nick/bin/file_rename_script.py) directly as a shell script, not using a python interpreter...
First verify that you get Python prompt if you run /usr/bin/env python3. If not, try to look for python3 by running which python3
Second, double check that there are no spaces or empty lines before the she-bang line #!. If you are creating file on Windows, save the file without BOM (this is an invisible character at the beginning of a file). Line must end with a new line character (\n). You can run head and od to print bytes (characters) from the first line of the file:
head -1 /Users/Nick/bin/file_rename_script.py | od -cx
It should print (The first character is #, and the last one is \n):
0000000 # ! / u s r / b i n / e n v p
2123 752f 7273 622f 6e69 652f 766e 7020
0000020 y t h o n 3 \n
7479 6f68 336e 000a
0000027

You should add the python exe location to environment path if you use windows or linux.
For widows and linux
Add the python exexutable path variable to path
If you use virtual env,please activate the virrual env first.add the virtual env python package path.such as
!#env python
Script must be add into upper directory of env
Must run the env call env/Scripts/activate if you use virtual env
Directory structure should be
App
-- env
-- yourscript.py
Install the virtual env by pip.

What does "Error 309" mean?

In our build we're creating an executable file with unit tests like this:
tests = env.Program(os.path.join(env['testDir'], name + '_test'),
src + createManifest(env),
LIBS = libs,
LIBPATH = buildLibPath(env),
LINKFLAGS = env['LINKFLAGS'],
CPPPATH = cppPath)
This correctly creates an executable, which later is being run by the following builder:
action = tests[0].abspath + '&& echo %DATE% %TIME% > ${TARGET}'
runTests = env.Command(source = tests,
target = 'test_'+name+'.tmp',
action = action)
Up to this point everything works fine: the tests are being run during the build.
I've recently found Visual Leak Detector tool and wanted to include this in the build. So, I've changed the environment for the builders like this:
vldInclude = os.path.join(os.path.normpath(env['vldIncDir']), 'vld.h')
env.Append(CPPFLAGS='/FI' + vldInclude)
env.Append(LIBPATH = env['vldLibDir'])
vldLib = os.path.join(env['vldLibDir'], 'vld.lib')
libs.append(vldLib) # used in the Program call for the LIBS parameter, see above
scons: *** [build\debug\libname\test_libname.dummy] Error 309
This error message isn't very helpful. What does it mean and how to fix it?

It turns out that the magic number 309 is more googleable when written as: 0xC0000135 (no idea why C, but 135HEX == 309DEC), and it is an identifier of the STATUS_DLL_NOT_FOUND error.
So, it's not a SCons error, but Windows error, that leaks through SCons.
This means that some DLLs are missing, needed by VLD. Lurking into the VLD installation directory (usually: C:\Program Files (x86)\Visual Leak Detector) two DLL files and one manifest file can be found in the bin\Win32 subdirectory.
Not to have the build being dependent on the machine's environment, you can either add the directory to env['ENV']['PATH'] or copy the files to the directory where the tests are being run.
To do the latter:
You need another VLD configuration option, beside the library directory, namely the binaries directory. Let's call it vldBinDir. At the build's startup you can copy these files to the build directory:
def setupVld(env):
sourcePath = env['vldBinDir']
targetPath = env['testDir']
toCopy = ['dbghelp.dll',
'vld_x86.dll',
'Microsoft.DTfW.DHL.manifest']
nodes = []
for c in toCopy:
n = env.Command(os.path.join(targetPath, c),
os.path.join(sourcePath, c),
SCons.Defaults.Copy("${TARGET}", "${SOURCE}"))
nodes.append(n)
env['vldDeps'] = nodes
And then, when creating particular tests, make sure to add the dependency:
for n in env['vldDeps']:
env.Depends(tests, n)

Activate Virtual Environment and Install Packages via Script

I often find myself recreating file structures for Flask apps so I have decided to make a script to do all that for me. I would like the script to create all the folders I need as well as the files with some basic boilerplate, which it does, that part is working fine. However I would also like to create a virtual environment and install Flask to that environment. That is where I am encountering the problem. The script runs but it installs Flask to my system installation of Python.
I followed the advice in this question here but it's not working. I am running Ubuntu 12.04.4 LTS via crouton on a Chromebook.
#!/usr/bin/python
from os import mkdir, chdir, getcwd, system
import sys
APP_NAME = sys.argv[1]
ROOT = getcwd()
PROJECT_ROOT = ROOT + '/' + APP_NAME
# dictionary represents folder structure. Key is the folder name and the value is it's contents
folders = {APP_NAME : {'app' : {'static': {'css' : '', 'img' : '', 'js' : ''}, 'templates' : ''} } }
def create_folders(dic):
for key in dic:
if isinstance(dic[key], dict):
mkdir(key)
prev = getcwd() + '/' + key
chdir(prev)
create_folders(dic[key])
else:
mkdir(key)
create_folders(folders)
chdir(PROJECT_ROOT)
open('config.py', 'a').close()
with open('run.py', 'a') as run:
run.write("""stuff""")
with open('app/__init__.py', 'a') as init:
init.write("""stuff""")
with open('app/views.py', 'a') as views:
views.write("""stuff""")
open('app/models.py', 'a').close()
open('app/forms.py', 'a').close()
with open('app/templates/layout.html', 'a') as layout:
layout.write("""stuff""")
system('chmod a+x run.py')
system('virtualenv venv')
system('. venv/bin/activate;sudo pip install flask') # this does not seem to be working the way I am expecting it to

I suppose your calls are not within the same console session and therefore the console environment is not as expected. I suggest to concatenate the related commands in one system call using subprocess.Popen like this (including suggestions by limasxgoesto0):
subprocess.Popen('virtualenv venv;source venv/bin/activate;pip install flask')
You should probably be using subprocess; os.system is deprecated.