I want to run subprocess.call (or any other subprocess function) with a cwd that is really long (longer than 260 characters). I am using a recent Windows 10.
I read here that in order to support long paths, you either have to set a registry key or add \\?\ in front of the path. I did both.
It works if the executable I want to run has a long path. But it does not work if the cwd is a long path:
import os, sys
import subprocess
PATH_TO_WRITE_EXE = r"C:\Windows\write.exe"
print(os.path.isfile(PATH_TO_WRITE_EXE))
# error:
my_cwd = "\\\\?\\C:\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path"
print(os.path.isdir(my_cwd))
# no error:
#my_cwd = "\\\\?\\C:\\a\\not\\so\\long\\path"
#print(os.path.isdir(my_cwd))
o = subprocess.call([PATH_TO_WRITE_EXE], timeout=None, cwd=my_cwd)
print(o)
Note that os.path.isdir() returns True on both the short and the long path.
How can I use a long path as cwd on Windows 10?
This is just a proof of concept, and you will would probably want to do something different, but here is "an" answer that will work if you run your script as admin (which is a bad idea... maybe? (depending on the scope)).
import os
import win32file
import subprocess
long_path = '\\\\?\\C:\\Temp\\3d\\RsTYjcEwAA26\\aFmtI0e\\v\\ZZ7\\AWgMBtUP5\\JRGtyZXFj2\\f2rqXnYX3yJ4\\39X11fdRbYEA\\NtPySHqx\\htyDGAtZWv8NDK\\d2VRFFJPuBUVXET\\2QSlBOlMkgO8h\\mES\\sQfPZ1nBAKZNIogOb\\wyGm5Z0RwHV\\n54Si\\2BqDwGnK6TOxjs2P\\p4SnwEre4\\KQzs1NXu5QEZcuZOIct\\YrMfsGq5g5gnMN69ko\\QFIq\\J4IKjZ3vxNrC\\OVDWtz\\Jp1H0M1UclBJqeBuX\\bjN7dA\\lCFmKDg7G1\\OhYtim9AxgX9Bm9\\vrLaaL\\KLvkkJeI0ofdwb\\Es\\ZJi3Q54oIXbQ8NOi10\\VR\\HH3\\O\\5\\zn7\\7EKj96k3BC\\8Q1OqP\\FdX8RLhl1Ce\\mPG\\OtmJWbzFk\\AheYZ8Ypwo\\085dmIvlrg\\Y8tmeJt\\cDYqXPq\\G6EYcqVXaLxv\\XXq6tIfVDhv8WoF\\xM\\PCYkVfFT1Uam9N0e\\G9PfRMOv\\GUWbc6eot4aEuVQIMd\\0NMEq9iDzqgLGOJx09\\HpUN5rBfaq9\\Ve\\Tp0E\\wpXyehjLotcDa4x\\HlBy1LMD83sxzQF0\\1\\NH1be07kdb61aomggou\\D0\\SF\\n0NLPfYTEh\\3k1AooSmx4y2CS6Mrp\\sgAd9N6x1v31jZ\\hof1X6XGdBAU8\\zyzuxVDHuX54PiYW0\\nVJc8\\r\\ukx63N2kY\\6gf8dhUTYad\\L8w4JWwZq\\iixvKOcH13FXljY5D\\zgGuUlXFH1hd\\2Ykw1isPKOKXR4Osv1U\\ncmRIMWf\\i1ioae6pqcsfDsI\\AU7fhnbPCtpaOphXL\\Vxn\\gJFO1o6JAMBmBWP\\8EKwcdps\\JGd\\SgfwKrEd5\\pGSxLp\\DuA8th1\\YRx8u0LF8Cgs6JEfwA\\dIV0Ay\\PEc2\\CSli\\nyRaOzgBtLuM8S09st\\vMd9Ctvc8c6\\2\\H5tpHh\\K6TsNhH\\jXmon6\\BqvEDk\\gsMH20FxEgwlY'
file_name = "test_file"
symlink_name = "C:\\Temp\\long_link"
os.makedirs(long_path)
with open(os.path.join(long_path, file_name), "w") as file:
file.write("I'm some test data in a long path!")
win32file.CreateSymbolicLink(symlink_name, long_path, 0x3)
subprocess.call("type %s" % file_name, shell=True, timeout=None, cwd=symlink_name)
I'm some test data in a long path!0
As #eryksun mentioned in the comments: Creating a symbolic link requires SeCreateSymbolicLinkPrivilege, which by default is only assigned to elevated administrators. (However, it can be explicitly assigned to users and groups.) If os.symlink raises OSError, you can create a junction via _winapi.CreateJunction or CMD's mklink /j command.
Finally here is another answer which should enable the same behavior if you create a junction. I have not tested this answer in conjunction with your question, but it should work.
Edit: If you're running >= Python 3.5 you can use the CreateJunction call to replace the symlink above.
import _winapi
_winapi.CreateJunction(source, target)
I'm trying to create a loop inside a Shell Script and I want to break out of the loop and finish the shell script execution when i find an integer different than 0 in a specific string(using Python).The problem is even after the first occurrence of an integer different than 0 in that specific string the shell script keeps executing.I tried to debug it by echoing the value of GET_OUT_OF_LOOP but it just keeps echoing 0 even after finding the kind of integer I was looking for. I already looked on the web for a way to do this but I still didn't figure it out...
Here's my shell script:
#!/bin/sh
export GET_OUT_OF_LOOP=0
while [ $GET_OUT_OF_LOOP -ne 1 ]; do
python3 provas.py provas.txt
./provas < provas.txt >> data.txt
python3 test.py data.txt
sh clear_data.sh
done
And here is my Python code(test.py) where I'm trying to change the value of the GET_OUT_OF_LOOP variable using os.environ:
#!usr/env/bin python3
import sys
import os
import re
script, filename = sys.argv
os.environ['GET_OUT_OF_LOOP'] = '0'
fin = open("data.txt", 'r')
for line in fin:
if "A percentagem de aprovação foi de" in line:
if int(re.search(r'\d+', line).group()) != 0:
print(line)
os.environ['GET_OUT_OF_LOOP'] = '1'
The python process is a subprocess of the shell process, and it can not modify environment vars of its parent process.
For your case, you can use the exit code to pass the message; i.e.
shell script:
python3 test.py data.txt || GET_OUT_OF_LOOP=1
python:
#!usr/env/bin python3
import sys
import os
import re
script, filename = sys.argv
fin = open("data.txt", 'r')
for line in fin:
if "A percentagem de aprovação foi de" in line:
if int(re.search(r'\d+', line).group()) != 0:
print(line)
sys.exit(1)
sys.exit(0)
That is just the way environment variables work: you can't in a sub-process change variables in the environment of the process which called it.
(And in shell script, almost all lines of code, but for control structures, are external sub-processes)
What you can have is a simple unsigned byte return value of your sub-process that can be read in the shell script as the implicit $? variable.
In Python's case, you terminate the program with this return value by calling sys.exit()
So, in your shell script you can do this to assign the variable:
python3 test.py data.txt
GET_OUT_OF_LOOP=$?
And the Python in the Python script change:
os.environ['GET_OUT_OF_LOOP'] = '1'
for
sys.exit(1)
Of course, it would be much more sane and maintainable if yu just use Python all the way from the top - the shutils module in the stdlib makes it easy to copy files around, and you, above all, get a consistent syntax across all lines of your script, much easier to use comparison operators and variables.
Here are two similar stackoverflow questions that might explain yours:
how-do-i-make-environment-variable-changes-stick-in-python
environment-variables-in-python-on-linux
So the real reason causing this issue is that when we run a process, the environment variables being changed by the process are only available during the process runtime, it won't change the external variables, here is a simplified script of yours to prove it:
#test.py
import os
os.environ['test_env_var'] = '1'
#test.sh
export test_env_var=0
while [ $test_env_var -ne 1 ]; do
python test.py
echo $test_env_var
done
As you might have already seen what's coming, the loop will echo $tev to be 0 forever.
Hence the solution to solve this problem to my understanding, would be to out-source the change into the external system files, if it's necessary. Append changes to the configuration files of the regarding systems, for instance of this example, you can append "export test_env_var=1" into ~/.bashrc, if you are a linux bash user.
So I have this uber script which constantly checks the system path for a program (openvpn). When you install openvpn it adds itself to the system path. I run my script in the console and, while it runs and checks, I install openvpn. In that console my script will never find openvpn in sys path. If I open a new console and run the same script it finds it.
Any idea how I can make my script a little less dumb?
import os
import time
import subprocess
def cmd( command ):
return subprocess.check_output( command, shell = True )
def program_in_path( program ):
path = cmd( "path" ).split(";")
for p in path:
if "openvpn" in p.lower():
return True
return False
if __name__ == '__main__':
while True:
print program_in_path("openvpn")
time.sleep( 2 )
I presume it's from the shell = True thing but how else would I find it if not with path or WHERE openvpn /Q ? Running with no sehll I get WindowsError: [Error 2] The system cannot find the file specified
Here's slightly the same program done in ruby which works 100%:
loop do
puts system( "WHERE openvpn /Q" )
sleep( 5 )
end
Unfortunately my project is too deep into python to switch languages now. Too bad.
It's actually because when your program starts, it has an environment configured. Part of that environment is the system path. When you start a subshell, it inherits the environment of the parent process.
I'm not a Windows programmer, and I don't have a Windows machine available to test on right now. But according to that bug report, if you import nt in your script and reload(nt) in your while True loop that it will pull down a fresh copy of the environment from the system. I don't know whether that's true or not. It might be worth a try.
For what it's worth, you can see the same behavior from the cmd window by, for instance, opening a command window, adding a program folder to the System Path, and then trying to run an exe from that program folder in your existing cmd window. It won't work -- but open a new cmd window, and it will.
The bug report you cite is about a different problem. That problem outlined there is that from within Python, if you load in one of the system DLLs and use a particular function Windows provides for manipulating your environment, Python does not reflect the change. However, if you make a change to os.environ, Python recognizes that change. The conclusion from the community was that the particular function that the reporter was using, was not the correct function to use to get the results he expected.
Perhaps this approach works for you, getting the PATH variable straight from the registry (since you're on Windows).
For instance you could do something like this:
import winreg
def PathFromReg():
loc = r'SYSTEM\CurrentControlSet\Control\Session Manager\Environment'
reg = winreg.ConnectRegistry(None, winreg.HKEY_LOCAL_MACHINE)
key = winreg.OpenKey(reg, loc)
n_val = winreg.QueryInfoKey(key)[1]
for i in range(n_val):
val = winreg.EnumValue(key, i)
if val[0] == 'Path':
return val[1]
path = PathFromReg()
print('openvpn' in path.lower())
I think you only need to assign the key once and then query the values inside the loop.
Note: In Python 2 the module is called _winreg.
Background
I would like my Python script to pause before exiting using something similar to:
raw_input("Press enter to close.")
but only if it is NOT run via command line. Command line programs shouldn't behave this way.
Question
Is there a way to determine if my Python script was invoked from the command line:
$ python myscript.py
verses double-clicking myscript.py to open it with the default interpreter in the OS?
If you're running it without a terminal, as when you click on "Run" in Nautilus, you can just check if it's attached to a tty:
import sys
if sys.stdin and sys.stdin.isatty():
# running interactively
print("running interactively")
else:
with open('output','w') as f:
f.write("running in the background!\n")
But, as ThomasK points out, you seem to be referring to running it in a terminal that closes just after the program finishes. I think there's no way to do what you want without a workaround; the program is running in a regular shell and attached to a terminal. The decision of exiting immediately is done just after it finishes with information it doesn't have readily available (the parameters passed to the executing shell or terminal).
You could go about examining the parent process information and detecting differences between the two kinds of invocations, but it's probably not worth it in most cases. Have you considered adding a command line parameter to your script (think --interactive)?
What I wanted was answered here: Determine if the program is called from a script in Python
You can just determine between "python" and "bash". This was already answered I think, but you can keep it short as well.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import psutil
import os
ppid = os.getppid() # Get parent process id
print(psutil.Process(ppid).name())
I don't think there's any reliable way to detect this (especially in a cross-platform manner). For example on OS X, when you double-click a .py file and it tuns with "Python Launcher", it runs in a terminal, identically to if you execute it manually.
Although it may have other issues, you could package the script up with something like py2exe or Platypus, then you can have the double-clickable icon run a specific bit of code to differentiate (import mycode; mycode.main(gui = True) for example)
If you run python IDLE then "pythonw.exe" is being used to run coding while when you run the command line "python.exe" is used to run coding. The python folder path can vary so you have to revert the path to the python folder. m = '\\' and m = m[0] is to get m to be '\' because of escaping.
import sys
a = sys.executable
m = '\\'
m = m[0]
while True:
b = len(a)
c = a[(b - 1)]
if c == m:
break
a = a[:(b - 1)]
if sys.executable == a + 'pythonw.exe':
print('Running in Python IDLE')
else:
print('Running in Command line')
Update for later versions (e.g. Python 3.6 on Ubuntu 16.04): The statement to get the name has changed to psutil.Process(os.getpid()).parent().name()
I believe this CAN be done. At least, here is how I got it working in Python 2.7 under Ubuntu 14.04:
#!/usr/bin/env python
import os, psutil
# do stuff here
if psutil.Process(os.getpid()).parent.name == 'gnome-terminal':
raw_input("Press enter to close...")
Note that -- in Ubuntu 14 with the Gnome desktop (aka Nautilus) -- you might need to do this:
from a Nautilus window (the file browser), select Edit(menu)->Preferences(item) then Behavior(tab)->Executable Text Files(section)->Ask Each Time(radio).
chmod your script to be executable, or -- from a Nautilus window (the file browser) -- right click on the file->Properties(item) then Permissions(tab)->Execute:Allow executing file as program(checkbox)
double-click your file. If you select "Run in Terminal", you should see the "Type enter to close..." prompt.
now try from a bash prompt; you should NOT see the prompt.
To see how this works, you can fiddle with this (based on the answer by from #EduardoIvanec):
#!/usr/bin/env python
import os
import sys
import psutil
def parent_list(proc=None, indent=0):
if not proc:
proc = psutil.Process(os.getpid())
pid = proc.pid
name = proc.name
pad = " " * indent
s = "{0}{1:5d} {2:s}".format(pad, pid, name)
parent = proc.parent
if parent:
s += "\n" + parent_list(parent, indent+1)
return s
def invoked_from_bash_cmdline():
return psutil.Process(os.getpid()).parent.name == "bash"
def invoked_as_run_in_terminal():
return psutil.Process(os.getpid()).parent.name == "gnome-terminal"
def invoked_as_run():
return psutil.Process(os.getpid()).parent.name == "init"
if sys.stdin.isatty():
print "running interactively"
print parent_list()
if invoked_as_run_in_terminal():
raw_input("Type enter to close...")
else:
with open('output','w') as f:
f.write("running in the background!\n")
f.write("parent list:\n")
f.write(parent_list())
From the idea behind this answer, adding for Win10 compatibility (Ripped from Python 2.7 script; modify as needed):
import os, psutil
status = 1
if __name__ =="__main__":
status = MainFunc(args)
args = sys.argv
running_windowed = False
running_from = psutil.Process(os.getpid()).parent().name()
if running_from == 'explorer.exe':
args.append([DEFAULT OR DOUBLE CLICK ARGS HERE])
running_windowed = True
if running_windowed:
print('Completed. Exit status of {}'.format(status))
ready = raw_input('Press Enter To Close')
sys.exit(status)
There is a number of switch like statements you could add to be more universal or handle different defaults.
This is typically done manually/, I don't think there is an automatic way to do it that works for every case.
You should add a --pause argument to your script that does the prompt for a key at the end.
When the script is invoked from a command line by hand, then the user can add --pause if desired, but by default there won't be any wait.
When the script is launched from an icon, the arguments in the icon should include the --pause, so that there is a wait. Unfortunately you will need to either document the use of this option so that the user knows that it needs to be added when creating an icon, or else, provide an icon creation function in your script that works for your target OS.
My solution was to create command line scripts using setuptools. Here are a the relevant parts of myScript.py:
def main(pause_on_error=False):
if run():
print("we're good!")
else:
print("an error occurred!")
if pause_on_error:
raw_input("\nPress Enter to close.")
sys.exit(1)
def run():
pass # run the program here
return False # or True if program runs successfully
if __name__ == '__main__':
main(pause_on_error=True)
And the relevant parts of setup.py:
setup(
entry_points={
'console_scripts': [
'myScript = main:main',
]
},
)
Now if I open myScript.py with the Python interpreter (on Windows), the console window waits for the user to press enter if an error occurs. On the command line, if I run 'myScript', the program will never wait for user input before closing.
Although this isn't a very good solution, it does work (in windows at least).
You could create a batch file with the following contents:
#echo off
for %%x in (%cmdcmdline%) do if /i "%%~x"=="/c" set DOUBLECLICKED=1
start <location of python script>
if defined DOUBLECLICKED pause
If you want to be able to do this with a single file, you could try the following:
#echo off
setlocal EnableDelayedExpansion
set LF=^
:: The 2 empty lines are necessary
for %%x in (%cmdcmdline%) do if /i "%%~x"=="/c" set DOUBLECLICKED=1
echo print("first line of python script") %LF% print("second and so on") > %temp%/pyscript.py
start /wait console_title pyscript.py
del %temp%/pyscript.py
if defined DOUBLECLICKED pause
Batch code from: Pausing a batch file when double-clicked but not when run from a console window?
Multi-line in batch from: DOS: Working with multi-line strings
Okay, the easiest way I found and made was to simply run the program in the command line, even if it was ran in the Python IDLE.
exist = lambda x: os.path.exists(x) ## Doesn't matter
if __name__ == '__main__':
fname = "SomeRandomFileName" ## Random default file name
if exist(fname)==False: ## exist() is a pre-defined lambda function
jot(fname) ## jot() is a function that creates a blank file
os.system('start YourProgram.py') ## << Insert your program name here
os.system('exit'); sys.exit() ## Exits current shell (Either IDLE or CMD)
os.system('color a') ## Makes it look cool! :p
main() ## Runs your code
os.system("del %s" % fname) ## Deletes file name for next time
Add this to the bottom of your script and once ran from either IDLE or Command Prompt, it will create a file, re-run the program in the CMD, and exits the first instance.
Hope that helps! :)
I also had that question and, for me, the best solution is to set an environment variable in my IDE (PyCharm) and check if that variable exists to know if the script is being executed either via the command line or via the IDE.
To set an environment variable in PyCharm check:
How to set environment variables in PyCharm?
Example code (environment variable: RUNNING_PYCHARM = True):
import os
# The script is being executed via the command line
if not("RUNNING_PYCHARM" in os.environ):
raw_input("Press enter to close.")
I hope it works for you.
Based on existing solutions and using sets:
import psutil
def running_interactively():
"""Return True if any of our parent processes is a known shell."""
shells = {"cmd.exe", "bash.exe", "powershell.exe", "WindowsTerminal.exe"}
parent_names = {parent.name() for parent in psutil.Process().parents()}
# print(parent_names)
# print(f"Shell in parents? {shells & parent_names}")
return bool(shells & parent_names)
if not running_interactively():
input("\nPress ENTER to continue.")
This answer is currently specific to Windows, but it can be reconfigured to work with other operating systems in theory. Rather than installing psutil module like most of these answers recommend, you can make use of the subprocess module and the Windows tasklist command to explicitly get the name of the parent process of your Python program.
import os
import subprocess
shells = {"bash.exe", "cmd.exe", "powershell.exe", "WindowsTerminal.exe"}
# These are standard examples, but it can also be used to detect:
# - Nested python.exe processes (IDLE, etc.)
# - IDEs used to develop your program (IPython, Eclipse, PyCharm, etc.)
# - Other operating system dependent shells
s = subprocess.check_output(["tasklist", "/v", "/fo", "csv", "/nh", "/fi", f"PID eq {os.getppid()}"])
# Execute tasklist command to get the verbose info without the header (/nh) of a single process in CSV format (/fo csv)
# Such that its PID is equal to os.getppid()
entry = s.decode("utf-8").strip().strip('"').split('","')
# Decode from bytes to str, remove end whitespace and quotations from CSV format
# And split along the quote delimited commas
# This process may differ and require adjustment when used for an OS other than Windows
condition = entry and entry[0] in shells
# Check first that entry is not an empty sequence, meaning the process has already ended
# If it still exists, check if the first element -- the executable -- exists as an element of the set of executables you're looking for
I hope this is helpful for anyone looking for an answer to this problem while minimizing the number of dependencies you'd need.
This was tested in Python 3.8 and uses an f-string in the subprocess.check_output line of the code, so please be sure to convert the f-string to a compatible syntax if you're working with a version of Python before f-strings were introduced.
the supper class I use is http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/, my code is below:
import os
import sys, time
from daemon import Daemon
class MyDaemon(Daemon):
def run(self):
while True:
cmd='cat test.txt > output.txt'
os.system(cmd)
time.sleep(6000)
if __name__ == "__main__":
daemon = MyDaemon('/tmp/DebugDaemon.pid')
daemon.start()
If I run DebugDaemon.py, the /tmp/DebugDaemon.pid can be created.
However, ouput.txt file can not be created, why?
If I call it directly (ie: No using the daemon code) work fine.
cmd is a local variable. Your assignment to it doesn't actually do anything, since no code uses it.
The subprocess module allows you to call other programs from within Python. I don't know how it interacts with daemons though.
Daemon appears to chdir() to /. I bet your process doesn't have write permissions for /.
Your daemon needs to chdir() to the directory where test.txt resides (and for which the process has write permissions). Alternatively, use full paths everywhere:
cmd = 'cat /tmp/test.txt > /tmp/output.txt'
the
cat test.txt > output.txt
is executed in / because the super class does
# decouple from parent environment
os.chdir("/")
the pid-file can be written, because everybody can write to /tmp - / is not writable for everybody.