We Keep Coding

Running .bat file in Google Colab

I am trying to run a .bat file in my Google Colab notebook, howere I cannot seem to make it happen. Whenever I navigate to the folder the code says the directory or file does not exist.
from subprocess import Popen
p = Popen("batch.bat", cwd=r"/content/drive/MyDrive/sd/stable-diffusion/merge-models-main/")
stdout, stderr = p.communicate()

Colab is an Ubuntu Linux environment so it will struggle if the file to be run contains Windows like commands. If the file contains Linux shell commands then the following code illustrates how to execute these.
This cell makes a batch.bat file (purists would argue that it should be batch.sh).
# This is a Unix shell script
with open('batch.bat', 'w') as f:
f.write('var=$(date)\r\n')
f.write('echo "$var" > output.txt\r\n')
The file is placed into /content/ by default. If you want to use a file from your own Google Drive, you have to mount this yourself.
To execute the commands in the file do this. Note how Popen takes a list with the location of the file to execute as the second parameter.
from subprocess import Popen
p = Popen(["/bin/sh", "/content/batch.bat"])
stdout, stderr = p.communicate()
Look for the file output.txt and observe the timestamp in it. This should give an indication whether it is working.

File paths within my Python script fail when launching via double-click (Windows 10)

I have a python script that reads and writes to files that are located relative to it, in directories above and beside it. When I run my script via Cygwin using
python script.py
The program works perfectly. However, when I run it by navigating through the windows GUI to my file and double clicking, I get a blank cmd prompt and then my program runs fine until I reach the point where I need to access the other files, at which point it fails and gives me this message in the cmd prompt that opens itself:
../FFPRM.TXT
../2025510296/FFPRM_000.TXT
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\rbanks\AppData\Local\Programs\Python\Python35-32\lib\tkinter\__init__.py", line 1549, in __call__
return self.func(*args)
File "C:\Users\rbanks\Desktop\TSAC\EXECUTABLE\T-SAC_GUI.py", line 705, in run_exe
invalid_entry, output_text = self.apply()
File "C:\Users\rbanks\Desktop\TSAC\EXECUTABLE\T-SAC_GUI.py", line 694, in apply
p = subprocess.Popen(['cp', output_file_path, output_file_path_id])
File "C:\Users\rbanks\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "C:\Users\rbanks\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 1220, in _execute_child startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I am deploying this script as well as the directory structure as a zip for users to be able to unzip and use anywhere on their PC, so it is important for me to be able to run it with a simple double click and my relative file paths.
My first thought was the cmd prompt that was opening and executing my script was in a different environment, but when I run:
cd
pause
in a .cmd script, I get:
C:\Users\rbanks\Desktop\TSAC\EXECUTABLE>pause
Which is the correct location.
I am not having any luck with Google, I assume because I can't seem to construct a sufficient search query. Could someone point me in the right direction please?

[edit] the other answer is correct(at least i suspect) but i will leave this here in the hopes that it helps the op in the future with path problems ... and doing something like this is just generally good practice
use
BASEPATH = os.path.abspath(os.path.dirname(__file__))
at the top of your script
the later
txt_file = os.path.join(BASEPATH,"my_file.txt")
or even
txt_file = os.path.join(BASEPATH,"..","my_file.txt")
this gives you the benefit of being able to do things like
if not os.path.exists(txt_file):
print "Cannot find file: %r"%txt_file
which will likely give you a better idea about what your problem might actually be (if its simply path related at least)

The problem is not the current directory. It is correct when double clicking on the icon
The problem is: Cygwin commands are not in the windows path
You are using python, so don't run simple copy commands like this, which make your script non-portable and subject to variations, requiring installation of cygwin, etc...
p = subprocess.Popen(['cp', output_file_path, output_file_path_id])
can be replaced by
import shutil
shutil.copyfile(output_file_path, output_file_path_id)
shutil module complements os module for file manipulation functions that aren't native in the operating system.
now you have a 100% pythonic solution, native, which will throw exceptions if cannot read/write files, so fully integrated in the rest of your program.
Before running an external command from Python make sure that no python way exists. There are so many useful modules out there.
Other examples of how to avoid running basic commands from python (of course if you need to run a C compilation it's different!):
zipfile package: much better than running zip.exe
gzip package: can open gzipped files natively from python
os.listdir() instead of parsing the output of cmd /c dir /B
os.remove() instead of calling rm or del
etc... python rules!

subprocess open ('source venv/bin/activate'),no such file?

I want get into virtual environment in python files.But it raise no such files.
import subprocess
subprocess.Popen(['source', '/Users/XX/Desktop/mio/worker/venv/bin/activate'])
Traceback (most recent call last):
File "/Users/Ru/Desktop/mio/worker/run.py", line 3, in
subprocess.Popen(['source', '/Users/Ru/Desktop/mio/worker/venv/bin/activate'])
File"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File"/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I think your code doesn't work because you are separating the 'source' command from the virtualenv path argument, from the documentation:
"Note in particular that options (such as -input) and arguments (such
as eggs.txt) that are separated by whitespace in the shell go in
separate list elements, while arguments that need quoting or
backslash escaping when used in the shell (such as filenames
containing spaces or the echo command shown above) are single list
elements."
You should try one of two things:
First, write the source and the virtualenv file path as a single string argument:
import subprocess
subprocess.Popen(['source '/Users/XX/Desktop/mio/worker/venv/bin/activate'])
I'm working on OSX and that doesn't seem to work, but it might be due to the shell you are using. To ensure this will work, you can use the shell=True flag:
import subprocess
subprocess.Popen(['source '/Users/XX/Desktop/mio/worker/venv/bin/activate'],shell=True)
This will use /bin/sh shell by default. Again, you can read more in the documentation.
Tom.

There is another simpler way to do what you want.
If you want a python script to use a virtualenv you can always use the python interpreter from the virualenv itself.
/Users/Ru/Desktop/mio/worker/venv/bin/python my_python_file.py
This will run the my_python_file.py with the properties/libraries of the virtualenv.
If you want to run that file inside a subprocess you can do something similar to the method I described above:
import subprocess
subprocess.Popen(['/Users/Ru/Desktop/mio/worker/venv/bin/python my_python_file.py])
and have my_python_file.py import pika and do other actions you wish to do.
Tom.

OSError: [Error 1] Operation not permitted

I am trying to run a python script which uses a binary file (xFiles.bin.addr_patched) created by a postlinker. However, I am getting this error.
File "abc.py", line 74, in ParseCmd
shutil.copy(gOptions.inputX, gWorkingXFile)
File "/usr/lib/python2.6/shutil.py", line 89, in copy
copymode(src, dst)
File "/usr/lib/python2.6/shutil.py", line 66, in copymode
os.chmod(dst, mode)
OSError: [Errno 1] Operation not permitted: 'myPath/xFiles.bin.addr_patched'
When I checked the permissions of this xFiles.bin, by ls-l, it shows that
-rwxrwxrwx 1 nobody nogroup
I presume the error is because this file was created by some other application, the python script I am running does not have access to it. Since I am beginner wrt ubuntu, I don't really know how to fix it. Any suggestions on how to fix this?
SOLVED:
As one of the answers Suggested : chown username:groupname file name fixes this issue

You could try (from the command line, but I'm sure there's a syntax in python):
sudo chown your_username:your_groupname filename
Note: The group is usually just your username.
I feel like there's something wrong with those permissions though. Read Write Execute for everyone seems to be off. How was this file created? How did it get to be created by the user nobody?

Python code to change the permission:
from getpwnam import pwd
from getgrnam import grp
import os
uid = getpwnam("YOUR_USERNAME")[2]
gid = grp.getgrnam("YOUR_GROUPNAME")[2]
os.chown("myPath/xFiles.bin.addr_patched", uid, gid)
Run the script with sudo and you're done.

I had this problem when running a python script on my mac (10.14 Mojave) trying to access /Users/xxx/Pictures/Photos Library.photoslibrary.
The full solution can be found in http://osxdaily.com/2018/10/09/fix-operation-not-permitted-terminal-error-macos/
Summary:
Go to System Preferences > Security & Privacy > Privacy > Full Disk Access and add your IDE or python interpreter to the list.

My guess is that you should be looking at the permissions for myPath folder instead. Seems like you can't write to it, hence the problem. Try ls -l myPath/.. and see the permissions for myPath. If that's the problem, change the permissions on the folder with chmod.
P.S. Also, see Google top result on Linux file permissions.

Using subprocess to run Python script on Windows

Is there a simple way to run a Python script on Windows/Linux/OS X?
On the latter two, subprocess.Popen("/the/script.py") works, but on Windows I get the following error:
Traceback (most recent call last):
File "test_functional.py", line 91, in test_functional
log = tvnamerifiy(tmp)
File "test_functional.py", line 49, in tvnamerifiy
stdout = PIPE
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
monkut's comment: The use case isn't clear. Why use subprocess to run a python script? Is there something preventing you from importing the script and calling the necessary function?
I was writing a quick script to test the overall functionality of a Python-command-line tool (to test it on various platforms). Basically it had to create a bunch of files in a temp folder, run the script on this and check the files were renamed correctly.
I could have imported the script and called the function, but since it relies on sys.argv and uses sys.exit(), I would have needed to do something like..
import sys
import tvnamer
sys.argv.append("-b", "/the/folder")
try:
tvnamer.main()
except BaseException, errormsg:
print type(errormsg)
Also, I wanted to capture the stdout and stderr for debugging incase something went wrong.
Of course a better way would be to write the script in more unit-testable way, but the script is basically "done" and I'm doing a final batch of testing before doing a "1.0" release (after which I'm going to do a rewrite/restructure, which will be far tidier and more testable)
Basically, it was much easier to simply run the script as a process, after finding the sys.executable variable. I would have written it as a shell-script, but that wouldn't have been cross-platform. The final script can be found here

Just found sys.executable - the full path to the current Python executable, which can be used to run the script (instead of relying on the shbang, which obviously doesn't work on Windows)
import sys
import subprocess
theproc = subprocess.Popen([sys.executable, "myscript.py"])
theproc.communicate()

How about this:
import sys
import subprocess
theproc = subprocess.Popen("myscript.py", shell = True)
theproc.communicate() # ^^^^^^^^^^^^
This tells subprocess to use the OS shell to open your script, and works on anything that you can just run in cmd.exe.
Additionally, this will search the PATH for "myscript.py" - which could be desirable.

Yes subprocess.Popen(cmd, ..., shell=True) works like a charm. On Windows the .py file extension is recognized, so Python is invoked to process it (on *NIX just the usual shebang). The path environment controls whether things are seen. So the first arg to Popen is just the name of the script.
subprocess.Popen(['myscript.py', 'arg1', ...], ..., shell=True)

It looks like windows tries to run the script using its own EXE framework rather than call it like
python /the/script.py
Try,
subprocess.Popen(["python", "/the/script.py"])
Edit: "python" would need to be on your path.

For example, to execute following with command prompt or BATCH file we can use this:
C:\Python27\python.exe "C:\Program files(x86)\dev_appserver.py" --host 0.0.0.0 --post 8080 "C:\blabla\"
Same thing to do with Python, we can do this:
subprocess.Popen(['C:/Python27/python.exe', 'C:\\Program files(x86)\\dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:\\blabla'], shell=True)
or
subprocess.Popen(['C:/Python27/python.exe', 'C:/Program files(x86)/dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:/blabla'], shell=True)

You are using a pathname separator which is platform dependent. Windows uses "\" and Unix uses "/".

When you are running a python script on windows in subprocess you should use python in front of the script name. Try:
process = subprocess.Popen("python /the/script.py")

Supplemental info: It is worth noting that the documentation states that you need to use shell=True if you are using a dos shell command like dir
without it you get something like this.
>>> import subprocess
>>> subprocess.run(['dir'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __ini
t__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _exe
cute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>> subprocess.run(['dir'], shell=True)
Volume in drive J is garbage
Volume Serial Number is 5EE7-B084
Also you can use path like objects for the args which is recent addition.
from pathlib import Path
subprocess.run(Path('c:/proj/myfile.bat'))
Also worth noting there is a whole set of windows specific controls that allow you to control how a process is spawned which concurrent operations can use.
So controlling subprocesses on windows is not as simple as posix style.