I'm trying to assign the name of the file that is extracted from the zip as a string that I can pass around.
The following is my code so far:
cl = '7z x -ppassowrd Week45_10.zip'
args = shlex.split(cl)
p = subprocess.call(args)
file = open('VENDATA_10', 'r')
VENDATA_10 is the name of the file that is extracted, however, the name of the file extracted from the zip will vary. How do I find the name of the extracted file so that I can open whatever that file is called instead of VENDATA_10?
You could capture the output from the process. 7z will print the filenames that it extracts to the standard output. The output might look like this:
7-Zip 4.44 beta Copyright (c) 1999-2007 Igor Pavlov 2007-01-20
p7zip Version 4.44 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)
Processing archive: Week45_10.zip
Extracting VENDATA_10
Everything is Ok
So you want to capture the lines that start with "Extracting ". To capture the output you'll need to pipe the output from the process into you program and read it. You can used communicate() from subprocess to get the full output.
I've extended you code a bit to do this:
from subprocess import Popen, PIPE
import shlex
cl = '7z x -ppassowrd Week45_10.zip'
args = shlex.split(cl)
p = Popen(args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
(stdoutdata, stderrdata) = p.communicate()
START_WITH = "Extracting "
N = len(START_WITH)
lines = [line.strip() for line in stdoutdata.split('\n')]
files = [line[N:] for line in lines if line.startswith(START_WITH)]
print files
You should be aware that there might arise problems if the files already exist, because then 7z will prompt the user if it should overwrite the files. If you want to handle that then you need to control the input to the 7z process also. You might want to look at the pexpect module for that.
Related
I am trying to use the below code to run a command and extract the data from the cmd.
the file with the commands and data is a txt file. (let me know if I should change it or use an excel if better).
the commands look something like this: ping "host name" which would result in some data in the cmd.there is list of these in the file. so it would ping "hostname1" then line two ping "hostname2"..etc
THE QUESTION: I want it to run every line individually and extract the results from the cmd and store them in a txt file or excel file - Ideally I want all the results in the same file. is this possible? and how?
here is the code so far:
root_dir = pathlib.Path(r"path to file here")
cmds_file = root_dir.joinpath('actual file here with commands and data')
#fail = []
cmds = cmds_file.read_text().splitlines()
try:
for cmd in cmds:
args = cmd.split()
print(f"\nRunning: {args[0]}")
output = subprocess.check_output(args)
print(output.decode("utf-8"))
out_file = root_dir.joinpath(f"Name of file where I want results printed in")
out_file.write_text(output.decode("utf-8"))
except:
pass
You can use a module called subprocess import subprocess
Then you can define a variable like this
run = subprocess.run(command_to_execute, capture_output=True)
After that you can do print(run.stdout) to print the command output.
If you want to write it to a file you can do this after you run the above code
with open("PATH TO YOUR FILE", "w") as file:
file.write(run.stdout)
This should write a file which contains the output of your command
After that close the file using file.close() and reopen it but in "a" mode
with open("PATH TO YOUR FILE", "a") as file:
file.write(\n + run.stdout)
This should append data to your file.
Remember to close the file just for best practice, I have some bad memorys about not closing the file after I opened it :D
My plan is simple:
Open input, output file
Read input file line by line
Execute the command and direct the output to the output file
#!/usr/bin/env python3
import pathlib
import shlex
import subprocess
cmds_file = pathlib.Path(__file__).with_name("cmds.txt")
output_file = pathlib.Path(__file__).with_name("out.txt")
with open(cmds_file, encoding="utf-8") as commands, open(output_file, "w", encoding="utf-8") as output:
for command in commands:
command = shlex.split(command)
output.write(f"\n# {shlex.join(command)}\n")
output.flush()
subprocess.run(command, stdout=output, stderr=subprocess.STDOUT, encoding="utf-8")
Notes
Use shlex.split() to simulate the bash shell's command split
The line output.write(...) is optional. You can remove it
With subprocess.run(...), the stdout=output will redirect the command's output to the file. You don't have to do anything.
Update
I updated the subprocess.run line to redirect stderr to stdout, so error will show.
Im trying to record docker stats for every file in the mydata directory. For example if one of the files is names piano.txt I would like the output file to be piano_stuff.txt. This is what I have so far:
import subprocess
import signal
import os
for file_name in os.listdir('mydata'):
data_txt = "./" + file_name.split(".")[0] + "_stuff.txt"
dockerStats = subprocess.Popen("docker stats --format {{.MemUsage}} >> ${data_txt}", shell=True)
os.killpg(os.getpgid(dockerStats.pid), signal.SIGTERM)
Don't use shell=True. Open the file locally, and pass the file object as the stdout argument. You can also use the --no-stream option to have the command exit after producing one line of output, rather than asynchronously trying to kill the process as soon as possible. (You might get multiple lines of output, or you might get none, depending on when the OS schedules the Docker process to run.)
with open(data_txt, "a") as f:
subprocess.run(["docker", "stats", "--format", "{{.MemUsage}}", "--no-stream"], stdout=f)
In Python 2.7 I have the following code inside certain loop
file = open("log.txt", 'a+')
last_position = file.tell()
subprocess.Popen(["os_command_producing_error"], stderr = file)
file.seek(last_position)
error = file.read()
print(error) # example of some action with the error
The intention is that the error that was just given by stderr gets, say printed, while file is keeping the whole record.
I am a beginner in Python and I am not clear what happens in the stderr = file.
My problem is that error keeps being empty, even though errors keep getting logged in the file.
Could someone explain why?
I have tried adding closing and opening the file again, or file.flush() right after the subprocess line. But still the same effect.
Edit: The code in the answer below makes sense to me and it seems to work for for the author of that post. For me (in Windows) it is not working. It gives an empty err and an empty file log.txt. If I run it line by line (e.g. debugging) it does work. How to understand and solve this problem?
Edit: I changed the Popen with call and now it works. I guess call waits for the subprocess to finish in order to continue with the script.
error is empty because you are reading too soon before the process has a chance to write anything to the file. Popen() starts a new process; it does not wait for it to finish.
call() is equivalent to Popen().wait() that does wait for the child process to exit that is why you should see non-empty error in this case (if the subprocess does write anything to stderr).
#!/usr/bin/env python
import subprocess
with open("log.txt", 'a+') as file:
subprocess.check_call(["os_command_producing_error"], stderr=file)
error = file.read()
print(error)
You should be careful with mixing buffered (.read()) and unbuffered I/O (subprocess).
You don't need the external file here, to read the error:
#!/usr/bin/env python
import subprocess
error = subprocess.check_output(["os_command_producing_error"],
stderr=subprocess.STDOUT)
print(error)
It merges stderr and stdout and returns the output.
If you don't want to capture stdout then to get only stderr, you could use Popen.communicate():
#!/usr/bin/env python
import subprocess
p = subprocess.Popen(["os_command_producing_error"], stderr=subprocess.PIPE)
error = p.communicate()[1]
print(error)
You could both capture stderr and append it to a file:
#!/usr/bin/env python
import subprocess
error = bytearray()
p = subprocess.Popen(["os_command_producing_error"],
stderr=subprocess.PIPE, bufsize=1)
with p.stderr as pipe, open('log.txt', 'ab') as file:
for line in iter(pipe.readline, b''):
error += line
file.write(line)
p.wait()
print(error)
See Python: read streaming input from subprocess.communicate().
Try these following codes:
file = open("log.txt", 'a+')
sys.stderr = file
last_position = file.tell()
try:
subprocess.call(["os_command_producing_error"])
except:
file.close()
err_file = open("log.txt", 'r')
err_file.seek(last_position)
err = err_file.read()
print err
err_file.close()
sys.stderr map the standard error message like sys.stdout(map standard output) and sys.stdin(map standard input).
And this will map the standard error to file. So all of the standard error will be write to the file log.txt.
I'm trying to call a shell script (segment.sh) within python.
The syntax that produce correct results at the console is:
> ./segment.sh ctb file.txt utf-8 0
As can be seen, this shell script expects an text file as input. What I am trying to do is to open up the file and read its content in python (later preferably from an HTML POST form), and somehow pass the variable containing the content to the python subprocess
call.
The following function is not working. But if I just provide a file name such as:
Popen(["/bin/bash", "./segment.sh", "ctb", "file.txt", "utf-8", "0"])`
then, it will work, but I want to pass the input from a variable. Would you please give me some pointers?
def pySegment(text):
op = subprocess.Popen(["/bin/bash", "./segment.sh", "ctb", "utf-8", "0"],
stdout = subprocess.PIPE,
stdin = subprocess.PIPE,
stderr = subprocess.STDOUT,
)
results = op.communicate(input=text)[0]
return results
if __name__ == "__main__":
filename = "./file.txt"
text = open(filename).read()
result = pySegment(text)
print result
I suggest to use a named pipe:
import os, tempfile, shutil, subprocess
temp_dir = tempfile.mkdtemp()
filename = os.path.join(temp_dir, 'file.txt')
text = '<text>'
os.mkfifo(filename)
try:
subprocess.Popen(('segment.sh', 'ctf', filename, 'utf-8', '0'))
with open(filename, 'w') as f:
f.write(text)
finally:
shutil.rmtree(temp_dir)
The named pipe will offer the same interface of a file without really creating that file as you need.
This is the python script will do. The question is how to call the external cmd file within the function?
Read a CSV file in the directory.
If the content in 6th column is equal to 'approved', then calls an
external windows script 'TransferProd.cmd'
.
def readCSV(x):
#csvContents is a list in the global scope that will contain lists of the
#items on each line of the specified CSV file
try:
global csvContents
file = open(csvDir + x + '.csv', 'r') #Opens the CSV file
csvContents = file.read().splitlines() #Appends each line of the CSV file to csvContents
#----------------------------------------------------------------------
#This takes each item in csvContents and splits it at "," into a list.
#The list created replaces the item in csvContents
for y in range(0,len(csvContents)):
csvContents[y] = csvContents[y].lower().split(',')
if csvContents[y][6] == 'approved':
***CALL TransferProd.cmd***
file.close()
return
except Exception as error:
log(logFile, 'An error has occurred in the readCSV function: ' + str(error))
raise
Take a look at the subprocess module.
import subprocess
p = subprocess.Popen(['TransferProd.cmd'])
You can specify where you want output/errors to go (directly to a file or to a file-like object), pipe in input, etc.
import os
os.system('TransferProd.cmd')
This works in both unix/windows flavors as it send the commands to the shell. There are some variations in returned values though! Check here.
If you don't need output of the command you could use: os.system(cmd)
The better solution is to use:
from subprocess import Popen, PIPE
proc = Popen(cmd, shell = True, close_fds = True)
stdout, stderr = proc.communicate()