Im trying to record docker stats for every file in the mydata directory. For example if one of the files is names piano.txt I would like the output file to be piano_stuff.txt. This is what I have so far:
import subprocess
import signal
import os
for file_name in os.listdir('mydata'):
data_txt = "./" + file_name.split(".")[0] + "_stuff.txt"
dockerStats = subprocess.Popen("docker stats --format {{.MemUsage}} >> ${data_txt}", shell=True)
os.killpg(os.getpgid(dockerStats.pid), signal.SIGTERM)
Don't use shell=True. Open the file locally, and pass the file object as the stdout argument. You can also use the --no-stream option to have the command exit after producing one line of output, rather than asynchronously trying to kill the process as soon as possible. (You might get multiple lines of output, or you might get none, depending on when the OS schedules the Docker process to run.)
with open(data_txt, "a") as f:
subprocess.run(["docker", "stats", "--format", "{{.MemUsage}}", "--no-stream"], stdout=f)
Related
I am trying to use the below code to run a command and extract the data from the cmd.
the file with the commands and data is a txt file. (let me know if I should change it or use an excel if better).
the commands look something like this: ping "host name" which would result in some data in the cmd.there is list of these in the file. so it would ping "hostname1" then line two ping "hostname2"..etc
THE QUESTION: I want it to run every line individually and extract the results from the cmd and store them in a txt file or excel file - Ideally I want all the results in the same file. is this possible? and how?
here is the code so far:
root_dir = pathlib.Path(r"path to file here")
cmds_file = root_dir.joinpath('actual file here with commands and data')
#fail = []
cmds = cmds_file.read_text().splitlines()
try:
for cmd in cmds:
args = cmd.split()
print(f"\nRunning: {args[0]}")
output = subprocess.check_output(args)
print(output.decode("utf-8"))
out_file = root_dir.joinpath(f"Name of file where I want results printed in")
out_file.write_text(output.decode("utf-8"))
except:
pass
You can use a module called subprocess import subprocess
Then you can define a variable like this
run = subprocess.run(command_to_execute, capture_output=True)
After that you can do print(run.stdout) to print the command output.
If you want to write it to a file you can do this after you run the above code
with open("PATH TO YOUR FILE", "w") as file:
file.write(run.stdout)
This should write a file which contains the output of your command
After that close the file using file.close() and reopen it but in "a" mode
with open("PATH TO YOUR FILE", "a") as file:
file.write(\n + run.stdout)
This should append data to your file.
Remember to close the file just for best practice, I have some bad memorys about not closing the file after I opened it :D
My plan is simple:
Open input, output file
Read input file line by line
Execute the command and direct the output to the output file
#!/usr/bin/env python3
import pathlib
import shlex
import subprocess
cmds_file = pathlib.Path(__file__).with_name("cmds.txt")
output_file = pathlib.Path(__file__).with_name("out.txt")
with open(cmds_file, encoding="utf-8") as commands, open(output_file, "w", encoding="utf-8") as output:
for command in commands:
command = shlex.split(command)
output.write(f"\n# {shlex.join(command)}\n")
output.flush()
subprocess.run(command, stdout=output, stderr=subprocess.STDOUT, encoding="utf-8")
Notes
Use shlex.split() to simulate the bash shell's command split
The line output.write(...) is optional. You can remove it
With subprocess.run(...), the stdout=output will redirect the command's output to the file. You don't have to do anything.
Update
I updated the subprocess.run line to redirect stderr to stdout, so error will show.
So I know how to write in a file or read a file but how do I RUN another file?
for example in a file I have this:
a = 1
print(a)
How do I run this using another file?
file_path = "<path_to_your_python_file>"
using subprocess standard lib
import subprocess
subprocess.call(["python3", file_path])
or using os standard lib
import os
os.system(f"python3 {file_path}")
or extract python code from the file and run it inside your script:
with open(file_path, "r+", encoding="utf-8") as another_file:
python_code = another_file.read()
# running the code inside the file
exec(python_code)
exec is a function that runs python strings exactly how python interpreter runs python files.
IN ADDITION
if you want to see the output of the python file:
import subprocess
p = subprocess.Popen(
["python3", file_path],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
err, output = p.communicate()
print(err)
print(output)
EXTRA
for people who are using python2:
execfile(file_path)
exec_file documentation
I want to run multiple os process. The output of this process I route to file and every line in this file has wrong character
e.g.
for module in modules:
module_path = os.path.join(git_dir, module.name)
os.chdir(module_path)
my_env = os.environ.copy()
file_out = open("ouput.txt", "w")
file_err = open("err.txt", "w")
p = Popen(module.run_command, env=my_env, stdout=file_out, stderr=file_err)
will produce string like that in output.txt
...
[0m[[0m[0minfo[0m] [0m[0mLoading global plugins from /home/myuser/.sbt/1.0/plugins[0m
...
instead of
...
[info] Loading global plugins from /home/myuser/.sbt/1.0/plugins
...
I can't manipulate the strings that ship to output text, because of its processing inside subprocess library or somewhere in os level.
Can anyone tell me how can I fix it?
I'm trying to assign the name of the file that is extracted from the zip as a string that I can pass around.
The following is my code so far:
cl = '7z x -ppassowrd Week45_10.zip'
args = shlex.split(cl)
p = subprocess.call(args)
file = open('VENDATA_10', 'r')
VENDATA_10 is the name of the file that is extracted, however, the name of the file extracted from the zip will vary. How do I find the name of the extracted file so that I can open whatever that file is called instead of VENDATA_10?
You could capture the output from the process. 7z will print the filenames that it extracts to the standard output. The output might look like this:
7-Zip 4.44 beta Copyright (c) 1999-2007 Igor Pavlov 2007-01-20
p7zip Version 4.44 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)
Processing archive: Week45_10.zip
Extracting VENDATA_10
Everything is Ok
So you want to capture the lines that start with "Extracting ". To capture the output you'll need to pipe the output from the process into you program and read it. You can used communicate() from subprocess to get the full output.
I've extended you code a bit to do this:
from subprocess import Popen, PIPE
import shlex
cl = '7z x -ppassowrd Week45_10.zip'
args = shlex.split(cl)
p = Popen(args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
(stdoutdata, stderrdata) = p.communicate()
START_WITH = "Extracting "
N = len(START_WITH)
lines = [line.strip() for line in stdoutdata.split('\n')]
files = [line[N:] for line in lines if line.startswith(START_WITH)]
print files
You should be aware that there might arise problems if the files already exist, because then 7z will prompt the user if it should overwrite the files. If you want to handle that then you need to control the input to the 7z process also. You might want to look at the pexpect module for that.
This is the python script will do. The question is how to call the external cmd file within the function?
Read a CSV file in the directory.
If the content in 6th column is equal to 'approved', then calls an
external windows script 'TransferProd.cmd'
.
def readCSV(x):
#csvContents is a list in the global scope that will contain lists of the
#items on each line of the specified CSV file
try:
global csvContents
file = open(csvDir + x + '.csv', 'r') #Opens the CSV file
csvContents = file.read().splitlines() #Appends each line of the CSV file to csvContents
#----------------------------------------------------------------------
#This takes each item in csvContents and splits it at "," into a list.
#The list created replaces the item in csvContents
for y in range(0,len(csvContents)):
csvContents[y] = csvContents[y].lower().split(',')
if csvContents[y][6] == 'approved':
***CALL TransferProd.cmd***
file.close()
return
except Exception as error:
log(logFile, 'An error has occurred in the readCSV function: ' + str(error))
raise
Take a look at the subprocess module.
import subprocess
p = subprocess.Popen(['TransferProd.cmd'])
You can specify where you want output/errors to go (directly to a file or to a file-like object), pipe in input, etc.
import os
os.system('TransferProd.cmd')
This works in both unix/windows flavors as it send the commands to the shell. There are some variations in returned values though! Check here.
If you don't need output of the command you could use: os.system(cmd)
The better solution is to use:
from subprocess import Popen, PIPE
proc = Popen(cmd, shell = True, close_fds = True)
stdout, stderr = proc.communicate()