We Keep Coding

Run command in CMD via python and extract the data

I am trying to use the below code to run a command and extract the data from the cmd.
the file with the commands and data is a txt file. (let me know if I should change it or use an excel if better).
the commands look something like this: ping "host name" which would result in some data in the cmd.there is list of these in the file. so it would ping "hostname1" then line two ping "hostname2"..etc
THE QUESTION: I want it to run every line individually and extract the results from the cmd and store them in a txt file or excel file - Ideally I want all the results in the same file. is this possible? and how?
here is the code so far:
root_dir = pathlib.Path(r"path to file here")
cmds_file = root_dir.joinpath('actual file here with commands and data')
#fail = []
cmds = cmds_file.read_text().splitlines()
try:
for cmd in cmds:
args = cmd.split()
print(f"\nRunning: {args[0]}")
output = subprocess.check_output(args)
print(output.decode("utf-8"))
out_file = root_dir.joinpath(f"Name of file where I want results printed in")
out_file.write_text(output.decode("utf-8"))
except:
pass

You can use a module called subprocess import subprocess
Then you can define a variable like this
run = subprocess.run(command_to_execute, capture_output=True)
After that you can do print(run.stdout) to print the command output.
If you want to write it to a file you can do this after you run the above code
with open("PATH TO YOUR FILE", "w") as file:
file.write(run.stdout)
This should write a file which contains the output of your command
After that close the file using file.close() and reopen it but in "a" mode
with open("PATH TO YOUR FILE", "a") as file:
file.write(\n + run.stdout)
This should append data to your file.
Remember to close the file just for best practice, I have some bad memorys about not closing the file after I opened it :D

My plan is simple:
Open input, output file
Read input file line by line
Execute the command and direct the output to the output file
#!/usr/bin/env python3
import pathlib
import shlex
import subprocess
cmds_file = pathlib.Path(__file__).with_name("cmds.txt")
output_file = pathlib.Path(__file__).with_name("out.txt")
with open(cmds_file, encoding="utf-8") as commands, open(output_file, "w", encoding="utf-8") as output:
for command in commands:
command = shlex.split(command)
output.write(f"\n# {shlex.join(command)}\n")
output.flush()
subprocess.run(command, stdout=output, stderr=subprocess.STDOUT, encoding="utf-8")
Notes
Use shlex.split() to simulate the bash shell's command split
The line output.write(...) is optional. You can remove it
With subprocess.run(...), the stdout=output will redirect the command's output to the file. You don't have to do anything.
Update
I updated the subprocess.run line to redirect stderr to stdout, so error will show.

How do you use a python variable in popen()?

Im trying to record docker stats for every file in the mydata directory. For example if one of the files is names piano.txt I would like the output file to be piano_stuff.txt. This is what I have so far:
import subprocess
import signal
import os
for file_name in os.listdir('mydata'):
data_txt = "./" + file_name.split(".")[0] + "_stuff.txt"
dockerStats = subprocess.Popen("docker stats --format {{.MemUsage}} >> ${data_txt}", shell=True)
os.killpg(os.getpgid(dockerStats.pid), signal.SIGTERM)

Don't use shell=True. Open the file locally, and pass the file object as the stdout argument. You can also use the --no-stream option to have the command exit after producing one line of output, rather than asynchronously trying to kill the process as soon as possible. (You might get multiple lines of output, or you might get none, depending on when the OS schedules the Docker process to run.)
with open(data_txt, "a") as f:
subprocess.run(["docker", "stats", "--format", "{{.MemUsage}}", "--no-stream"], stdout=f)

Fix encoding when running process in background using subprocess

I want to run multiple os process. The output of this process I route to file and every line in this file has wrong character
e.g.
for module in modules:
module_path = os.path.join(git_dir, module.name)
os.chdir(module_path)
my_env = os.environ.copy()
file_out = open("ouput.txt", "w")
file_err = open("err.txt", "w")
p = Popen(module.run_command, env=my_env, stdout=file_out, stderr=file_err)
will produce string like that in output.txt
...
[0m[[0m[0minfo[0m] [0m[0mLoading global plugins from /home/myuser/.sbt/1.0/plugins[0m
...
instead of
...
[info] Loading global plugins from /home/myuser/.sbt/1.0/plugins
...
I can't manipulate the strings that ship to output text, because of its processing inside subprocess library or somewhere in os level.
Can anyone tell me how can I fix it?

Python checking integrity of gzip archive

Is there a way in Python using gzip or other module to check the integrity of the gzip archive?
Basically is there equivalent in Python to what the following does:
gunzip -t my_archive.gz

Oops, first answer (now deleted) was result of misreading the question.
I'd suggest using the gzip module to read the file and just throw away what you read. You have to decode the entire file in order to check its integrity in any case. https://docs.python.org/2/library/gzip.html
Something like ( Untested code)
import gzip
chunksize=10000000 # 10 Mbytes
ok = True
with gzip.open('file.txt.gz', 'rb') as f:
try:
while f.read(chunksize) != b'':
pass
except:
ok = False
I don't know what exception reading a corrupt zipfile will throw, you might want to find out and then catch only this particular one.

you can use subprocess or os module to execute this command and read the output. something like this
Using os module
import os
output = os.popen('gunzip -t my_archive.gz').read()
Using Subprocess Module
import subprocess
proc = subprocess.Popen(["gunzip", "-t", "my_archive.gz"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()

How to provide input to a python subprocess call that expects filename, rather than variable?

I'm trying to call a shell script (segment.sh) within python.
The syntax that produce correct results at the console is:
> ./segment.sh ctb file.txt utf-8 0
As can be seen, this shell script expects an text file as input. What I am trying to do is to open up the file and read its content in python (later preferably from an HTML POST form), and somehow pass the variable containing the content to the python subprocess
call.
The following function is not working. But if I just provide a file name such as:
Popen(["/bin/bash", "./segment.sh", "ctb", "file.txt", "utf-8", "0"])`
then, it will work, but I want to pass the input from a variable. Would you please give me some pointers?
def pySegment(text):
op = subprocess.Popen(["/bin/bash", "./segment.sh", "ctb", "utf-8", "0"],
stdout = subprocess.PIPE,
stdin = subprocess.PIPE,
stderr = subprocess.STDOUT,
)
results = op.communicate(input=text)[0]
return results
if __name__ == "__main__":
filename = "./file.txt"
text = open(filename).read()
result = pySegment(text)
print result

I suggest to use a named pipe:
import os, tempfile, shutil, subprocess
temp_dir = tempfile.mkdtemp()
filename = os.path.join(temp_dir, 'file.txt')
text = '<text>'
os.mkfifo(filename)
try:
subprocess.Popen(('segment.sh', 'ctf', filename, 'utf-8', '0'))
with open(filename, 'w') as f:
f.write(text)
finally:
shutil.rmtree(temp_dir)
The named pipe will offer the same interface of a file without really creating that file as you need.