This question already has answers here:
TypeError: can't use a string pattern on a bytes-like object in re.findall()
(3 answers)
Closed 1 year ago.
I have a problem, I need the output of the command in python and exactly get the download speed from the wget command. My code is :
#!/usr/bin/python3
import os
import re
import subprocess
command = "wget ftp://ftp:password#172.17.1.129:1111/test.bin -p -nv "
process = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True)
ftp_result = re.findall(('\d+.\d+'),process)
print (ftp_result)
TypeError: cannot use a string pattern on a bytes-like object
What I'm doing wrong?
On Python 3, subprocess returns bytes unless you explicitly request decoding.
Also, you generally want to avoid shell=True if you are not actually using the shell for anything.
#!/usr/bin/python3
# import os # not used
import re
import subprocess
process = subprocess.check_output(
["wget", "ftp://ftp:password#172.17.1.129:1111/test.bin", "-p", "-nv"],
text=True) # this is the beef
ftp_result = re.findall(('\d+.\d+'),process)
print (ftp_result)
Notice also how check_output already defaults to capturing the output, so you don't need to explicitly direct stdout to a subprocess.PIPE, either.
Perhaps a more robust solution would be to switch to curl, which allows you to separately retrieve just the download speed using a format specifier.
Related
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 5 months ago.
I am trying to use Popen to scp a file from my laptop to a device on my network. The process is pretty straight foward...I can get the file to transfer but I cant get the output from the command to display. I am specificlly looking for the percentage complete. Here is what I have:
from subprocess import Popen, STDOUT, PIPE
scp_command = 'scp -i c:<local_key> <filepath to local file> <user>#<destination_device>:\path'
local_scp_command = Popen(scp_command, text=True, stout=PIPE)
output = local_scp_transfer.communicate
print(output)
I have tried a number of different combinations of stdout and printing the output. I cant even remember all the ways I have tried this. I imagine that there is something kind of easy that I am missing. I am pretty new at programming so even the easy things are compliacted for me.
Thank you so much for all your help!
Use poll() to determine whether or not the process has finished and read a line:
from subprocess import Popen, STDOUT, PIPE
import shlex
scp_command = 'scp -i c:<local_key> <filepath to local file> <user>#<destination_device>:\path'
local_scp_command = Popen(shlex.split(scp_command), text=True, stdout=PIPE)
while local_scp_command.poll() is None and line := local_scp_command.stdout.readline():
print(line)
I added a shlex.split because that's the proper format for Popen.
This question already has answers here:
subprocess wildcard usage
(3 answers)
Passing string to subprocess.run seems to encase it in single quotes negating a wildcard search?
(1 answer)
Closed 1 year ago.
I'm trying to run a bash command with Python3 on Ubuntu. Here is the code (sc.py)
import subprocess
path = '/home/user2/*'
result = subprocess.run(['ls', path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(result.stdout.decode("utf-8"))
print(result.stderr.decode("utf-8"))
Both python3 sc.py and sudo python3 sc.py give
ls: cannot access '/home/u0502210301/*': No such file or directory
However, directly running the command ls /home/user2/* with bash outputs as expected.
All the tests above are run with a sudo account user1.
Could someone give me a hint about this?
The path does not expand globbing * so this is invalid:
path = '/home/user2/*'
Instead do this with expanding the glob into arguments entries to the ls command:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import subprocess
import glob
globPath = '/home/user2/*'
result = subprocess.run(['ls'] + glob.glob(globPath), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(result.stdout.decode("utf-8"))
print(result.stderr.decode("utf-8"))
This has to do with glob expansion. You are expecting * to be handled in python subprocess run as it would be in bash, but that is not the case. To solve the "not found" problem, you can just remove the /* and replace it with /, so:
import subprocess
path = '/home/user2/'
result = subprocess.run(['ls', path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(result.stdout.decode("utf-8"))
print(result.stderr.decode("utf-8"))
this will solve part of the problem, but may not work for your use case (if you want to list all the files/divs in the subdirectories of user2)
Here's a solution which will work for your exact case, I believe:
import subprocess
import glob
path = '/home/user2/*'
cmd = ['ls']
cmd.extend(glob.glob(path))
result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(result.stdout.decode("utf-8"))
print(result.stderr.decode("utf-8"))
I am trying to read a gtf file and then edit it (using subprocess, grep and awk) before loading into pandas.
I have a file name that has header info (indicated by #), so I need to grep that and remove it first. I can do it in python but I want to introduce grep into my pipeline to make processing more efficient.
I tried doing:
import subprocess
from io import StringIO
gtf_file = open('chr2_only.gtf', 'r').read()
gtf_update = subprocess.Popen(["grep '^#' " + StringIO(gtf_file)], shell=True)
and
gtf_update = subprocess.Popen(["grep '^#' " + gtf_file], shell=True)
Both of these codes throw an error, for the 1st attempt it was:
Traceback (most recent call last):
File "/home/everestial007/PycharmProjects/stitcher/pHASE-Stitcher-Markov/markov_final_test/phase_to_vcf.py", line 39, in <module> gtf_update = subprocess.Popen(["grep '^#' " + StringIO(gtf_file)], shell=True)
TypeError: Can't convert '_io.StringIO' object to str implicitly
However, if I specify the filename directly it works:
gtf_update = subprocess.Popen(["grep '^#' chr2_only.gtf"], shell=True)
and the output is:
<subprocess.Popen object at 0x7fc12e5ea588>
#!genome-build v.1.0
#!genome-version JGI8X
#!genome-date 2008-12
#!genome-build-accession GCA_000004255.1
#!genebuild-last-updated 2008-12
Could someone please provide different examples for problem like this, and also explain why am I getting the error and why/how it would be possible to run subprocess directly on files loaded on console/memory?
I also tried using subprocess with call, check_call, check_output, etc., but I've gotten several different error messages, like these:
OSError: [Errno 7] Argument list too long
and
Subprocess in Python: File Name too long
Here is a possible solution that allows you to send a string to grep. Essentially, you declare in the Popen constructor that you want to communicate with the called program via stdin and stdout. You then send the input via communicate and receive the output as return value from communicate.
#!/usr/bin/python
import subprocess
gtf_file = open('chr2_only.gtf', 'r').read()
gtf_update = subprocess.Popen(["grep '^#' "], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# stdout, stderr (but latter is empty)
gtf_filtered, _ = gtf_update.communicate(gtf_file)
print gtf_filtered
Note that it is wise not to use shell=True. Therefore, the Popen line should be written as
gtf_update = subprocess.Popen(["grep", '^#'], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
The rationale is that you don't need the shell to parse the arguments to a single executable. So you avoid unnecessary overhead. It is also better from a security point of view, at least if some argument is potentially unsafe as it comes from a user (think of a filename containing |). (This is obviously not the case here.)
Note that from a performance point of view, I expect that reading the file directly with grep is faster than first reading the file with python, and then sending it to grep.
This question already has answers here:
Formatting a command in python subprocess popen
(2 answers)
Closed 7 years ago.
I have a bunch of different text files, and am trying to sort the texts into one file. I am using python's subprocess, and I wrote the following code
command_line = "sort -m 1.txt 2.txt > a.txt"
args = shlex.split(command_line)
subprocess.call(args)
and the subprocess.call(args) returned 2 as a result, and nothing was written in a.txt. Anything wrong with my code?
If you want to use the shell redirection operator > in your command line, you have to pass shell=True to subprocess.call. Otherwise, '>' and 'a.txt' are passed as command line arguments to sort. With shell=True, the command line is passed to and interpreted by an actual shell, and you should therefore not shlex.split it. It may be easier to use os.system instead of subprocess.call, which uses a shell by default.
This question already has answers here:
How to just call a command and not get its output [duplicate]
(4 answers)
Closed 5 years ago.
For the following command:
subprocess.call(shlex.split(
"""/usr/local/itms/bin/iTMSTransporter -m lookupMetadata
-apple_id %s -destination %s"""%(self.apple_id, self.destination))
It prints the entire output into the Terminal window. How would I suppress ALL output here? I tried doing subprocess.call(shlex.split(<command> > /dev/null 2&1)), but it didn't produce the required results. How would I do this here?
You can use the stdout= and stderr= parameters to subprocess.call() to direct stdout or stderr to a file descriptor of your choice. So maybe something like this:
import os
devnull = open(os.devnull, 'w')
subprocess.call(shlex.split(
'/usr/local/itms/bin/iTMSTransporter -m lookupMetadata '
'-apple_id %s -destination %s' % (self,apple_id, self.destination)),
stdout=devnull, stderr=devnull)
Using subprocess.PIPE, if you're not reading from the pipe, could cause your program to block if it generates a lot of output.
Update
As #yanlend mentions in a comment, newer (3.x) versions of Python include subprocess.DEVNULL to solve this problem in a more convenient and portable fashion. In that case, the code would look like:
subprocess.call(shlex.split(
'/usr/local/itms/bin/iTMSTransporter -m lookupMetadata '
'-apple_id %s -destination %s' % (self,apple_id, self.destination)),
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
What worked for me is appending 2>/dev/null at the end of the command.