How to remove b symbol from python3 script?
import subprocess
get_data=subprocess.check_output(["df -k | awk '{print $6}'"],shell=True)
data_arr=get_data.split()
data_arr.pop(0)
data_arr.pop(0)
for i in data_arr[:]:
print(str(i))
Output
b'/dev/shm'
b'/run'
b'/sys/fs/cgroup'
b'/'
b'/tmp'
b'/test'
b'/boot'
b'/home'
b'/var'
b'/mnt/install'
b'/mnt/snapshot'
b'/mnt/share'
b'/mnt/storage'
b'/mnt/linux'
b'/mnt/download'
b'/run/user/1001'
The b symbol indicates that the output of check_process is a bytes rather than a str. The best way to remove it is to convert the output to string before you do any further work on it:
byte_data=subprocess.check_output(["df -k | awk '{print $6}'"],shell=True)
str_data = byte_data.decode('utf-8')
data_arr=str_data.split()
...
The decode method will take care of any unicode you may have in the string. If your default encoding (or the one used by awk I suppose) is not UTF-8, substitute the correct one in the example above.
Possibly an even better way to get around this issue is to tell check_output to open stdout as a text stream. The easiest way is to add a universal_newlines=True argument, which will use the default encoding for your current locale:
str_data = subprocess.check_output(["df -k | awk '{print $6}'"], shell=True, universal_newlines=True)
Alternatively, you can specify an explicit encoding:
str_data = subprocess.check_output(["df -k | awk '{print $6}'"], shell=True, encoding='utf-8')
In both of these cases, you do not need to decode because the output will already be str rather than bytes.
from my SO question:
read_key = ["binary", "arg1", "arg2", "arg3"]
proc = subprocess.Popen(read_key, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='utf-8')
output = proc.communicate()[0]
print(output)
MY_EXPECTED_OUTPUT_STRING
try this:
a=str(yourvalue,'utf-8')
Related
At my python application I'm doing the following to get some values from a larger JSON
ffgrading_json = exec_command(f'{options["path_ffprobe"]} -loglevel quiet -print_format json -read_intervals "%+#2" -select_streams v:0 -show_entries side_data "{path}" | egrep -m 1 -A 10 "Mastering display metadata" | grep -v "Mastering display metadata"|tr -d "[:space:]"| tr -d "\n"')
on the bash console, the raw command returns the following:
"red_x":"34000/50000","red_y":"16000/50000","green_x":"13250/50000","green_y":"34500/50000","blue_x":"7500/50000","blue_y":"3000/50000","white_point_x":"15635/50000","white_point_y":"16450/50000","min_luminance":"50/10000","max_luminance":"40000000/10000"
But if I write the output to a file using python like so:
with open('/tmp/grading.json', 'w') as f:
f.write(str(ffgrading_json))
The following will get written:
b'"red_x":"34000/50000","red_y":"16000/50000","green_x":"13250/50000","green_y":"34500/50000","blue_x":"7500/50000","blue_y":"3000/50000","white_point_x":"15635/50000","white_point_y":"16450/50000","min_luminance":"50/10000","max_luminance":"40000000/10000"'
Not sure how important it is but this is my exec_command function:
def exec_command(string):
"""
Shell command interface
Returns returnCode, stdout, stderr
"""
log('DEBUG', f'[Command] {string}')
output = run(string, shell=True, check=True, capture_output=True)
return output.returncode, output.stdout, output.stderr
The question now is, how can I get rid of b' and where does it come from ???
you need to use .decode() on the output.stdout the output is in bytes so this will convert it to a string
when you print bytes python tries to convert what it can into ascii characters but it will display b'' around the printed data to indicate that the variable you printed contains bytes and it is not a string
I am trying to escape the following, so I can grab the version of iDevice attached via USB:
system_profiler SPUSBDataType | sed -n -e 's/ */ /g' -e '/iPad/,/Version/p' -e '/iPhone/,/Version/p' | grep 'iPad\|iPhone\|Version' | awk 'NR%2{printf $0;next;}1'
So I can run it via Popen, however everytime I always get an issue on iPad\|iPhone\|Version, my code is the following, in an attempt to escape the single quotes:
cmd1 = Popen([r'system_profiler', 'SPUSBDataType'], stdout=subprocess.PIPE)
cmd2 = Popen([r'sed','-n','-e','\'s/ */ /g\'','-e','\'/iPad/,/Version/p\'', '-e', '\'/iPhone/,/Version/p\''], stdin=cmd1.stdout, stdout=subprocess.PIPE)
cmd3 = Popen([r'grep', '\'iPad\|iPhone\|Version\''], stdin=cmd2.stdout, stdout=subprocess.PIPE)
cmd4 = Popen([r'awk', '\'NR%2{printf $0;next;}1\''], stdin=cmd3.stdout, stdout=subprocess.PIPE)
cmd1.stdout.close()
ver = cmd4.communicate()[0]
Use a raw string literal, or double the backslashes; \| has a meaning in a Python string definition syntax too, resulting in no backslash being present in the resulting value. You don't need those quotes either (the shell would have removed them too):
cmd3 = Popen([r'grep', r"iPad\|iPhone\|Version"], stdin=cmd2.stdout, stdout=subprocess.PIPE)
It'd be much easier to apply the string filtering and replacements in Python code, in my opinion.
Played around with grep and managed to extract what I needed from system_profiler. However Martijn's answer is more suitable if you cannot grep for the necessary string.
prof = Popen(['system_profiler', 'SPUSBDataType'], stdout=subprocess.PIPE)
grep1 = Popen(['grep','-e','iPhone','-e','iPad','-e','iPod', '-A', '4'], stdin=prof.stdout, stdout=subprocess.PIPE)
grep2 = Popen(['grep', 'Version'], stdin=grep1.stdout, stdout=subprocess.PIPE)
prof.stdout.close() # Allow ps_process to receive a SIGPIPE if grep_process exits.
stdoutver = grep2.communicate()[0]
I have a text file that has data delimited with '|'
E.g.
123 | 456 | 789
I want to print the second column only.
I can use awk in the shell like this: awk -F'|' '{print $2}' file.txt
However, I want to use python subprocess to do this. And also the input file must be a variable.
Right now, this is what I have.
import subprocess
file = "file-03-10-2016.txt"
with open('another_file.txt', 'wb') as output:
var = subprocess.check_call(['awk', '{print $2}', file])
print var
This prints the second column but it uses space as a delimiter. I want to change the delimiter to '|' using the -F option for awk.
Try:
var = subprocess.check_call(['awk', '-F|', '{print $2}', file])
However, I feel like I should point out that this task is very easy to do in pure python:
def awk_split(file_name, column, fs=None):
with open(file_name, 'r') as file_stream:
for line in file_stream:
yield line.split(fs)[column]
for val in awk_split(file, 1, fs='|'):
# do something...
subprocess.check_call takes a list of strings that are joined with space characters and passed to the shell. So you can just add the -F'|' argument as an item in the list. The only catch, is that the list is using single quotes. If you want to be consistent, you need to escape the single quotes in your argument:
var = subprocess.check_call(['awk', '-F\'|\'', '{print $2}', file])
Alternatively, python also accepts doublequotes as string delimiters:
var = subprocess.check_call(['awk', "-F'|'", '{print $2}', file])
Hope that helps.
I'm trying to count the number of times the string OW appears in a file with the following scrip,
import subprocess
subprocess.call("grewp Ow file.txt | wc -l", shell=True)
but it always returns the correct answer followed by zero
>>> subprocess.call("grep OW production_run.gro | wc -l", shell=True)
2638
0
>>>
and when I try to declare a variable with it, it stores 0.
Does anyone here have any idea of why that's happening and how to fix it?
Python's docs about subprocess.call
Run the command described by args. Wait for command to complete, then
return the returncode attribute.
Try check_output instead
based on document subprocess.check_output :
Run command with arguments and return its output as a byte string.
Thus you can use this:
count = int(subprocess.check_output("grewp Ow file.txt | wc -l", shell=True))
With awk it is very easy to extract a column of data in the bash terminal using.
awk '{print $1}'
I am doing this inside a python script where i use a bash sequence to extract the data i'm interested in
os.system(" qstat | awk '{print $1}' ")
If i call this in a certain context I get a column of numbers. I would like to load all of those numbers into a python list. Can this be done easily?
Use subprocess instead of os.system():
import subprocess
proc = subprocess.Popen('ls | awk "{print $1}"', shell=True, stdout=subprocess.PIPE)
stdout_value = proc.communicate()[0]
for item in stdout_value.split('\n'):
print item
Pipe the output of awk to your Python script.
$ awk '{print $1}' input.txt | python script.py
To read from the pipe in Python use sys.stdin:
import sys
lines = sys.stdin.readlines()