I am retrieving information about my wireless connection by using
In [4]: subprocess.check_output('iwconfig')
eth0 no wireless extensions.
lo no wireless extensions.
Out[4]: 'wlan0 ...'
I get the string that I want, but I would like to clean the shell from the information about eth0 and lo (notice that these lines are not in the check_ouput return string, they are printed to the shell between ipython's In [4] and Out [4]).
My current guess is that these two lines are considered a warning and are not caught by check_output. Hoping they output to stderr I tried a few variations on
iwconfig > /dev/null 2>&1 # empties my wanted string
iwconfig 2 > /dev/null # throws an error
but without success so far.
How can I prevent the check_output from outputting anything to the shell?
If you want the stderr as your stdout:
subprocess.check_output('iwconfig', stderr=subprocess.STDOUT)
If what you want is just to wipe out your stderr:
import os
subprocess.check_output('iwconfig', stderr=open(os.devnull, 'w'))
Related
I am playing with ssh and run the python scripts with help of these answers - Run local python script on remote server
For connecting the server using ssh, I am currently using the subprocess from python
#ssh command : ssh user#machine python < script.py - arg1 arg2
output = subprocess.Popen(f"ssh user#machine python3 < script.py", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
Now, I get the output of the script as tuples of bytes of string, it is hard to decode the output information. The output having other information like warnings.
I tried decoding the output, but that is not looks great,
Is there any other possible ways to get the output back from ssh, for example, the script.py print a python dictionary or returns a json data, which can get it back in the same format and save in the output variable?
Thanks
There are a couple of ways to do this with subprocess depending on how up to date your Python is.
In general any byte string in Python you can turn into 'proper' text by decoding it. In your example, the command is returning a tuple which is the output of (stdout, stderr) so we need to pick one of those tuple elements with output[0] or output[1]:
>>> output = subprocess.Popen(f"ssh git#github.com", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
>>> output[1].decode('utf-8')
"PTY allocation request failed on channel 0\r\nHi Chris! You've successfully authenticated, but GitHub does not provide shell access.\nConnection to github.com closed.\r\n"
You can tidy that up further by including the universal_newlines=True option.
Assuming you are Python 3.7 or later you can simplify your command so you capture a decoded output straight away:
>>> subprocess.run(['ssh', 'git#github.com'], capture_output=True, text=True).stderr
"PTY allocation request failed on channel 0\nHi Chris! You've successfully authenticated, but GitHub does not provide shell access.\nConnection to github.com closed.\n"
If you are struggling with the escaped newlines (\n) one approach would be to split them so you get an array of lines instead:
>>> output = subprocess.run(['ssh', 'git#github.com'], capture_output=True, text=True, universal_newlines=True).stderr.split("\n")
>>> output
['PTY allocation request failed on channel 0', "Hi Chris! You've successfully authenticated, but GitHub does not provide shell access.", 'Connection to github.com closed.', '']
>>> for line in output:
... print(line)
PTY allocation request failed on channel 0
Hi Chris! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.
print() will happily accept an array of terms to print, normally separated by a space. But you can override the separator:
print(*output, sep="\n")
Are there any differences between the following 2 lines:
subprocess.Popen(command + '> output.txt', shell=True)
subprocess.Popen(command +' &> output.txt', shell=True)
As the popen already triggers the command to run in the background, should I use &? Does use of & ensure that the command runs even if the python script ends executing?
Please let me know the difference between the 2 lines and also suggest which of the 2 is better.
Thanks.
&> specifies that standard error has to be redirected to the same destination that standard output is directed. Which means both the output log of the command and error log will also be written in the output.txt file.
using > alone makes only standard output being copies to the output.txt file and the standard error can be written using command 2> error.txt
I am running a python script using pandas.read_csv to import a csv file. Pandas provides console warnings when it doesn't see what it expects such as:
Skipping line 163: Expected 41 fields in line 163, saw 42
How can I log this to a text file?
If I run the script from a command line, python > logfile.txt only the output of print shows up in the file, not the warnings.
Use stderr
In bash, there are 3 default streams:
STDOUT: Redirected by > or 1>, this is standard output from a program.
STDERR: Redirected by 2>, this is diagnostic output from a program.
STDIN: Input from console, use < to input it.
./prog 2> errorlog.txt >logfile.txt
If you want to redirect ALL output to STDOUT, use:
./prog 2>&1
Here is some more information:
I/O Redirection
This line worked for me
python script.py >>script.out 2>&1
which appends STDOUT (1) to script.out file and redirects STDERR (2) to the same file (&1)
Note that output is appended, so you may want to call > script.out to reset the file beforehand, as in
> script.out; python script.py >> script.out 2>&1
Sources
https://stackoverflow.com/a/876242/8462678
https://stackoverflow.com/a/57272607/8462678 (accepted answer)
import sys
sys.stderr=openfile("file.txt",'w')
df=pandas.read_csv(filepath,on_bad_lines='warn')
sys.stderr.close()
This code will store all the warnings that will be generated by pandas.read_csv
Can anyone please advise what am I doing wrong, that there is no output showed when executing netsh command on windows using python subprocess library?
Example:
p = subprocess.run('netsh dhcp show server', shell=True, stdout=subprocess.PIPE)
print(p.stdout.decode('utf-8'))
Output: empty string
When I execute some other command, etc. echo Hi, I get an output:
p = subprocess.run('echo Hi', shell=True, stdout=subprocess.PIPE)
print(p.stdout.decode('utf-8'))
My intention is to get list of our DHCP servers and parse the output.
Thanks!
Try to change the code page of the process to UTF-8 before executing sub-processes.
chcp 65001
I have a script that uses subprocesses to fetch HTML:
misha#misha-K42Jr:~/git/domain_classifier$ python webkit_retrieve.py error-cut.txt html/error -N 5
http://kurabo.co.jp HostNotFoundError
http://monarch.com HostNotFoundError
http://nssmgmt.com HostNotFoundError
http://sbcglobal.net HostNotFoundError
http://dynamixcorp.com SslHandshakeFailedError
http://groupe-synox.com RemoteHostClosedError
QFont::setPixelSize: Pixel size <= 0 (0)
http://www.cnn.com NoError
http://pacbell.net TimeoutError
If I run the same script, but redirect output to a file, I get nothing in the output:
misha#misha-K42Jr:~/git/domain_classifier$ python webkit_retrieve.py error-cut.txt html/error -N 5 > stdout.txt
QFont::setPixelSize: Pixel size <= 0 (0)
misha#misha-K42Jr:~/git/domain_classifier$ cat stdout.txt
misha#misha-K42Jr:~/git/domain_classifier$
Why is the output empty? Should it not contain the same things that were printed to stdout in the first case?
The question is not about merge stdout and stderr but why redirected stdout produce an empty file
use &> for redirection, this should redirect stdout and stderr to designated file
You have sent stdout to the file, but your program is reporting errors which go to stderr. To setup redirection of stderr, use 2> syntax.
This link might help: http://www.tldp.org/LDP/abs/html/io-redirection.html
I think I was looking for solution a couple of moment ago.
Your best bet is to run something like this:
yourcommand | cat > redirectfile
Your output is going to stderror. Try redirecting both stderror and stdout using the following:
<command> 2>&1 <file>