I am running a python script using pandas.read_csv to import a csv file. Pandas provides console warnings when it doesn't see what it expects such as:
Skipping line 163: Expected 41 fields in line 163, saw 42
How can I log this to a text file?
If I run the script from a command line, python > logfile.txt only the output of print shows up in the file, not the warnings.
Use stderr
In bash, there are 3 default streams:
STDOUT: Redirected by > or 1>, this is standard output from a program.
STDERR: Redirected by 2>, this is diagnostic output from a program.
STDIN: Input from console, use < to input it.
./prog 2> errorlog.txt >logfile.txt
If you want to redirect ALL output to STDOUT, use:
./prog 2>&1
Here is some more information:
I/O Redirection
This line worked for me
python script.py >>script.out 2>&1
which appends STDOUT (1) to script.out file and redirects STDERR (2) to the same file (&1)
Note that output is appended, so you may want to call > script.out to reset the file beforehand, as in
> script.out; python script.py >> script.out 2>&1
Sources
https://stackoverflow.com/a/876242/8462678
https://stackoverflow.com/a/57272607/8462678 (accepted answer)
import sys
sys.stderr=openfile("file.txt",'w')
df=pandas.read_csv(filepath,on_bad_lines='warn')
sys.stderr.close()
This code will store all the warnings that will be generated by pandas.read_csv
Related
I would like to save the input code and the output result into a file. For example the following python code code.py:
print(2+2)
print(3+2)
to create a code-and-output.txt:
>>> print(2+2)
4
>>> print(3+2)
5
But I can not get it working. Basically, I want to code-and-output.txt to capture what would happen if I run interpreted python and run statements in python interactive environment (code + output).
Ways that I have tried so far:
Redirect stdout:
python code.py > code-and-output.txt
It only saves the output.
Redirect stdout and stdin:
python < code.py > code-and-output.txt
It does the same (only output).
nohup
nohup python code.py
The same problem: only output.
Script
script -q code-and-output.txt
python
print(2+2)
print(2+3)
ctr+d
ctr+d
It works but I need to do it manually. Moreover, it saves some garbage that I can not make them quiet with -q.
Bash Script
# bash-file.sh
python &
print(2+2)
print(2+3)
Does not work: commands run in console bash, not python. It does not work with & either: never ends python repl.
Using tty
open another terminal like /dev/pts/2 and send above bash-file.sh
cat bash-file.sh > /dev/pts/2
It just copies but does not run.
I am not interested in solutions like Jupyter and iPython. They have their own problems that does not address my requirement.
Any solution through linux commands (preferably) or python? Thank you.
Save this is as repl_sim.py in the same directory as your code.py:
with open("code.py", 'r') as input_file:
for line in input_file:
print(f">>> {line.strip()}")
eval(line)
Then run in your terminal with the following if you want to redirect the output to a text file named code-and-output.txt:
python repl_sim.py > code-and-output.txt
-OR-
Then run in your terminal with the following if you want to see the output as well as the make the text file matching:
python repl_sim.py | tee code-and-output.txt
It at least works for the example you provided as code.py.
Pure Python version of first option above so that you don't need shell redirect.
Save this code as repl_sim.py:
import contextlib
with open('code-and-output.txt', 'w') as f:
with contextlib.redirect_stdout(f):
with open("code.py", 'r') as input_file:
for line in input_file:
print(f">>> {line.strip()}")
eval(line)
Then run in your terminal with:
python repl_sim.py
That will result in code-and-output.txt with your desired content.
Contextlib use based on Raymond Hettinger's August 17th 2018 Tweet and contextlib.redirect_stdout() documentation
.
Are there any differences between the following 2 lines:
subprocess.Popen(command + '> output.txt', shell=True)
subprocess.Popen(command +' &> output.txt', shell=True)
As the popen already triggers the command to run in the background, should I use &? Does use of & ensure that the command runs even if the python script ends executing?
Please let me know the difference between the 2 lines and also suggest which of the 2 is better.
Thanks.
&> specifies that standard error has to be redirected to the same destination that standard output is directed. Which means both the output log of the command and error log will also be written in the output.txt file.
using > alone makes only standard output being copies to the output.txt file and the standard error can be written using command 2> error.txt
I am retrieving information about my wireless connection by using
In [4]: subprocess.check_output('iwconfig')
eth0 no wireless extensions.
lo no wireless extensions.
Out[4]: 'wlan0 ...'
I get the string that I want, but I would like to clean the shell from the information about eth0 and lo (notice that these lines are not in the check_ouput return string, they are printed to the shell between ipython's In [4] and Out [4]).
My current guess is that these two lines are considered a warning and are not caught by check_output. Hoping they output to stderr I tried a few variations on
iwconfig > /dev/null 2>&1 # empties my wanted string
iwconfig 2 > /dev/null # throws an error
but without success so far.
How can I prevent the check_output from outputting anything to the shell?
If you want the stderr as your stdout:
subprocess.check_output('iwconfig', stderr=subprocess.STDOUT)
If what you want is just to wipe out your stderr:
import os
subprocess.check_output('iwconfig', stderr=open(os.devnull, 'w'))
I'm using a Python script to call a batch file that call after the generation tools and compiler. I save this to a output file using:
os.system("run.bat >output.txt")
The problem is that warnings and compiler error are not saved in this output file - they are only displayed on the Python console.
How would I go about saving the output to a file?
You can do several things:
Use the subprocess family to get handles to the stdout and stderr files
Use a redirect of the stderr in your os.system call.
The first one is more pythonic, and allows your python script to be aware of the errors.
A simple way to redirect stderr to the same file is like this:
os.system("run.bat > output.txt 2>&1")
Don't use os.system ~ you have no error handling there, or even worse, you have no indicator if an error happened.
I would solve this problem like this:
from subprocess import Popen, PIPE
proc = Popen(['run.bat'], stdout=PIPE, stderr=PIPE)
result = proc.communicate()
with open('output.txt', 'w') as output:
output.write(result[0])
with open('errors.txt', 'w') as errors:
errors.write(result[-1])
I have a script that uses subprocesses to fetch HTML:
misha#misha-K42Jr:~/git/domain_classifier$ python webkit_retrieve.py error-cut.txt html/error -N 5
http://kurabo.co.jp HostNotFoundError
http://monarch.com HostNotFoundError
http://nssmgmt.com HostNotFoundError
http://sbcglobal.net HostNotFoundError
http://dynamixcorp.com SslHandshakeFailedError
http://groupe-synox.com RemoteHostClosedError
QFont::setPixelSize: Pixel size <= 0 (0)
http://www.cnn.com NoError
http://pacbell.net TimeoutError
If I run the same script, but redirect output to a file, I get nothing in the output:
misha#misha-K42Jr:~/git/domain_classifier$ python webkit_retrieve.py error-cut.txt html/error -N 5 > stdout.txt
QFont::setPixelSize: Pixel size <= 0 (0)
misha#misha-K42Jr:~/git/domain_classifier$ cat stdout.txt
misha#misha-K42Jr:~/git/domain_classifier$
Why is the output empty? Should it not contain the same things that were printed to stdout in the first case?
The question is not about merge stdout and stderr but why redirected stdout produce an empty file
use &> for redirection, this should redirect stdout and stderr to designated file
You have sent stdout to the file, but your program is reporting errors which go to stderr. To setup redirection of stderr, use 2> syntax.
This link might help: http://www.tldp.org/LDP/abs/html/io-redirection.html
I think I was looking for solution a couple of moment ago.
Your best bet is to run something like this:
yourcommand | cat > redirectfile
Your output is going to stderror. Try redirecting both stderror and stdout using the following:
<command> 2>&1 <file>