I am trying to save all output (stdout and all errors) of a cell to a file. To save stdout, I am using the following:
import sys
old_stdout = sys.stdout
sys.stdout = open('test.txt', 'w')
print("Hello World! ")
In this case, the output is not displayed and gets saved in the file as expected. To save errors, I used:
#Doesn't work
sys.stderr = open('error.txt','w')
print(a) #Should raise NameError
When I run this cell, I get the error in the notebook, and not in the file as expected:
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-5-de3efd936845> in <module>()
1 #Doesn't work
----> 2 sys.stderr = open('error.txt','w')
3 print("Test")
4 print(a)
NameError: name 'sys' is not defined
I would like this saved in a file and not shown in the notebook. What is the correct code for this?
I think that the problem here is that IPython kernels spawned for the notebook use a ZMQInteractiveShell instance, which catches the errors before they make it to stderr, in order to send the error information to the various potential frontends (consoles, jupyter notebooks, etc). See ipykernel/ipkernel.py#L397-L413 for the code which catches exceptions, then InteactiveShell._showtraceback for the base implementation (print to sys.stderr), and ZMQInteractiveShell._showtraceback for that used by notebook kernels (send stderr-channel messages over zmq to frontends).
If you're not bothered about getting exact stderr output, you could take advantage of IPython's existing error logging, which logs errors to a StreamHandler with the prefix "Exception in execute request:". To use this, set the ipython log level, and alter the provided handler's stream:
import logging
import sys
my_stderr = sys.stderr = open('errors.txt', 'w') # redirect stderr to file
get_ipython().log.handlers[0].stream = my_stderr # log errors to new stderr
get_ipython().log.setLevel(logging.INFO) # errors are logged at info level
Alternatively, to get your shell errors to print directly to a file without alteration, you can monkey-patch the _showtraceback method to print traceback to file as well as zmq message queues:
import sys
import types
# ensure we don't do this patch twice
if not hasattr(get_ipython(), '_showtraceback_orig'):
my_stderr = sys.stderr = open('errors.txt', 'w') # redirect stderr to file
# monkeypatch!
get_ipython()._showtraceback_orig = get_ipython()._showtraceback
def _showtraceback(self, etype, evalue, stb):
my_stderr.write(self.InteractiveTB.stb2text(stb) + '\n')
my_stderr.flush() # make sure we write *now*
self._showtraceback_orig(etype, evalue, stb)
get_ipython()._showtraceback = types.MethodType(_showtraceback, get_ipython())
Related
Roughly speaking, I want to port this to pure Python:
#!/bin/bash
{
python test.py
} &> /tmp/test.log
This didn't work for some unknown reasons:
import os.path, sys
import tempfile
with open(os.path.join(tempfile.gettempdir(), "test.log"), "a") as fp:
sys.stdout = sys.stderr = fp
raise Exception("I'm dying")
The resulting test.log was empty (and I didn't see anything on my console,) when I tested it with Python 2.6.6, Python 2.7.8 and Python 3.4.2 on CentOS x86_64.
But Ideally I'd like a solution for Python 2.6.
(For now, it's tolerable to clutter the log file with intermixed output from stdout and stderr or multithreading, as long as any data won't simply disappear into a blackhole.)
Show me a concise and portable solution which is confirmed to work with an exception stack trace on sys.stderr. (Preferably something other than os.dup2)
Remember that file objects are closed after with blocks :)
Use simply this:
sys.stdout = sys.stderr = open("test.log","w")
raise Exception("Dead")
Content of test.log after exit:
Traceback (most recent call last):
File "test.py", line 5, in <module>
raise Exception("Dead")
Exception: Dead
You can use a method like this one:
import traceback
import sys
from contextlib import contextmanager
#contextmanager
def output_to_file(filepath, write_mode='w'):
stdout_orig = None
stderr_orig = None
stdout_orig = sys.stdout
stderr_orig = sys.stderr
f = open(filepath, write_mode)
sys.stdout = f
sys.stderr = f
try:
yield
except:
info = sys.exc_info()
f.write('\n'.join(traceback.format_exception(*info)))
f.close()
sys.stdout = stdout_orig
sys.stderr = stderr_orig
And the the usage is:
with output_to_file('test.log'):
print('hello')
raise Exception('I am dying')
And the cat test.log produces:
hello
Traceback (most recent call last):
File "<ipython-input-3-a3b702c7b741>", line 20, in outputi_to_file
yield
File "<ipython-input-4-f879d82580b2>", line 3, in <module>
raise Exception('I am dying')
Exception: I am dying
This works for me:
#!/usr/bin/env python
from __future__ import print_function
import os, os.path, sys, tempfile
old_out = os.dup(sys.stdout.fileno())
old_err = os.dup(sys.stderr.fileno())
with open(os.path.join(tempfile.gettempdir(), "test.log"), "a") as fp:
fd = fp.fileno()
os.dup2(fd, sys.stdout.fileno())
os.dup2(fd, sys.stderr.fileno())
print("Testing")
print('testing errs', file=sys.stderr)
raise Exception("I'm dying")
The future is just for cleaner handling of Python2 or Python3 with the same example. (I've also changed the raise statement to instantiate an exception, strings as exceptions have been deprecated for a long time and they're not properly supported under Python3).
The old_* values are just if we wanted to restore our original stdout and/or stderr after using our redirected file.
How to know which file is calling which file in filesystem, like file1.exe is calling file2.exe
so file2.exe is modified,
and file1.exe is entered in log file.
winos
I have searched INTERNET but not able to find any samples.
In order know which file is calling which file you can use the Trace module
exp: if you have 2 files
***file1.py***
import file2
def call1():
file2.call2()
***file2.py***
def call2():
print "---------"
u can use it using console:
$ python -m trace --trackcalls path/to/file1.py
or within a program using a Trace object
****tracefile.py***
import trace,sys
from file1 import call1
#specify what to trace here
tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix], trace=0, count=1)
tracer.runfunc(call1) #call the function call1 in fille1
results = tracer.results()
results.write_results(summary=True, coverdir='.')
I have been stuck on this for the past hour. I had plugged logging into my tkinter gui, but could not get it to work. I then started removing parts until I got at the bare bones example in the very python docs and it will not work. At this point I have nothing else to remove.
The code is as follows:
import logging
LOG_FILENAME = r'logging_example.out'
logging.basicConfig(filename=LOG_FILENAME ,level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
f = open(LOG_FILENAME, 'rt')
try:
body = f.read()
finally:
f.close()
print('FILE:')
print (body)
The warning is printed to stdout, but the file is not generated.
I am runing python 3.4, x64 on a windows 7. It is a anacondas distribution, so this is running in Ipython inside spyder.
I guess this should be working
As Jonas Byström noted, this does work outside Ipython. It seems that Ipython configures a logging handler before I get the chance to do so. Also, basicConfig will do nothing if a handler is already present. So, in order to have it working in Ipython, one must do one of three things: 1) Add a new handler, OR 2)reload logging, OR 3) remove existing handlers. I did number 2 bellow.
import logging
from imp import reload
reload(logging)
LOG_FILENAME = r'logging_example.out'
logging.basicConfig(filename=LOG_FILENAME ,level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
f = open(LOG_FILENAME, 'rt')
try:
body = f.read()
finally:
f.close()
print('FILE:')
print (body)
See theese for more information:
Logging in ipython;
More on the same
I want to create an empty pcap file. I'm using wrpcap module of 'Scapy'. But wrpcap takes exactly 2 arguments: one is the file name and other is the packet list like:
wrpcap("my.pcap",my_pkt_list)
Since I want to make it, empty and I don't have a packet list, I'm writing an empty string to the pcap file. Which is creating the file but also giving a warning as well as errors since a string doesn't match to a packet type.
WARNING: PcapWriter: unknown LL type for str. Using type 1 (Ethernet)
Traceback (most recent call last):
File "test.py", line 35, in <module>
wrpcap("pcap/FU.pcap","")
File "/usr/lib/python2.7/site-packages/scapy/utils.py", line 466, in wrpcap
PcapWriter(filename, *args, **kargs).write(pkt)
File "/usr/lib/python2.7/site-packages/scapy/utils.py", line 646, in write
self._write_packet(pkt)
File "/usr/lib/python2.7/site-packages/scapy/utils.py", line 688, in _write_packet
sec = int(packet.time)
AttributeError: 'str' object has no attribute 'time'
For now, I'm able to suppress the errors with try and except but unable to suppress the warning.
Code
from scapy.all import *
try:
wrpcap("my.pcap","")
except:
pass
and the warning is still there:
WARNING: PcapWriter: unknown LL type for str. Using type 1 (Ethernet)
How to suppress it from inside the python code?
Python has a built in method in the standard library for supressing warnings:
import warnings
warnings.simplefilter("ignore", Warning)
You can read more about the warnings library in the docs.
EDIT
It doesn't look like scapy uses the warnings library to generate this. Instead it logs them to a logger called scapy.runtime with a level of warning. There's obviously some default logging config to echo that to stdout. You could add your own logging handler for scapy.runtime to supress them.
You can avoid that warning by using PcapWriter method of scapy.
from scapy.all import *
try:
writer=PcapWriter("my.pcap")
except:
pass
This creates your empty pcap file. When you want to write some packets to it, just use the following code:
writer.write(<Your_packets>)
writer.flush()
You can suppress this warning by disabling warning logging before the call wrpcap:
import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
The text "WARNING: ..." is not part of the exception and will thus not be surpressed by your try clause.
An idea I got was to redirect stdout during the call
from scapy.all import *
try:
old_stdout = sys.stdout
sys.stdout = None
wrpcap("my.pcap","")
sys.stdout = old_stdout
except:
pass
that should surpress any and all output during the call
Someone inadvertently moved the open log file used by a python program.
The program uses the logging module with a TimedRotatingFileHandler. When the time came to roll-over the file this error was output:
Traceback (most recent call last):
File "/python_root/lib/logging/handlers.py", line 78, in emit
self.doRollover()
File "/python_root/lib/logging/handlers.py", line 338, in doRollover
os.rename(self.baseFilename, dfn)
OSError: [Errno 2] no such file or directory
Logged from file logtest.py, line 16
The error was repeated on each subsequent attempt to log something. The logged messages did not go into the old (moved) log file.
This reproduces the problem (if you move the log file :))
import time
import logging
from logging import handlers
f = logging.Formatter( "%(asctime)s %(message)s" )
h = handlers.TimedRotatingFileHandler(
"testlog", when='s', interval=5, backupCount=10 )
h.setFormatter( f )
logger = logging.getLogger( 'test' )
logger.setLevel( logging.INFO )
logger.addHandler( h )
logger.info( "LOGTEST started" )
for i in range( 10 ):
time.sleep( 5 )
logger.info( "Test logging " + str( i ) )
My concern here is that subsequent log messages are lost. What I'd like to achieve is, in ascending order of preference:
An exception that exits.
An exception I can catch and handle.
The logger displays the error, but subsequent messages go to the old log file.
The logger displays this error, but opens a new log file and continues as normal.
I've skimmed the docs/cookbook for relevant hooks, but nothing's popped out at me. Pointers there are equally welcome.
Thanks for your help,
Jonathan
Exceptions that are raised in doRollover are passed to the handleError method of the handler. You can define a subclass and override this method to do whatever it is you want to do to handle the error.