I am having trouble figuring out what the issue with my code snippet for writing print messages in my console to multiple log-files is doing.
The code snippet I have posted below is supposed to create a new directory test, then write 11 log-files, 1 global log file, and 10 loop log files to this directory. However, the 1st 2 print messages to my global log file is missing when I run this and I cannot figure out what the issue is?
import sys
import os
# Create a test folder to store these global and loop log files.
path = os.getcwd()
test_dir_name = 'test'
test_dir_path = os.path.join(path, test_dir_name)
os.mkdir(test_dir_path)
# Keep a reference to the original stdout.
orig_stdout = sys.stdout
# Define global logfile path.
global_log_name = "global-log.txt"
global_log_path = os.path.join(test_dir_path, global_log_name)
# Problematic code-snippet
sys.stdout = open(global_log_path, 'w')
print("This is a global log file.") # Why is my code omitting this line?
print("The loop is now creating 10 individual log files.") # And this one?
sys.stdout.close()
for i in range(10):
sys.stdout = open(global_log_path, 'w')
print("Creating loop log file {}...".format(i))
sys.stdout.close()
loop_log_name = "local-log-{}.txt".format(i)
loop_log_path = os.path.join(test_dir_path, loop_log_name)
sys.stdout = open(loop_log_path, 'w')
print("This is loop log file {}".format(i))
print("Closing this loop log file...")
sys.stdout.close()
sys.stdout = open(global_log_path, 'w')
print("Loops have concluded.") # But then it includes this line.
print("Now closing global log file.") # And this line in the global log file.
sys.stdout.close()
sys.stdout = orig_stdout
print("Back to original console.")
Some assistance would be greatly appreciated.
The principal issue with this code snippet is the inappropriate use of open(global_log_path, 'w') to append further print messages to global-log.txt. After you have initially executed:
sys.stdout = open(global_log_path, 'w')
print("This is a global log file.") # Why is my code omitting this line?
print("The loop is now creating 10 individual log files.") # And this one?
Subsequent redirections of stdout to global-log.txt instead require passing the argument a, standing for append to open() like so:
sys.stdout = open(global_log_path, 'a')
print("Creating loop log file {}...".format(i))
This prevents previously redirected text from being overwritten, which was happening with your code snippet.
Related
I'd like to launch a mp.Process which can write to a text file. But I'm finding that at the end of the script, the data written to the file isn't actually saved to disk. I don't know what's happening. Here's a minimum working example:
import os, time, multiprocessing
myfile = open("test.dat", "w")
def main():
proc = multiprocessing.Process(target=writer)
proc.start()
time.sleep(1)
print "Times up! Closing file..."
myfile.flush()
os.fsync(myfile.fileno())
print "Closing %s" % (myfile)
myfile.close()
print "File closed. Have a nice day!"
print "> cat test.dat"
def writer():
data = "0000"
for _ in xrange(5):
print "Writing %s to %s" % (data, myfile)
myfile.write(str(data) + '\n')
# if you comment me, writing to disk works!
# myfile.flush()
# os.fsync(myfile.fileno())
if __name__ == "__main__":
main()
Does anyone have suggestions? The context is that this Process will be eventually listening for incoming data, so it really needs to run independently of other things happening in the script.
The problem is that you're opening the file in the main process. Open files are not passed to the subprocesses, so you need to open it inside your function.
Also every code outside the function is executed once for each process, so you're overwriting the file multiple times.
def main():
# create the file empty so it can be appended to
open("test.dat", "w").close()
proc = multiprocessing.Process(target=writer)
proc.start()
def writer():
with open('test.dat', 'a') as myfile: # opens the file for appending
...
myfile.write(...)
...
Now, some OSes don't allow a file to be opened by multiple processes at the same time. The best solution is to use a Queue and pass the data to the main process which then writes to the file.
I just want to make the print output redirect to a file. my code as below:
import sys
# define the log file that receives your log info
log_file = open("message.log", "w")
# redirect print output to log file
sys.stdout = log_file
print ("Now all print info will be written to message.log")
# any command line that you will execute
...
log_file.close()
print ("Now this will be presented on screen")
After execute the script, it comes an error:
[~/Liaohaifeng]$ python3 log.py
Traceback (most recent call last):
File "log.py", line 14, in <module>
print ("Now this will be presented on screen")
ValueError: I/O operation on closed file.
why does this happen? if I update my script as below:
import sys
# make a copy of original stdout route
stdout_backup = sys.stdout
# define the log file that receives your log info
log_file = open("message.log", "w")
# redirect print output to log file
sys.stdout = log_file
print ("Now all print info will be written to message.log"
# any command line that you will execute
...
log_file.close()
# restore the output to initial pattern
sys.stdout = stdout_backup
print ("Now this will be presented on screen")
It will be OK. So, could you please kindly tell me the theory in this issue?
As mentioned in comments, print does not print to a closed filehandle and you have closed sys.stdout, potentially breaking any prints evoked after it closed. Which may happen even without your knowledge, eg somewhere in the imported code. That's why you shouldn't fiddle with sys.* variables (or any variables you didn't create, really) unless you absolutely need to. There is a proper way to redirect print output to a file and it goes like this:
log_file = open('message.log', 'w')
print('Stuff to print in the log', file=log_file)
log_file.close()
Or even safer like this:
with open('message.log', 'w') as log_file:
# Do stuff
print('Stuff to print in the log', file=log_file)
The handle will automatically flush and close when the with block finishes.
I am writing code where i want to redirect output two different files in python
one for log purpose and another for fabric code creation.
Below is the code which redirect output to two different files:
import sys
print('###################################Creating storage commands Variables##########################')
storage_file = open("Storage-output.txt", 'w')
sys.stdout = storage_file
print ('network port show')
print ('Storage Commands are completed')
storage_file.close()
sys.stdout = sys.__stdout__
# write fabric code
fabric_file = open("Fabric_code.py", 'w')
print"from fabric.api import run"
print"def host_type():"
print" run('rows 0', shell=False)"
print" run('networ port show', shell=false)"
fabric_file.close()
sys.stdout = sys.__stdout__
storage_file = open("Storage-output.txt", 'a')
sys.stdout = storage_file
print('###############Genarating aggregate command#############')
print ('storage aggregate create -aggregate aggr_fab -diskcount 6 -raidtype raid_dp", shell=False')
storage_file.close()
sys.stdout = sys.__stdout__
# write fabric code
fabric_file = open("Fabric_code.py", 'a')
print" run('storage aggregate create -aggregate aggr_fab -diskcount 6 - raidtype raid_dp', shell=False) "
fabric_file.close()
sys.stdout = sys.__stdout__
In Above code creating one for log file Storage_file to store this file for records and Fabric_code file to genarate fabric code.
My code generates 1000's of commands I do not want to open and close multiple times for two different files again and again in the python code.
Instead of this is there any solution where i can redirect print output to two direct files without opening and closing
You should refactor your code by opening the files once in the beginning, then writing your files and closing the files in the end. From the above example we can do the following:
import sys
# Open files
storage_file = open("Storage-output.txt", 'w')
fabric_file = open("Fabric_code.py", 'w')
# ===================
# Write Block
# ===================
print('###################################Creating storage commands Variables##########################')
sys.stdout = storage_file
print ('network port show')
print ('Storage Commands are completed')
sys.stdout = sys.__stdout__
# write fabric code
print"from fabric.api import run"
print"def host_type():"
print" run('rows 0', shell=False)"
print" run('networ port show', shell=false)"
sys.stdout = sys.__stdout__
sys.stdout = storage_file
print('###############Genarating aggregate command#############')
print ('storage aggregate create -aggregate aggr_fab -diskcount 6 -raidtype raid_dp", shell=False')
sys.stdout = sys.__stdout__
# write fabric code
print" run('storage aggregate create -aggregate aggr_fab -diskcount 6 - raidtype raid_dp', shell=False) "
sys.stdout = sys.__stdout__
# closing files
storage_file.close()
fabric_file.close()
Suppose I am writing stdout to a file, like this:
sys.stdout = open("file.txt", "w")
# print stuff here
Doing this doesn't work:
sys.stdout.close()
How can I close a file after writing stdout to it?
I took your question to mean: "How can I redirect sys.stdout to a file?"
import sys
# we need this to restore our sys.stdout later on
org_stdout = sys.stdout
# we open a file
f = open("test.txt", "w")
# we redirect standard out to the file
sys.stdout = f
# now everything that would normally go to stdout
# now will be written to "test.txt"
print "Hello world!\n"
# we have no output because our print statement is redirected to "test.txt"!
# now we redirect the original stdout to sys.stdout
# to make our program behave normal again
sys.stdout = org_stdout
# we close the file
f.close()
print "Now this prints to the screen again!"
# output "Now this prints to the screen again!"
# we check our file
with open("test.txt") as f:
print f.read()
# output: Hello World!
Is this an answer to your question?
You can also do this if you want to redirect all print() to a file, which is a fast way and also usefull by my opinion but it could have other effects. If I'm wrong please correct me.
import sys
stdoutold = sys.stdout
sys.stdout = fd = open('/path/to/file.txt','w')
# From here every print will be redirected to the file
sys.stdout = stdoutold
fd.close()
# From here every print will be redirected to console
You can do this:
import sys
class writer(object):
""" Writes to a file """
def __init__(self, file_name):
self.output_file = file_name
def write(self, something):
with open(self.output_file, "a") as f:
f.write(something)
if __name__ == "__main__":
stdout_to_file = writer("out.txt")
sys.stdout = stdout_to_file
print "noel rocks"
The file is only open when you write to it like this.
Looking for some help logging/saving the prints to two file locations as seen below, does anyone know a way to do this?
### Create output file/open it for editing
output_file = open('FILE.txt','w')
output_file1 = open('FILE_APPENDING.txt','a')
## Create a backup of current setting
old_stdout = sys.stdout
sys.stdout = output_file
sys.stdout = output_file1
print "stuff here"
## loop here printing stuff
## Revert python to show prints as normal
sys.stdout=old_stdout
## Close the file we are writing too
output_file.close()
output_file1.close()
Thanks in advance
- Hyflex
You can reassign sys.stdout with some class that writes to multiple files:
class MultiWrite(object):
def __init__(self, *files):
self.files = files
def write(self, text):
for file in self.files:
file.write(text)
def close(self):
for file in self.files:
file.close()
import sys
# no need to save stdout. There's already a copy in sys.__stdout__.
sys.stdout = MultiWrite(open('file-1', 'w'), open('file-2', 'w'))
print("Hello, World!")
sys.stdout.close()
sys.stdout = sys.__stdout__ #reassign old stdout.
Anyway, I agree with Ashwini. It seems that you are searching a hack to obtain something, when you should really use a different approach.
Simply use file.write:
with open('FILE.txt','w') as output_file:
#do something here
output_file.write(somedata) # add '\n' for a new line
with open('FILE_APPENDING.txt','a') as output_file1:
#do something here
output_file1.write(somedata)
help on file.write:
>>> print file.write.__doc__
write(str) -> None. Write string str to file.
Note that due to buffering, flush() or close() may be needed before
the file on disk reflects the data written.