Basically the back story is that i've built a selecting of python scripts for a client to process importing and exporting batch jobs between their operations database and their ecomms site database. this works fine. these scripts write to stdout to update the user on the status of the batch script.
I'm trying to produce a framework for these scripts to be run via a django view and to post the stdout to the webpage to show the user the progress of these batch processes.
the plan was to
- call the batch script as a subprocess and then save stdout and stderr to a file.
- return a redirect to a display page that will reload every 2 seconds and display line by line the contents of the file that stdout is being written to.
however the problem is, that the stdout/stderr file is not being actually written to until the entire batch script has finished running or errors out.
i've tried a number of things, but none seem to work.
heres the current view code.
def long_running(app, filename):
"""where app is ['command', 'arg1', 'arg2'] and filename is the file used for output"""
# where to write the result (something like /tmp/some-unique-id)
fullname = temppath+filename
f = file(fullname, "a+")
# launch the script which outputs something slowly
subprocess.Popen(app, stdout=f, stderr=f)# .communicate()
# once the script is done, close the output
f.close()
def attributeexport(request):
filename = "%d_attribute" %(int(time.time())) #set the filename to be the current time stamp plus an identifier
app = ['python','/home/windsor/django/applications/attribute_exports.py']
#break thread for processing.
threading.Thread(target=long_running, args=(app,filename)).start()
return HttpResponseRedirect('/scripts/dynamic/'+filename+'/')
pass
def dynamic(request, viewfile):
fileobj = open(temppath+viewfile, 'r')
results = []
for line in fileobj:
results.append(line)
if '~END' in line:
#if the process has completed
return render_to_response('scripts/static.html', {'displaylist':results, 'filename':viewfile})
return render_to_response('scripts/dynamic.html', {'displaylist':results, 'filename':viewfile})
pass
It helps if you use the following:
['python','-u','path/to/python/script.py']
Related
I am trying to run an external application using Popen and print the output in the console or separated console (better) and at the same time save the output to the file. There is no user interaction via console, app.bat just sends (writes) the data and should terminate automatically when the execution is finished.
Running the following command will result only in printing the results in the python console.
p = subprocess.Popen("app.bat --x --y", shell=False)
If I add stdout as file I can redirect the output to the file, but nothing is written in the console, which does not give users any feedback (and the feedback needs to be in real-time, not after the execution because app runs approximately 1-3min).
file_ = open("ouput.txt", "w+")
p = subprocess.Popen("app.bat --x --y", shell=False,stdout=file_)
Therefore, my question is how to run the external app and at the same time write in the console and in the file?
For what you want to do I'd encourage you to use the logging module.
A good starter here is https://docs.python.org/2/howto/logging-cookbook.html
It even describes your usecase almost exactly.
If you want to post-process the output of your Popen() call, you should typically redirect stdout to PIPE and then read the output from there. This will allow you to e.g. both write to file and to screen:
import subprocess
logfile ='output.txt'
command = ['app.bat', '--x', '--y']
p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
with open(logfile, 'w+') as f:
for line in p.stdout:
print(line.rstrip())
f.write(line)
Now, this will block until app.bat finishes, which may be exactly what you want. But, if you want your Python script to continue to run, and have app.bat run in te background, you can start a thread that will handle your subprocess stdout:
import subprocess
import threading
logfile ='output.txt'
command = ['app.bat', '--x', '--y']
def writer(p, logfile):
with open(logfile, 'w+') as f:
for line in p.stdout:
print(line.rstrip())
f.write(line)
p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
t = threading.Thread(target=writer, args=(p,logfile))
t.start()
# Other commands while app.bat runs
t.join()
I have written a simple python script that is supposed to run a "sh" file when it is executed.
The problem is, that the script runs but it does not start the ".sh" file. When I execute the ".sh" file manually using "puffy" it does the job, but not when I use my python script. So, what do I have to change in my script in order for it to work?
I will post the methods below so you could get a better idea. I also am using python 3.3.5, Oracle Linux 6.8.
The method that calls the ".sh" file, I have used Popen.
def runPrestage2Stage(self):
import time
time.sleep(60)
reload(Queries)
if "FINISHED" in Queries.statusS2S:
#run batch file
p = Popen(["sh", "/u01/app/Oracle_ODI_11/oracledi/agent/bin/start_prestage2stage_simple.sh"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
print("Prestage2Stage has started")
elif 'ERROR' in Queries.statusS2S:
print("Can not start Prestage Converter, please check runtime status of Stage Converter")
else:
print("Prestage2Stage Converter cannot be started")
Part of main method, that calls the method runPRestage2Stage.
DailyLoadsAutomation.DailyLoads.runPrestage2Stage(self)
load_paraprak_prestage = True
count2 = 0
while load_paraprak_prestage:
reload(Queries)
sleep(300) #waits 300 seconds (5 minutes) and re-checks load status.
if "FINISHED" in Queries.statusPreStage:
load_paraprak_prestage = False
else:
load_paraprak_prestage = True
if count2 == 8:
import sys
sys.exit()
else:
count2 += 1
print("PreStage is finished")
When I run the script,
It will print "Prestage2Stage has started", and "Prestage is finished", as it is supposed to, but It will not run the ".sh" file.
Any idea what is wrong?
usually it is related to user rights or errors in path. You can replace you .sh script with some simple one like "echo "I am running"" and see if it can be accessed and executed. If it is under linux, I hope you are are giving execution rights to you sh script via chmod.
You can run sh file by importing os module like this:
import os
os.system('sh filename.sh')
I wrote a script and when I run it in the shell, it prints the values, output correct(sudo python /home/pi/map/apps/assistant/IFTTT.py):
def GetCalenderMessages():
print("test")
CalenderMessage = bus_service.receive_queue_message('calendar', peek_lock=True)
if CalenderMessage != None:
message = str(CalenderMessage.body)
queuemessage = message.split('|')[1]
print(queuemessage)
sys.stdout.write(queuemessage)
saytts(queuemessage)
CalenderMessage.delete()
I have an interface with an On switch and when I press On this script should be executed, which works, but I'm not getting the print output defined I the script above.
#app_bp.route("/on", methods=["POST"])
#opsoroapp.app_api
def on():
print('test')
cmd = "sudo python /home/pi/OnoSW/apps/assistant/IFTTT.py"
p = subprocess.Popen(cmd, shell=True, stdout=PIPE, stderr=STDOUT)
for line in iter(p.stdout.readline,''):
print line
I tried different things with subprocess, like subprocess.check_output etc but it doesn't give the printed values back.
Thank You
1) Instead of using pipes and Popen, just import the script and call GetCalenderMessages
2) Do you want it just printed to the console or outputted to the webapp user?
In case the second scenario is the case:
in your route function, you don't seem to return anything. Remember how flask routes work, what you return (on success) gets outputted to the user (in the form of a web page) so you'll need to store your result and then return it as a string OR add it to a HTML template and render that template instead.
See more
in case you want it printed to console, trying 1) could fix your problem
I am trying to capture the Behave outputs in a file(let's say a log file). I am dynamically creating a new log file at '#then' step for every run of Behave based on datetime. Below is the sample code given in the steps/xx.py file.
def filecreation(filename):
chwd=os.chdir('C:\\Users\\xxx\\Desktop\\features\\test_features')
with open(filename, 'w+') as d:
pass
cur_ts = datetime.datetime.now()
log_time_stamp = str(cur_ts).split('.')[0].replace(' ',':').replace('-',':').replace(':','')
file_name = 'ATMorFrameRelay' + log_time_stamp + '.log'
filecreation(file_name)
pass
Now i am trying to send the Behave output at every run to the log file created above. I am aware that the command "Behave -o [file name]" will create a file for every run , but thought will rather send the STDOUT to the above created file for every new run. Also is it fine/safer to use the STDOUT to write into files in a production like environment and not cause any issues.
I am a newbie to both Python and Behave, so looking forward for any solution/suggestions on how it can be achieved. Any relevant materials or information will be really appreciated too.
Thanks in advance
Maybe something like this, where cmd is actually behave command to run the tests.
cmd = [
'behave',
'--no-capture',
'--no-capture-stderr',
'--format', 'progress2',
'--logging-level', 'INFO',
'--no-source', '--no-skipped', '--no-summary',
'--tags', 'MACRO'
]
with io.open(filename, 'a') as writer, io.open(filename, 'rb', 1) as reader:
process = subprocess.Popen(cmd, env=env, stdout=writer,stderr=writer)
while process.poll() is None:
sys.stdout.write(reader.read())
sys.stdout.flush()
time.sleep(0.1)
sys.stdout.write(reader.read())
sys.stdout.flush(
I would like to log all the output of a Python script. I tried:
import sys
log = []
class writer(object):
def write(self, data):
log.append(data)
sys.stdout = writer()
sys.stderr = writer()
Now, if I "print 'something' " it gets logged. But if I make for instance some syntax error, say "print 'something# ", it wont get logged - it will go into the console instead.
How do I capture also the errors from Python interpreter?
I saw a possible solution here:
http://www.velocityreviews.com/forums/showpost.php?p=1868822&postcount=3
but the second example logs into /dev/null - this is not what I want. I would like to log it into a list like my example above or StringIO or such...
Also, preferably I don't want to create a subprocess (and read its stdout and stderr in separate thread).
I have a piece of software I wrote for work that captures stderr to a file like so:
import sys
sys.stderr = open('C:\\err.txt', 'w')
so it's definitely possible.
I believe your problem is that you are creating two instances of writer.
Maybe something more like:
import sys
class writer(object):
log = []
def write(self, data):
self.log.append(data)
logger = writer()
sys.stdout = logger
sys.stderr = logger
You can't do anything in Python code that can capture errors during the compilation of that same code. How could it? If the compiler can't finish compiling the code, it won't run the code, so your redirection hasn't even taken effect yet.
That's where your (undesired) subprocess comes in. You can write Python code that redirects the stdout, then invokes the Python interpreter to compile some other piece of code.
I can't think of an easy way. The python process's standard error is living on a lower level than a python file object (C vs. python).
You could wrap the python script in a second python script and use subprocess.Popen. It's also possible you could pull some magic like this in a single script:
import os
import subprocess
import sys
cat = subprocess.Popen("/bin/cat", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
os.close(sys.stderr.fileno())
os.dup2(cat.stdin.fileno(), sys.stderr.fileno())
And then use select.poll() to check cat.stdout regularly to find output.
Yes, that seems to work.
The problem I foresee is that most of the time, something printed to stderr by python indicates it's about to exit. The more usual way to handle this would be via exceptions.
---------Edit
Somehow I missed the os.pipe() function.
import os, sys
r, w = os.pipe()
os.close(sys.stderr.fileno())
os.dup2(w, sys.stderr.fileno())
Then read from r
To route the output and errors from Windows, you can use the following code outside of your Python file:
python a.py 1> a.out 2>&1
Source: https://support.microsoft.com/en-us/help/110930/redirecting-error-messages-from-command-prompt-stderr-stdout
Since python 3.5 you can use contextlib.redirect_stderr
with open('help.txt', 'w') as f:
with redirect_stdout(f):
help(pow)
For such a request, usually it would be much easier to do it in the OS instead of in Python.
For example, if you're going to run "a.py" and record all the messages it will generate into file "a.out", it would just be
python a.py 2>&1 > a.out
The first part 2>&1 redirects stderr to stdout (0: stdin, 1:stdout, 2:stderr), and the second redirects that to a file called a.out.
And as far as I know, this command works in Windows, Linux or MacOS! For other file redirection techniques, just search the os plus "file redirection"
I found this approach to redirecting stderr particularly helpful. Essentially, it is necessary to understand if your output is stdout or stderr. The difference? Stdout is any output posted by a shell command (think an 'ls' list) while sterr is any error output.
It may be that you want to take a shell commands output and redirect it to a log file only if it is normal output. Using ls as an example here, with an all files flag:
# Imports
import sys
import subprocess
# Open file
log = open("output.txt", "w+")
# Declare command
cmd = 'ls -a'
# Run shell command piping to stdout
result = subprocess.run(cmd, stdout=subprocess.PIPE, shell=True)
# Assuming utf-8 encoding
txt = result.stdout.decode('utf-8')
# Write and close file
log.write(txt)
log.close()
If you wanted to make this an error log, you could do the same with stderr. It's exactly the same code as stdout with stderr in its place. This pipes an error messages that get sent to the console to the log. Doing so actually keeps it from flooding your terminal window as well!
Saw this was a post from a while ago, but figured this could save someone some time :)
import sys
import tkinter
# ********************************************
def mklistenconsswitch(*printf: callable) -> callable:
def wrapper(*fcs: callable) -> callable:
def newf(data):
[prf(data) for prf in fcs]
return newf
stdoutw, stderrw = sys.stdout.write, sys.stderr.write
funcs = [(wrapper(sys.stdout.write, *printf), wrapper(sys.stderr.write, *printf)), (stdoutw, stderrw)]
def switch():
sys.stdout.write, sys.stderr.write = dummy = funcs[0]
funcs[0] = funcs[1]
funcs[1] = dummy
return switch
# ********************************************
def datasupplier():
i = 5.5
while i > 0:
yield i
i -= .5
def testloop():
print(supplier.__next__())
svvitch()
root.after(500, testloop)
root = tkinter.Tk()
cons = tkinter.Text(root)
cons.pack(fill='both', expand=True)
supplier = datasupplier()
svvitch = mklistenconsswitch(lambda text: cons.insert('end', text))
testloop()
root.mainloop()
Python will not execute your code if there is an error. But you can import your script in another script an catch exceptions. Example:
Script.py
print 'something#
FinalScript.py
from importlib.machinery import SourceFileLoader
try:
SourceFileLoader("main", "<SCRIPT PATH>").load_module()
except Exception as e:
# Handle the exception here
To add to Ned's answer, it is difficult to capture the errors on the fly during the compilation.
You can write several print statements in your script and you can stdout to a file, it will stop writing to the file when the error occurs. To debug the code you could check the last logged output and check your script after that point.
Something like this:
# Add to the beginning of the script execution(eg: if __name__ == "__main__":).
from datetime import datetime
dt = datetime.now()
script_dir = os.path.dirname(os.path.abspath(__file__)) # gets the path of the script
stdout_file = script_dir+r'\logs\log'+('').join(str(dt.date()).split("-"))+r'.log'
sys.stdout = open(stdout_file, 'w')
This will create a log file and stream the print statements to the file.
Note: Watch out for escape characters in your filepath while concatenating with script_dir in the second line from the last in the code. You might want something similar to raw string. You can check this thread for this.