I am preparing a web service using django in python. When I start web service, I would like to have a main function to run. However, code below did not work in web service. I want to have an initializer function which will fill the structures defined global. I can achieve it by directly defining an init function and call it end of the module but I don't know whether this is the proper way of doing so.
if __name__ == '__main__':
main()
Here similar question was posted. As I understand there is no clear solution for that and I find my way to do this. I defined a dummy function in web service module like:
def warmup(request):
response = HttpResponse()
response.write("ok")
return response
We start application with command sudo python manage.py runserver 0.0.0.0:8081 so I add a piece of code in manage.py such that:
#!/usr/bin/env python
import os
import sys
import requests
import threading
import time
def warmup_request():
print "warmup.."
time.sleep(1)
try:
r = requests.get("http://localhost:8081/warmup/")
if r.content == "ok":
return
except Exception, e:
pass
threading.Timer(0,warmup_request).start()
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
from django.core.management import execute_from_command_line
warmup_request()
execute_from_command_line(sys.argv)
If warmup service API returns ok it means loading initial data process has finished. By defining a global flag, we can block all other service API requests before warming up has finished. Or instead defining global flag we may use request_started signal stated here to block all incoming requests with more little effort.
Related
tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks
There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead
import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!
Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.
Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()
I built a python desktop webapp with a fairly simple flask backend which I'm currently serving simply through Chrome. I'm trying to package it using pywebview, but I ran into an intestesting roadblock.
I have code that looks like this in the main file of my package:
import os
import sys
from flasher import app
import httplib
import webview
import threading
def start_prod_server():
runserver(debug=False, reloader=False)
def start_gui():
webview.create_window("MyAppName", "http://localhost:5000", height=1000)
def runserver(debug=True, reloader=False):
port = int(os.environ.get('PORT', 5000))
url = "http://localhost:{}".format(port)
print("ready!")
app.run(host='::', port=port, debug=debug, use_reloader=reloader)
if __name__ == '__main__':
t = threading.Thread(target=start_prod_server)
t.daemon = True
t.start()
# This never works
threading.Timer(1.5, start_gui).start()
sys.exit()
This fails almost systematically, the window shows up blank and the backend doesn't respond to anything (including requests from a separate desktop browser).
I discovered accidentally that if I start two webview threads, it always works:
if __name__ == '__main__':
t = threading.Thread(target=start_prod_server)
t.daemon = True
t.start()
# This works every time
threading.Timer(1.5, start_gui).start()
threading.Timer(1.5, start_gui).start()
sys.exit()
What could be causing this? My understanding of threading in Python is fairly limited, so I'm not sure where to look. Is this likely to be a bug in pywebview, or am I doing something wrong with my threads?
I'm also open to suggestions of alternatives for the webview part, but I want to keep the python/flask part since the app already works fine as it is.
Well, after a bit of digging, I think I found the solution myself.
For whatever reason, webview.create_window() wasn't navigating to the page and getting stuck. Opening up the second window was causing both windows to navigate to the url and allowing the app to continue. (I still haven't understood why)
The problem is solved simply by adding webview.load_url("http://localhost:5000") like so:
if __name__ == '__main__':
t = threading.Thread(target=start_prod_server)
t.daemon = True
t.start()
threading.Timer(1.5, start_gui).start()
webview.load_url("http://localhost:5000")
sys.exit()
Any insight as to why this happens is still welcome though... In the source for pywebview, the two methods are making the exact same call to self.browser.web_browser.Navigate(url)
I'm automating Minitab 17 using Python's win32com library, and while all of commands execute correctly, I can't seem to get the process started by the Minitab process to exit when my script ends. My structure looks like
from myapi import get_data
import pythoncom
from win32com.client import gencache
def process_data(data):
# In case of threading
pythoncom.CoInitialize()
app = gencache.EnsureDispatch('Mtb.Application')
try:
# do some processing
pass
finally:
# App-specific command that is supposed to close the software
app.Quit()
# Ensure the object is released
del mtb
# In case of threading
pythoncom.CoUninitialize()
def main():
data = get_data()
process_data(data)
if __name__ == '__main__':
main()
I don't get any exceptions raised or error messages printed, the Mtb.exe process is still listed in task manager. Even more frustrating is if I run the following in an IPython session:
>>> from win32com.client import gencache
>>> app = gencache.EnsureDispatch('Mtb.Application')
>>> ^D
The Minitab process is closed immediately. I observe the same behavior in a normal python interactive session. Why would the process get closed correctly when running in an interactive session but not in a standalone script? What is done differently there that isn't being performed in my script?
I've also tried running process_data in a threading.Thread and in a multiprocessing.Process with no luck.
EDIT:
If I have a script containing nothing but
from win32com.client import gencache
app = gencache.EnsureDispatch('Mtb.Application')
then when I run it I see the Mtb.exe process in task manager, but once the script exits the process is killed. So instead my question is why does it matter if this COM object is declared at top-level vs. inside a function?
I don't have minitab so I can't verify but try forcing a shutdown of COM server by setting app = None just after the call to app.Quit? Python uses ref counting to manage object life cycle, so assuming there are no other refs to app then setting it to none should cause it to be finalized immediately. I have seen that cause similar issues. You should not need weak reference, something else is going on. The following, based on your answer, should work:
def process_data(mtb, data):
try:
mtb.do_something(data)
finally:
mtb.Quit()
def main(mtb):
data = get_data()
process_data(mtb, data)
if __name__ == '__main__':
pythoncom.CoInitialize()
mtb = gencache.EnsureDispatch('Mtb.Application')
main(mtb)
mtb.Quit()
mtb = None
pythoncom.CoUninitialize()
The problem was that the the garbage collector could clean up the reference to the underlying IUnknown object (the base type for all COM objects), and without the gc doing it's job the process stayed alive. I solve the problem by using the weakref module to immediately wrap the COM object in a weakref so it could be more easily deferenced:
from myapi import get_data
import weakref
from win32com.client import gencache
import pythoncom
def process_data(mtb_ref, data):
try:
mtb_ref().do_something(data)
finally:
mtb_ref().Quit()
def main(mtb_ref):
data = get_data()
process_data(mtb_ref, data)
if __name__ == '__main__':
pythoncom.CoInitialize()
mtb_ref = weakref.ref(gencache.EnsureDispatch('Mtb.Application'))
main(mtb_ref)
pythoncom.CoUninitialize()
I'm not sure I understand fully why this makes a difference, but I believe it's because there's never a direct reference to the object, only a weak reference, so all the functions that use the COM object only do so indirectly, allowing the GC to know that the object can be collected sooner. For whatever reason it still needs to be created at the top level of the module, but this at least makes it possible for me to write more reusable code that cleanly exits.
after pythoncom.CoUninitialize() i still see process
for me it help (based):
from comtypes.automation import IDispatch
from ctypes import c_void_p, cast, POINTER, byref
def release_reference(self, obj):
logger.debug("release com object")
oleobj = obj._oleobj_
addr = int(repr(oleobj).split()[-1][2:-1], 16)
pointer = POINTER(IDispatch)()
cast(byref(pointer), POINTER(c_void_p))[0] = addr
pointer.Release()
I'm writing python unit tests that test against a REST API that needs to be running as another process.
The REST server is a tomcat application that I call from the shell to run in development mode, so what I am looking to do in the python test is:
Start the server, return when the server is up.
Run unit tests
Send the server Ctrl+D so it shuts down gracefully.
Is there a way to use a single point of entry for python so that the server starts and unit tests run all from one python script call?
I've look around at python subprocess and multithreading in python, but I still don't quite see how to get there from here.
For those that are familiar, this is an Atlassian JIRA plugin we are developing, so the actual shell command is "atlas-run".
Since no one has offered any code to help with this problem, I would do something like the following. Turns out pexpect is very powerful and you don't need the signal module.
import os
import sys
import pexpect
def run_server():
server_dir = '/path/to/server/root'
current_dir = os.path.abspath(os.curdir)
os.chdir(server_dir)
server_call = pexpect.spawn('atlas-run')
server_response = server_call.expect(['Server Error!', 'Sever is running!'])
os.chdir(current_dir)
if server_response:
return server_call #return server spawn object so we can shutdown later
else:
print 'Error starting the server: %s'%server_response.after
sys.exit(1)
def run_unittests():
# several ways to do this. either make a unittest.TestSuite or run command line
# here is the second option
unittest_dir = '/path/to/tests'
pexpect.spawn('python -m unittest discover -s %s -p "*test.py"'%unittest_dir)
test_response = pexpect.expect('Ran [0-9]+ tests in [0-9\.]+s') #catch end
print test_response.before #print output of unittests before ending.
return
def main():
server = run_sever()
run_unittests()
server.sendcontrol('d') #shutdown server
if __name__ == "__main__":
main()
I'm new to Twisted and after finally figuring out how the deferreds work I'm struggling with the tasks. What I want to achieve is to have a script that sends a REST request in a loop, however if at some point it fails I want to stop the loop. Since I'm using callbacks I can't easily catch exceptions and because I don't know how to stop the looping from an errback I'm stuck.
This is the simplified version of my code:
def send_request():
agent = Agent(reactor)
req_result = agent.request('GET', some_rest_link)
req_result.addCallbacks(cp_process_request, cb_process_error)
if __name__ == "__main__":
list_call = task.LoopingCall(send_request)
list_call.start(2)
reactor.run()
To end a task.LoopingCall all you need to do is call the stop on the return object (list_call in your case).
Somehow you need to make that var available to your errback (cb_process_error) either by pushing it into a class that cb_process_error is in, via some other class used as a pseudo-global or by literally using a global, then you simply call list_call.stop() inside the errback.
BTW you said:
Since I'm using callbacks I can't easily catch exceptions
Thats not really true. The point of an errback to to deal with exceptions, thats one of the things that literally causes it to be called! Check out my previous deferred answer and see if it makes errbacks any clearer.
The following is a runnable example (... I'm not saying this is the best way to do it, just that it is a way...)
#!/usr/bin/python
from twisted.internet import task
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.web.client import Agent
from pprint import pprint
class LoopingStuff (object):
def cp_process_request(self, return_obj):
print "In callback"
pprint (return_obj)
def cb_process_error(self, return_obj):
print "In Errorback"
pprint(return_obj)
self.loopstopper()
def send_request(self):
agent = Agent(reactor)
req_result = agent.request('GET', 'http://google.com')
req_result.addCallbacks(self.cp_process_request, self.cb_process_error)
def main():
looping_stuff_holder = LoopingStuff()
list_call = task.LoopingCall(looping_stuff_holder.send_request)
looping_stuff_holder.loopstopper = list_call.stop
list_call.start(2)
reactor.callLater(10, reactor.stop)
reactor.run()
if __name__ == '__main__':
main()
Assuming you can get to google.com this will fetch pages for 10 seconds, if you change the second arg of the agent.request to something like http://127.0.0.1:12999 (assuming that port 12999 will give a connection refused) then you'll see 1 errback printout (which will have also shutdown the loopingcall) and have a 10 second wait until the reactor shuts down.