tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks
There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead
import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!
Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.
Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()
I'm learning python from Learn Python the Hard Way, and am currently on excercise 50 (https://learnpythonthehardway.org/book/ex50.html). When I run the script in PowerShell on Windows, it runs indefinitely and doesn't produce the predicted result (printing "Hello World" in the web browser). I'm using Python 2.7 The command line looks like this:
$ python bin/app.py
http://0.0.0.0:8080/
and the script doesn't terminate.
I'm running this script from the exercise:
import web
urls = (
'/', 'Index'
)
app = web.application(urls, globals())
render = web.template.render('templates/')
class Index(object):
def GET(self):
greeting = "Hello World"
return render.index(greeting = greeting)
if __name__ == "__main__":
app.run()
It's not supposed to terminate. It's running a web server; the exercise wants you to go to the address printed - http://0.0.0.0:8080/ - in your web browser, where you will see the message.
If you want to terminate the local server press ctr+c or ctrl+z in the powershell
I'm writing python unit tests that test against a REST API that needs to be running as another process.
The REST server is a tomcat application that I call from the shell to run in development mode, so what I am looking to do in the python test is:
Start the server, return when the server is up.
Run unit tests
Send the server Ctrl+D so it shuts down gracefully.
Is there a way to use a single point of entry for python so that the server starts and unit tests run all from one python script call?
I've look around at python subprocess and multithreading in python, but I still don't quite see how to get there from here.
For those that are familiar, this is an Atlassian JIRA plugin we are developing, so the actual shell command is "atlas-run".
Since no one has offered any code to help with this problem, I would do something like the following. Turns out pexpect is very powerful and you don't need the signal module.
import os
import sys
import pexpect
def run_server():
server_dir = '/path/to/server/root'
current_dir = os.path.abspath(os.curdir)
os.chdir(server_dir)
server_call = pexpect.spawn('atlas-run')
server_response = server_call.expect(['Server Error!', 'Sever is running!'])
os.chdir(current_dir)
if server_response:
return server_call #return server spawn object so we can shutdown later
else:
print 'Error starting the server: %s'%server_response.after
sys.exit(1)
def run_unittests():
# several ways to do this. either make a unittest.TestSuite or run command line
# here is the second option
unittest_dir = '/path/to/tests'
pexpect.spawn('python -m unittest discover -s %s -p "*test.py"'%unittest_dir)
test_response = pexpect.expect('Ran [0-9]+ tests in [0-9\.]+s') #catch end
print test_response.before #print output of unittests before ending.
return
def main():
server = run_sever()
run_unittests()
server.sendcontrol('d') #shutdown server
if __name__ == "__main__":
main()
Code sample of app.py:
# imports here
app = Flask(__name__)
app.config.from_pyfile('app.cfg')
db.init_app(app)
with app.app_context():
if os.path.exists('database/example.db'):
print 'already exists'
else:
print 'database created'
db.create_all()
#routes here
if __name__ == '__main__':
app.run()
On the initial load it will print 'database created' and then 'already exists'. If I load it again when the database has been created, 'already exists' prints twice.
already exists
* Running on http://127.0.0.1:5000/
* Restarting with reloader
already exists
Two part question:
1) Why is with app.app_context() block executing twice?
2) What is the better way to write this code?
To understand what happens go back to python basics. When you type python your_module.py python interpreter will read (interpreter) your code line by line. In this case it will execute your with app.app_context code when it arrives there and you will see the output database create first time or already exists next times. What happens next? Interpreter will simply continue to your if __name__ == '__main__': line and then since you run it directly it will call app.run method and by default the debug parameter is True and it will start the development server in debug mode and the automatic reloader will restarts your code (as mentioned by #burnpanck).
I created a simple plugin system for my application and for now, I want to run each plugin in a new thread.
Here is a part of my code:
def newThread(self, f, args=()):
t = threading.Thread(target=f, args=args)
t.deamon = True
t.start()
return t
print "s"
for mod in imported_modules:
if 'init' in vars(mod):
newThread(mod.init, None)
print 1
One of my plugins is a TCP server that is listening on the socket . If I run it in the main thread, the application doesn't load other plugins and wait until the server stops!
Also the above code does not run the init function on my plugin.
Now the question is:
How to call an external function in a new thread ?
Thanks in advance!
The problem is that when we are trying to create a new thread, we should pass args to the method we want to call it in new thread. If it doesn't get any params, we should pass it an empty tuple like this:
newThread(mod.init, ())