We Keep Coding

Stop child thread from main thread in python

I'm starting a webserver in a new thread. After all tests are run I want to kill the child thread with running server inside. The only one solution is interrupting entire process with all threads inside by calling "os.system('kill %d' % os.getpid())" (see the code below). I'm not sure it's the smartest solution. I'm not sure all threads will be killed after all. Could I send some kind of "Keyboard interrupt" signal to stop the thread before exiting main thread?
import http
import os
import sys
import unittest
import time
import requests
import threading
from addresses import handle_get_addresses, load_addresses
from webserver import HTTPHandler
def run_in_thread(fn):
def run(*k, **kw):
t = threading.Thread(target=fn, args=k, kwargs=kw)
t.start()
return t
return run
#run_in_thread
def start_web_server():
web_host = 'localhost'
print("starting server...")
web_port = 8808
httpd = http.server.HTTPServer((web_host, web_port), HTTPHandler)
try:
httpd.serve_forever()
except KeyboardInterrupt:
pass
class TestAddressesApi(unittest.TestCase):
WEB_SERVER_THREAD: threading.Thread = None
#classmethod
def setUpClass(cls):
cls.WEB_SERVER_THREAD = start_web_server()
pass
#classmethod
def tearDownClass(cls):
print("shutting down the webserver...")
# here someting like cls.WEB_SERVER_THREAD.terminate()
# instead of line below
os.system('kill %d' % os.getpid())
def test_get_all_addresses(self):
pass
def test_1(self):
pass
if __name__ == "__main__":
unittest.main()

Maybe threading.Event is you wanted.

Just found a solution. Daemon Threads stop executing when main thread stops working

How Do You Thread an External Hanging API Call in Flask?

Getting the specifics out of the way, I'm writing an open source P2P social network over IPFS and Flask -- I know, it's been done. I'm choosing Flask because pyinstaller can put it in an exe file.
I am attempting to update my IPNS every 10 minutes to publish all status updates I've added to the network during said 10 minutes. The cron function from setup class (from library.py) is where that updater function is stored. At first, I threaded the cron function from init of setup. The server hung. Then I moved the threading process over to app.before_first_request. The server still hangs.
https://pastebin.com/bXHTuH83 (main.py)
from flask import Flask, jsonify
from library import *
#=========================TO BE DELETED=========================================
def pretty(json):
json = dumps(loads(json), indent=4, sort_keys=True)
return json
#===============================================================================
app = Flask(__name__)
GANN = setup()
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
#app.route("/")
def home():
return "Hello World!!!"
if __name__ == "__main__":
app.run(port="80", debug=True, threaded=True)
https://pastebin.com/W5P8Tpvd (library.py)
from threading import Thread
from time import time, sleep
import urllib.request
from json import loads, dumps
def api(*argv, **kwargs):
url = "http://127.0.0.1:5001/api/v0/"
for arg in argv:
arg = arg.replace(" ", "/")
if arg[:-1] != "/":
arg += "/"
url += arg
url = url[0:-1]
if kwargs:
url+="?"
for val in kwargs:
url = url + val + "=" + kwargs[val] + "&"
url = url[0:-1]
print(url)
try:
with urllib.request.urlopen(url, timeout=300) as response:
return response.read()
except:
return b"""{"ERROR": "CANNOT CONNECT TO IPFS!"}"""
class setup():
def __init__(self):
api("files", "mkdir", arg="/GANN", parents="True")
self.root_hash = ""
def update_root(self):
try:
for entry in loads(api("files", "ls", l="True").decode())["Entries"]:
if entry["Name"] == "GANN":
self.root_hash = entry["Hash"]
except:
return """{"ERROR": "CANNOT FIND ROOT DIRECTORY"}"""
def publish_root(self):
api("name", "publish", arg=self.root_hash)
def cron(self):
while True:
print("CRON Thread Started!")
self.update_root()
self.publish_root()
sleep(600)
I have searched the web for a couple days and have yet to find a threading technique that will split from the main process and not hang the server from taking other requests. I believe I'm on a single stream connection, as IPFS blocks connections to every other device in my home when it's started. It takes a couple minutes for the CLI IPNS update to go through, so I set urllib's timeout to 300 seconds.

Well what I think the threading code is not correct.
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
Here you created a Thread object. The argument must be callable, but you called your method already here. so the right way would be
Thread(target=GANN.cron)
So the thread can call the cron function later. having said that, the Thread must be started, so it will call the function target you gave. So it must be ike
thread_cron = Thread(target=GANN.cron)
thread_cron.start()
Since you called the GANN.cron() , the method starts executing and your app hung!

Asynchronous Function Call

I would like to learn how to call a function asynchronously in Python3. I think Tornado can do this. Currently, my code is returning nothing on the command line:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
async def count(end):
"""Print message when start equals end."""
start = 0
while True:
if start == end:
print('start = {0}, end = {1}'.format(start, end))
break
start = start + 1
def main():
# Start counting.
yield count(1000000000)
# This should print while count is running.
print('Count is running. Async!')
if __name__ == '__main__':
main()
Thanks

To call an async function, you need to provide an event loop to handle it. If you have a Tornado app, it provides such a loop, which allows you to make your handlers asynchronous:
from tornado.web import RequestHandler, url
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
async def do_something_asynchronous():
# e.g. call another service, read from database etc
return {'something': 'something'}
class YourAsyncHandler(RequestHandler):
async def get(self):
payload = await do_something_asynchronous()
self.write(payload)
application = web.Application([
url(r'/your_url', YourAsyncHandler, name='your_url')
])
http_server = HTTPServer(application)
http_server.listen(8000, address='0.0.0.0')
IOLoop.instance().start()
Outside of a Tornado app you can get the event loop from any number of providers, including the built-in asyncio library:
import asyncio
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(do_something_asynchronous())
finally:
event_loop.close()

Run multiple servers in python at same time (Threading)

I have 2 servers in python, I want to mix them up in one single .py and run together:
Server.py:
import logging, time, os, sys
from yowsup.layers import YowLayerEvent, YowParallelLayer
from yowsup.layers.auth import AuthError
from yowsup.layers.network import YowNetworkLayer
from yowsup.stacks.yowstack import YowStackBuilder
from layers.notifications.notification_layer import NotificationsLayer
from router import RouteLayer
class YowsupEchoStack(object):
def __init__(self, credentials):
"Creates the stacks of the Yowsup Server,"
self.credentials = credentials
stack_builder = YowStackBuilder().pushDefaultLayers(True)
stack_builder.push(YowParallelLayer([RouteLayer, NotificationsLayer]))
self.stack = stack_builder.build()
self.stack.setCredentials(credentials)
def start(self):
self.stack.broadcastEvent(YowLayerEvent(YowNetworkLayer.EVENT_STATE_CONNECT))
try:
logging.info("#" * 50)
logging.info("\tServer started. Phone number: %s" % self.credentials[0])
logging.info("#" * 50)
self.stack.loop(timeout=0.5, discrete=0.5)
except AuthError as e:
logging.exception("Authentication Error: %s" % e.message)
if "<xml-not-well-formed>" in str(e):
os.execl(sys.executable, sys.executable, *sys.argv)
except Exception as e:
logging.exception("Unexpected Exception: %s" % e.message)
if __name__ == "__main__":
import sys
import config
logging.basicConfig(stream=sys.stdout, level=config.logging_level, format=config.log_format)
server = YowsupEchoStack(config.auth)
while True:
# In case of disconnect, keeps connecting...
server.start()
logging.info("Restarting..")
App.py:
import web
urls = (
'/', 'index'
)
app = web.application(urls, globals())
class index:
def GET(self):
greeting = "Hello World"
return greeting
if __name__ == "__main__":
app.run()
I want to run both together from single .py file together.
If I try to run them from one file, either of the both starts and other one starts only when first one is done working.
How can I run 2 servers in python together?

import thread
def run_app1():
#something goes here
def run_app2():
#something goes here
if __name__=='__main__':
thread.start_new_thread(run_app1)
thread.start_new_thread(run_app2)
if you need to pass args to the functions you can do:
thread.start_new_thread(run_app1, (arg1,arg2,....))
if you want more control in your threads you could go:
import threading
def app1():
#something here
def app2():
#something here
if __name__=='__main__':
t1 = threading.Thread(target=app1)
t2 = threading.Thread(target=app2)
t1.start()
t2.start()
if you need to pass args you can go:
t1 = threading.Thread(target=app1, args=(arg1,arg2,arg3.....))
What's the differences between thread vs threading? Threading is higher level module than thread and in 3.x thread got renamed to _thread... more info here: http://docs.python.org/library/threading.html but that's for another question I guess.
So in your case, just make a function that runs the first script, and the second script, and just spawn threads to run them.

Use tornado async code in a regular python script

I have some asynchronous functions using tornado gen.coroutine that I normally use as part of a tornado-based web application. However, I want to call some of them from a plain old python script to do some administration tasks. How do I do this?
from tornado import gen
import some_internal_stuff
#gen.coroutine
def myfunc(x):
y = yield some_internal_stuff.another_async_func(x)
raise gen.Return(y)
if __name__ == "__main__":
# What do I put here to call myfunc(1) and get the async return value?
pass
Update:
A more concrete example:
from tornado import gen
#gen.coroutine
def another_async_func(x):
print "aaf"
raise gen.Return(x + 1)
#gen.coroutine
def myfunc(x):
print "myfunc"
y = yield another_async_func(x)
print "back"
raise gen.Return(y)
def callback(y):
print "Callback called with %d" % y
if __name__ == "__main__":
myfunc(1, callback=callback)
Running this outputs:
myfunc
aaf

There's a built-in method run_sync in IOLoop to run a single call and then stop the loop, so it's pretty trivial to just add an event loop to a plain python script provided you have tornado in the PYTHONPATH.
With the concrete example:
from tornado import gen, ioloop
#gen.coroutine
def another_async_func(x):
print "aaf"
raise gen.Return(x + 1)
#gen.coroutine
def myfunc(x):
print "myfunc"
y = yield another_async_func(x)
print "back"
raise gen.Return(y)
#gen.coroutine
def main():
y = yield myfunc(1)
print "Callback called with %d" % y
if __name__ == "__main__":
ioloop.IOLoop.instance().run_sync(main)
This outputs:
myfunc
aaf
back
Callback called with 2
Note that run_sync doesn't nest well; if you call run_sync in a function called by run_sync on the same IOLoop the completion of the inner call will stop the IOLoop and no further yields after the inner call will return.

Here's another possibility, using threads, that would work depending on the problem complexity and your needs:
if __name__ == "__main__":
import threading, time
# The tornado IO loop doesn't need to be started in the main thread
# so let's start it in another thread:
t = threading.Thread(target=IOLoop.instance().start)
t.daemon = True
t.start()
myfunc(1, callback=callback)
# now the main loop needs wait; you can do that by polling a value, sleeping,
# or waiting on a lock. I've chosen to sleep here, but a lock is probably more
# appropriate; and, once computation is done, release the lock.
time.sleep(2)