We Keep Coding

How Do You Thread an External Hanging API Call in Flask?

Getting the specifics out of the way, I'm writing an open source P2P social network over IPFS and Flask -- I know, it's been done. I'm choosing Flask because pyinstaller can put it in an exe file.
I am attempting to update my IPNS every 10 minutes to publish all status updates I've added to the network during said 10 minutes. The cron function from setup class (from library.py) is where that updater function is stored. At first, I threaded the cron function from init of setup. The server hung. Then I moved the threading process over to app.before_first_request. The server still hangs.
https://pastebin.com/bXHTuH83 (main.py)
from flask import Flask, jsonify
from library import *
#=========================TO BE DELETED=========================================
def pretty(json):
json = dumps(loads(json), indent=4, sort_keys=True)
return json
#===============================================================================
app = Flask(__name__)
GANN = setup()
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
#app.route("/")
def home():
return "Hello World!!!"
if __name__ == "__main__":
app.run(port="80", debug=True, threaded=True)
https://pastebin.com/W5P8Tpvd (library.py)
from threading import Thread
from time import time, sleep
import urllib.request
from json import loads, dumps
def api(*argv, **kwargs):
url = "http://127.0.0.1:5001/api/v0/"
for arg in argv:
arg = arg.replace(" ", "/")
if arg[:-1] != "/":
arg += "/"
url += arg
url = url[0:-1]
if kwargs:
url+="?"
for val in kwargs:
url = url + val + "=" + kwargs[val] + "&"
url = url[0:-1]
print(url)
try:
with urllib.request.urlopen(url, timeout=300) as response:
return response.read()
except:
return b"""{"ERROR": "CANNOT CONNECT TO IPFS!"}"""
class setup():
def __init__(self):
api("files", "mkdir", arg="/GANN", parents="True")
self.root_hash = ""
def update_root(self):
try:
for entry in loads(api("files", "ls", l="True").decode())["Entries"]:
if entry["Name"] == "GANN":
self.root_hash = entry["Hash"]
except:
return """{"ERROR": "CANNOT FIND ROOT DIRECTORY"}"""
def publish_root(self):
api("name", "publish", arg=self.root_hash)
def cron(self):
while True:
print("CRON Thread Started!")
self.update_root()
self.publish_root()
sleep(600)
I have searched the web for a couple days and have yet to find a threading technique that will split from the main process and not hang the server from taking other requests. I believe I'm on a single stream connection, as IPFS blocks connections to every other device in my home when it's started. It takes a couple minutes for the CLI IPNS update to go through, so I set urllib's timeout to 300 seconds.

Well what I think the threading code is not correct.
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
Here you created a Thread object. The argument must be callable, but you called your method already here. so the right way would be
Thread(target=GANN.cron)
So the thread can call the cron function later. having said that, the Thread must be started, so it will call the function target you gave. So it must be ike
thread_cron = Thread(target=GANN.cron)
thread_cron.start()
Since you called the GANN.cron() , the method starts executing and your app hung!

Keep Python COM local server listening open in paralallel while the main code runs

I am trying to build a COM server to get real time information. The problem is that all other functionalities stop while localserver is opened. The rest of the code just runs when the localserver is closed.
I have searched for solutions and failed trying multiprocessing, not because this wouldn't work, I guess because I suck. Anyway, I am stuck in this part.
import pythoncom
import win32com
from win32com.server import localserver
from multiprocessing import Process
class PythonUtilities(object):
_reg_clsid_ = '{D9C54599-9011-4678-B1EB-A07FD272F0AF}'
_reg_desc_ = "Change information between programs"
_reg_progid_ = "Python.LetsTalk"
_public_attrs_ = ['speech', 'roger']
_readonly_attrs_ = ['roger']
_public_methods_ = ['talktome']
def __init__(self):
self.roger = 'roger'
self.speech = None
def talktome(self,speech):
self.speech = speech
print ('New speech received: ' + self.speech)
return self.roger
### ___ ###
def runserver(mess):
print(mess)
localserver.serve(['{D9C54599-9011-4678-B1EB-A07FD272F0AF}'])
if __name__=='__main__':
pu = PythonUtilities
print ("Registering COM Server ")
win32com.server.register.UseCommandLine(pu)
# Fine so far.
# The problem starts here:
localserver.serve(['{D9C54599-9011-4678-B1EB-A07FD272F0AF}'])
#... rest of the code waiting for localserver be closed
# Experiment... Doesnt work:
#proc = Process(target=runserver, args = ('starting process',))
#proc.start()
#proc.join()
It's important to say that all messages sent from the client seem to be correctly displayed BUT ONLY AFTER I close the local server manually. I want to receive it in real time like a chat app. I mean, I want to keep the localserver opened and being able to work with the information received along the rest of the code.

My problem was solved rewriting the localserver.serve() function and starting it in a new thread as the code below.
import pythoncom
from win32com.client import Dispatch # to get attributes
from win32com.server import register, factory
from threading import Thread
from queue import Queue
class PythonUtilities(object):
_reg_clsid_ = '{D9C54599-9011-4678-B1EB-A07FD272F0AF}'
_reg_desc_ = "Change information between programs"
_reg_progid_ = "Python.LetsTalk"
_public_attrs_ = ['speech']
_public_methods_ = ['talktome']
queue_speech = Queue()
def talktome(self,speech):
self.queue_speech.put(speech)
print ('New speech received: ' + speech)
return 'roger'
### ___ ###
# use instead localserver.serve()
def runserver():
# added - multithread support
pythoncom.CoInitialize()
clsids = ['{D9C54599-9011-4678-B1EB-A07FD272F0AF}']
infos = factory.RegisterClassFactories(clsids)
# commented - from original localserver.serve() method
#pythoncom.EnableQuitMessage(win32api.GetCurrentThreadId())
pythoncom.CoResumeClassObjects()
pythoncom.PumpMessages()
factory.RevokeClassFactories( infos )
pythoncom.CoUninitialize()
if __name__=='__main__':
#use this
server_thread = Thread(target=runserver) # Process works as well
server_thread.start()
#instead this
#localserver.serve(['{D9C54599-9011-4678-B1EB-A07FD272F0AF}'])
#... rest of the code now works in parallel
Also I have made some improvements like Queue to get data later. I hope it can help others.

Run multiple servers in python at same time (Threading)

I have 2 servers in python, I want to mix them up in one single .py and run together:
Server.py:
import logging, time, os, sys
from yowsup.layers import YowLayerEvent, YowParallelLayer
from yowsup.layers.auth import AuthError
from yowsup.layers.network import YowNetworkLayer
from yowsup.stacks.yowstack import YowStackBuilder
from layers.notifications.notification_layer import NotificationsLayer
from router import RouteLayer
class YowsupEchoStack(object):
def __init__(self, credentials):
"Creates the stacks of the Yowsup Server,"
self.credentials = credentials
stack_builder = YowStackBuilder().pushDefaultLayers(True)
stack_builder.push(YowParallelLayer([RouteLayer, NotificationsLayer]))
self.stack = stack_builder.build()
self.stack.setCredentials(credentials)
def start(self):
self.stack.broadcastEvent(YowLayerEvent(YowNetworkLayer.EVENT_STATE_CONNECT))
try:
logging.info("#" * 50)
logging.info("\tServer started. Phone number: %s" % self.credentials[0])
logging.info("#" * 50)
self.stack.loop(timeout=0.5, discrete=0.5)
except AuthError as e:
logging.exception("Authentication Error: %s" % e.message)
if "<xml-not-well-formed>" in str(e):
os.execl(sys.executable, sys.executable, *sys.argv)
except Exception as e:
logging.exception("Unexpected Exception: %s" % e.message)
if __name__ == "__main__":
import sys
import config
logging.basicConfig(stream=sys.stdout, level=config.logging_level, format=config.log_format)
server = YowsupEchoStack(config.auth)
while True:
# In case of disconnect, keeps connecting...
server.start()
logging.info("Restarting..")
App.py:
import web
urls = (
'/', 'index'
)
app = web.application(urls, globals())
class index:
def GET(self):
greeting = "Hello World"
return greeting
if __name__ == "__main__":
app.run()
I want to run both together from single .py file together.
If I try to run them from one file, either of the both starts and other one starts only when first one is done working.
How can I run 2 servers in python together?

import thread
def run_app1():
#something goes here
def run_app2():
#something goes here
if __name__=='__main__':
thread.start_new_thread(run_app1)
thread.start_new_thread(run_app2)
if you need to pass args to the functions you can do:
thread.start_new_thread(run_app1, (arg1,arg2,....))
if you want more control in your threads you could go:
import threading
def app1():
#something here
def app2():
#something here
if __name__=='__main__':
t1 = threading.Thread(target=app1)
t2 = threading.Thread(target=app2)
t1.start()
t2.start()
if you need to pass args you can go:
t1 = threading.Thread(target=app1, args=(arg1,arg2,arg3.....))
What's the differences between thread vs threading? Threading is higher level module than thread and in 3.x thread got renamed to _thread... more info here: http://docs.python.org/library/threading.html but that's for another question I guess.
So in your case, just make a function that runs the first script, and the second script, and just spawn threads to run them.

python: simple dbus example- os.fork() in service routine?

I am trying to write dbus server where I want to run some external shell program (grep here) to do the job.
when I do:
prompt$ server.py
then:
prompt$ client.py # works fine, ie. runs grep command in child process.
prompt$ client.py # ..., but second invocation produces following error message:
DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name org.example.ExampleService was not provided by any .service files
I am stuck. Are You able to help me?
here is server.py (client.py thereafter):
import gtk, glib
import os
import dbus
import dbus.service
import dbus.mainloop.glib
import subprocess
messages_queue=list()
grep_pid=0
def queue_msg(message):
global messages_queue
messages_queue.append(message)
return
def dequeue_msg():
global messages_queue,grep_pid
if grep_pid != 0:
try:
pid=os.waitpid(grep_pid,os.P_NOWAIT)
except:
return True
if pid[0] == 0:
return True
grep_pid=0
if len(messages_queue) == 0:
return True
else:
tekst=messages_queue.pop(0)
cmd="grep 'pp'"
print cmd
#works fine, when I do return here
#return True
grep_pid=os.fork()
if grep_pid != 0:
return True
os.setpgid(0,0)
pop=subprocess.Popen(cmd,shell=True,stdin=subprocess.PIPE)
pop.stdin.write(tekst)
pop.stdin.close()
pop.wait()
exit(0)
class DemoException(dbus.DBusException):
_dbus_error_name = 'org.example.Exception'
class MyServer(dbus.service.Object):
#dbus.service.method("org.example.ExampleInterface",
in_signature='', out_signature='')
def QueueMsg(self):
queue_msg("ppppp")
#dbus.service.method("org.example.ExampleInterface",
in_signature='', out_signature='')
def Exit(self):
mainloop.quit()
from dbus.mainloop.glib import threads_init
if __name__ == '__main__':
glib.threads_init()
threads_init()
dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
session_bus = dbus.SessionBus()
name = dbus.service.BusName("org.example.ExampleService", session_bus)
object = MyServer(session_bus, '/My')
glib.timeout_add_seconds(1, dequeue_msg)
mainloop = glib.MainLoop()
print "Running example service."
mainloop.run()
now client.py:
import sys
from traceback import print_exc
import dbus
def main():
bus = dbus.SessionBus()
try:
remote_object = bus.get_object("org.example.ExampleService",
"/My")
except dbus.DBusException:
print_exc()
sys.exit(1)
iface = dbus.Interface(remote_object, "org.example.ExampleInterface")
iface.QueueMsg()
if sys.argv[1:] == ['--exit-service']:
iface.Exit()
if __name__ == '__main__':
main()

You usually get this error message when you try to access a service that is no longer available. Check if your server is still running.
You can use d-feet to debug your dbus connections.

The error message about the missing .service file means that you need to create a service file in dbus-1/services.
For example:
# /usr/local/share/dbus-1/services/org.example.ExampleService.service
[D-BUS Service]
Name=org.example.ExampleService
Exec=/home/user1401567/service.py
A lot of tutorials don't include this detail (maybe .service files didn't use to be required?) But, at least on Ubuntu 12.04, dbus services can't be connected to without it.

How can I run 2 loops simultaneously in python?

I have an application (Gtk) that has an embedded server (using circuits). Both components (The GUI and Server) have infinite loops. How can I run both loops simultaneously ?
I also need the server loop to end when the gtk loop ends.
The code for the example server
from circuits.web import Server, Controller
import os
class MyServer(Controller):
def index(self):
return "Hello World"
server = Server(8000)
server += MyServer()
server.run()
and the code for example gtk application
import gtk
class App:
def __init__(self):
self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
self.window.connect("destroy",gtk.main_quit)
self.window.show_all()
gtk.main()
if __name__ == '__main__':
app = App()

You could use the multiprocessing module to do this:
from multiprocessing import Process
def run_app():
... run the app ...
def run_server():
... run the server ...
def main():
app = Process(target=run_app)
app.start()
server = Process(target=run_server)
server.start()
app.join()
server.terminate()
if __name__ == "__main__":
main()
Otherwise, if you're using Python < 2.6 on Unix, you could fiddle around with os.fork() to do the same sort of thing. Threading might work, but I don't know how well GTK or circuits plays with threads.

You could run the web server from another thread:
from threading import Thread
# ...
server = Server(8000)
server += MyServer()
web_server_thread = Thread(target=server.run)
web_server_thread.start()
gtk.main()