My webscraper takes about 10 mins to run, I am trying to use the threading library to allow my webscraper to run in the background after data has been returned to whomever made a call to my API I created with Flask.
My code looks something like this:
from threading import Thread
from flask import Flask
application = Flask(__name__)
class Compute(Thread):
def __init__(self, request):
print("init")
Thread.__init__(self)
self.request = request
def run(self):
print("RUN")
command = './webscrape.py -us "{user}" -p "{password}" -url "{url}"'.format(**self.request.json)
output = subprocess.call(['bash','-c', command])
#application.route('/scraper/run', methods=['POST'])
def init_scrape():
thread_a = Compute(request.__copy__())
thread_a.start()
return jsonify({'Scraping this site: ': request.json["url"]}), 201
if __name__ == '__main__':
application.run(host="0.0.0.0", port="8080")
Now I am testing my API with postman and when I make a POST request it prints out "init" but dosen't seem to go any further to start the run() function, what am I doing wrong?
Related
I am using a python script which uses cmdloop and takes input from the commandline:
import cmd
class SuperC2(cmd.Cmd):
def default(self, line):
f = open("command.txt", "w")
f.write(line)
f.close()
if __name__ == '__main__':
SuperC2().cmdloop('Shell')
Another script uses flask and reads the command.txt file and serves it (via GET request) to a client. It also listens for POSTs of clients that executed this command and posted the output:
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route('/command')
def command():
str = open('command.txt', 'r').read()
return str
#app.route('/result', methods = ['POST'])
def result():
if request.method == 'POST':
data = request.form
print(data["command"])
print(data["output"])
return "ok"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8000)
What I really would like to achieve is to have 1 script that runs the flask app and the cmdloop at the same time, with only the cmdloop showing the command and the result (it got from the flask POSTs). Can someone give an example on how it can be done? I'm guessing Process Queues or pipes?
Generally, the simplest way to transfer data between two scripts/threads/what have you is to use a socket from the socket library. Just have one script/thread host a server bound to 127.0.0.1 and connect to it from the other process!
I want to send a telegram notification when the user performs a specific task in my flask application. I'm using python-telegram-bot to handle telegram. Here's the simplified code:
#app.route('/route')
def foo():
# do some stuff...
# if stuff is completed successfully - send the notification
app.telegram_notifier.send_notification(some_data)
return ''
I'm using messagequeue from python-telegram-bot to avoid flood limits. As you might have expected, that's not working and I'm getting the following error:
telegram.ext.messagequeue.DelayQueueError: Could not process callback in stopped thread
I tried to launch it in a separate daemon thread but I also ended up with that error.
This functionality is used only once in the entire application so I want things to be simple and don't want to install more dependencies like Celery.
Is there a way to achieve this using threads or some other simple way?
EDIT (more code)
Here's simplified implementation of the telegram bot:
from telegram import Bot, ParseMode
from telegram.ext import messagequeue as mq
class Notifier(Bot):
def __init__(self):
super(Notifier, self).__init__('my token')
# Message queue setup
self._is_messages_queued_default = True
self._msg_queue = mq.MessageQueue(all_burst_limit=3, all_time_limit_ms=3500)
self.chat_id = 'my chat ID'
#mq.queuedmessage
def _send_message(self, *args, **kwargs):
return super(Notifier, self).send_message(*args, **kwargs)
def send_notification(self, data: str):
msg = f'Notification content: <b>{data}</b>'
self._send_message(self.chat_id, msg, ParseMode.HTML)
In the app factory method:
from notifier import Notifier
def init_app():
app = Flask(__name__)
app.telegram_notifier = Notifier()
# some other init stuff...
return app
The thing with threads I tried:
#app.route('/route')
def foo():
# do some stuff...
# First method
t = Thread(target=app.telegram_notifier.send_notification, args=('some notification data',), daemon=True)
t.start()
# Second method
t = Thread(target=app.telegram_notifier.send_notification, args=('some notification data',))
t.start()
t.join()
return ''
I am trying to use Flask to send a stream of events to a front-end client as documented in this question. This works fine if I don't access anything in the request context, but fails as soon as I do.
Here's an example to demonstrate.
from time import sleep
from flask import Flask, request, Response
app = Flask(__name__)
#app.route('/events')
def events():
return Response(_events(), mimetype="text/event-stream")
def _events():
while True:
# yield "Test" # Works fine
yield request.args[0] # Throws RuntimeError: Working outside of request context
sleep(1)
Is there a way to access the request context for server-sent events?
You can use the #copy_current_request_context decorator to make a copy of the request context that your event stream function can use:
from time import sleep
from flask import Flask, request, Response, copy_current_request_context
app = Flask(__name__)
#app.route('/events')
def events():
#copy_current_request_context
def _events():
while True:
# yield "Test" # Works fine
yield request.args[0]
sleep(1)
return Response(_events(), mimetype="text/event-stream")
Note that to be able to use this decorator the target function must be moved inside the view function that has the source request.
I am using web.py to host a simple web service. The web service runs an analytics application in the backend (inside ClassA). During the initialization of web.py, I'd like to pre-load all data into the memory (i.e call a = ClassA() only once when web server is started), and when the user sends a web request, the web server will just response with the pre-calculated result (i.e return a.do_something).
The code below seems to run init() of class 'add' everytime a HTTP POST request is received. This is a waste of time because the initialization stage takes pretty long. Is it possible to initialize ClassA only once?
import web
from aclass import ClassA
urls = (
'/add', 'add'
)
class add:
def __init__(self):
a = ClassA()
def POST(self):
return a.do_something()
if __name__ == "__main__":
app = web.application(urls, globals())
app.run()
Try:
class add:
a = ClassA()
def POST(self):
return add.a.do_something()
This will make it a class-bound parameter instead of a instance-bound one, i.e. only initializing it once.
I have a bottle app that I eventually wan't to deploy on apache (just fyi in case that's important).
Now I need to run a function once after the bottle app is started. I can't just put it into a routed function because it has to run even if no user has accessed the site yet.
Any best pratice to do this ?
The function starts a APScheduler Instance and adds a jobstore to it.
Here's what I do.
def initialize():
//init whatever you need.
if __name__ == '__main__':
initialize()
#bottle.run(port='8080', yatta yatta)
Honestly your problem is simply a sync vs async issue. Use gevent to easily convert to microthreads, and then launch each separately. You can even add a delay either in your function or before with gevent.sleep if you want to wait for the web server to finish launching.
import gevent
from gevent import monkey, signal, spawn, joinall
monkey.patch_all()
from gevent.pywsgi import WSGIServer
from bottle import Bottle, get, post, request, response, template, redirect, hook, abort
import bottle
#get('/')
def mainindex():
return "Hello World"
def apScheduler():
print "AFTER SERVER START"
if __name__ == "__main__":
botapp = bottle.app()
server = WSGIServer(("0.0.0.0", 80), botapp)
threads = []
threads.append(spawn(server.serve_forever))
threads.append(spawn(apScheduler))
joinall(threads)
Create an APScheduler class.
Look at examples of object use and creation in this same site bacause it's too general to give an especific example to copy.
I don't know if this helps.
class Shed(object):
def __init__(self): # this to start it
# instruccions here
def Newshed(self, data):
# Call from bottle
# more methods ...
...
# init
aps = Shed() # this activates Shed.__init__()
...
# in the #router
x = aps.Newshed(data) # or whatever
Anyway I'm still learning this stuff and it's just an idea.
import threading
import bottle
def init_app():
def my_function_on_startup():
# some code here
pass
app = bottle.app()
t = threading.Thread(target=my_function_on_startup)
t.daemon = True
t.start()
return app
app = init_app()
#app.route("/")
def hello():
return "App is running"
if __name__ == "__main__":
bottle.run(app, host='localhost', port=8080, debug=True)