We Keep Coding

How Do You Thread an External Hanging API Call in Flask?

Getting the specifics out of the way, I'm writing an open source P2P social network over IPFS and Flask -- I know, it's been done. I'm choosing Flask because pyinstaller can put it in an exe file.
I am attempting to update my IPNS every 10 minutes to publish all status updates I've added to the network during said 10 minutes. The cron function from setup class (from library.py) is where that updater function is stored. At first, I threaded the cron function from init of setup. The server hung. Then I moved the threading process over to app.before_first_request. The server still hangs.
https://pastebin.com/bXHTuH83 (main.py)
from flask import Flask, jsonify
from library import *
#=========================TO BE DELETED=========================================
def pretty(json):
json = dumps(loads(json), indent=4, sort_keys=True)
return json
#===============================================================================
app = Flask(__name__)
GANN = setup()
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
#app.route("/")
def home():
return "Hello World!!!"
if __name__ == "__main__":
app.run(port="80", debug=True, threaded=True)
https://pastebin.com/W5P8Tpvd (library.py)
from threading import Thread
from time import time, sleep
import urllib.request
from json import loads, dumps
def api(*argv, **kwargs):
url = "http://127.0.0.1:5001/api/v0/"
for arg in argv:
arg = arg.replace(" ", "/")
if arg[:-1] != "/":
arg += "/"
url += arg
url = url[0:-1]
if kwargs:
url+="?"
for val in kwargs:
url = url + val + "=" + kwargs[val] + "&"
url = url[0:-1]
print(url)
try:
with urllib.request.urlopen(url, timeout=300) as response:
return response.read()
except:
return b"""{"ERROR": "CANNOT CONNECT TO IPFS!"}"""
class setup():
def __init__(self):
api("files", "mkdir", arg="/GANN", parents="True")
self.root_hash = ""
def update_root(self):
try:
for entry in loads(api("files", "ls", l="True").decode())["Entries"]:
if entry["Name"] == "GANN":
self.root_hash = entry["Hash"]
except:
return """{"ERROR": "CANNOT FIND ROOT DIRECTORY"}"""
def publish_root(self):
api("name", "publish", arg=self.root_hash)
def cron(self):
while True:
print("CRON Thread Started!")
self.update_root()
self.publish_root()
sleep(600)
I have searched the web for a couple days and have yet to find a threading technique that will split from the main process and not hang the server from taking other requests. I believe I'm on a single stream connection, as IPFS blocks connections to every other device in my home when it's started. It takes a couple minutes for the CLI IPNS update to go through, so I set urllib's timeout to 300 seconds.

Well what I think the threading code is not correct.
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
Here you created a Thread object. The argument must be callable, but you called your method already here. so the right way would be
Thread(target=GANN.cron)
So the thread can call the cron function later. having said that, the Thread must be started, so it will call the function target you gave. So it must be ike
thread_cron = Thread(target=GANN.cron)
thread_cron.start()
Since you called the GANN.cron() , the method starts executing and your app hung!

While solving Captcha, Program gets stuck using Flask and Multiprocessing together

So I have been playing around with Captcha, Multiprocessing and Flask.
What have I done so far:
Basically what I have done is that I have created own Mutltiprocessing where I enter how many tasks/processers I want to run during this script. It will say if I enter 3 then it will give me a process of 3 "threads" which works fine.
Also whenever webbrowser.open('http://Thrillofit.baller.com:5000/solve') opens, It works aswell. Also get a captcha that is able to solve.
Basically meaning the captcha does work and also the multiprocessing:
What's the issue then?
The issue is sitting now whenever I run the program it gets stuck when solving, basically I can solve how many captchas as I want but it will not continue which I can't get a grip of why? A GIF pic of what is happening: https://i.gyazo.com/d5f183471f20be5eda6be939d255a157.mp4
In the video you can see that i am trying to solve the captcha but nothing happeneds to the program like it gets stuck.
MY thoughts is that it might be a issue between Multiprocessing and Captcha/flask but I can't see the issue and I have been completely blind at this point. Maybe there is someone that might see the problem?
The goals is to whenever I solve a captcha, It should print out a token (Inside the def passwd(): method but nothing prints out and just gets stuck pretty much...
import requests, time, json, re, sys, os, timeit, random, multiprocessing, traceback, threading, names, logging, webbrowser, _thread
from flask import Flask, render_template, request, redirect
tokens = []
def captureToken(token):
expiry = datetime.now().timestamp() + 115
tokenDict = {
'expiry': expiry,
'token': token
}
tokens.append(tokenDict)
return
def sendToken():
while not tokens:
pass
token = tokens.pop(0)
return token['token']
def manageTokens():
while True:
for item in tokens:
if item['expiry'] < datetime.now().timestamp():
tokens.remove(item)
time.sleep(5)
app = Flask(__name__)
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
#app.route('/', methods=['GET'])
def base():
return redirect("http://Thrillofit.baller.com:5000/solve", code=302)
#app.route('/solve', methods=['GET'])
def solve():
sitekey = "6LdyFRkUAAAAAF2YmQ9baZ6ytpVnbVSAymVpTXKi"
return render_template('index.html', sitekey=sitekey)
#app.route('/submit', methods=['POST'])
def submit():
token = request.form.get('g-recaptcha-response', '')
captureToken(token)
return redirect("http://Thrillofit.baller.com:5000/solve", code=302)
def info(thread):
global prod
prod = int(thread) + 1
passwd()
def passwd():
lilcapttoken = sendToken()
print(lilcaptoken)
def main():
webbrowser.open('http://Thrillofit.baller.com:5000/solve')
user_input = 0
while True:
try:
user_input = int(input(Fore.WHITE + 'How many tasks do you wanna run? [NUMBERS] \n' + Fore.RESET))
except ValueError:
print(Fore.RED + "Stop being stupid" + Fore.RESET)
continue
else:
HowManyThread = user_input
print()
i = 0
jobs = []
for i in range(HowManyThread):
p = multiprocessing.Process(target=info, args=(str(i),))
jobs.append(p)
time.sleep(.5)
p.start()
for p in jobs:
p.join()
sys.exit()
if __name__ == '__main__':
try:
_thread.start_new_thread(app.run, ())
_thread.start_new_thread(manageTokens, ())
main()
except Exception as e:
print(e)
print(traceback.print_exc())
print(traceback)

How to pass a really long string from one script to another in Python?

I have two python script sender.py and reciever.py.
I'm trying to send a really long string from sender to receiver by giving it at command line in os.system() but it gives weird errors on Ubuntu which I can't recall (two line error, each line with sh), while on windows it says that The command line is too long.
sender.py (invoking os.system() at the end)
class Bob(object):
def __init__(self):
print(self.generateReport())
def generateReport(self):
# variables used to format report are defined here.
report = '''some really really long string'''.format(fw_date, release_version, release_date, release_file_tag, crc, checksum[0], checksum[1],
checksum[2], checksum[3], checksum[4], checksum[5], pon_firmware, mib_release
)
os.system('python reciever.py '+report)
return report
def main():
try:
bObj = Bob()
# miniSVNObj.execute()
except Exception as ex:
# print ""
print(ex)
if __name__ == "__main__":
main()
reciever.py
import sys
message = "Dear All Concerned, \n"+sys.argv[1]
print(message)

The size of the command line is restricted by the operating system. Arbitrarily large pieces of data should be read via a file, either standard input or a file that is explicitly opened.
For example,
import subprocess
class Bob(object):
def __init__(self):
print(self.generateReport())
def generateReport(self):
# variables used to format report are defined here.
report = '''...'''.format(fw_date, release_version, release_date, release_file_tag, crc, checksum[0], checksum[1],
checksum[2], checksum[3], checksum[4], checksum[5], pon_firmware, mib_release
)
with tempfile.TemporaryFile() as fh:
fh.write(report)
fh.seek(0)
subprocess.call(['python', 'receiver.py', stdin=fh)
return report
def main():
try:
bObj = Bob()
# miniSVNObj.execute()
except Exception as ex:
# print ""
print(ex)
if __name__ == "__main__":
main()
And receiver.py becomes
import sys
msg = sys.stdin.read()
message = "Dear All Concerned, \n" + msg
print(message)

Receiver could get the string via its stdin. That's what it's for :-)

Streaming a response doesn't work with Flask-Restful

I have a scenario where I want to show output of a long running script through a Flask API. I followed an example given for Flask and it works. I get dmesg steam in my browser.
import subprocess
import time
from flask import Flask, Response
app = Flask(__name__)
#app.route('/yield')
def index():
def inner():
proc = subprocess.Popen(
['dmesg'], # call something with a lot of output so we can see it
shell=True,
stdout=subprocess.PIPE
)
for line in iter(proc.stdout.readline,''):
time.sleep(1) # Don't need this just shows the text streaming
yield line.rstrip() + '<br/>\n'
return Response(inner(), mimetype='text/html') # text/html is required for most browsers to show this
The thing is, I have been using Flask-Restful from a long time. So I want to do the streaming using it. I tried it and it's not working.
import subprocess
import time
from flask import Response
from flask_restful import Resource
class CatalogStrings(Resource):
def get(self):
return Response(inner(), mimetype='text/html')
def inner():
proc = subprocess.Popen(
['dmesg'], # call something with a lot of output so we can see it
shell=True,
stdout=subprocess.PIPE
)
for line in iter(proc.stdout.readline, ''):
time.sleep(1) # Don't need this just shows the text streaming
yield line.rstrip() + '<br/>\n'
Please help

Handbrake fails with "scan: unrecognized file type" in my Python conversion daemon

I'm developing a Python daemon responsible for converting multimedia files to .mp4 format.
The idea is to have the daemon running and, whenever the user requires, I add the desired video to a Queue and a thread eventually gets the video from the queue and calls Handbrake via Subprocess in order to do the conversion. For simplicity's sake, I'm using only one thread at the moment.
Here's my code.
First, the daemon (server.py, adapted from Kris Johnson's post here):
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os, sys
import os.path
import logging.config
import SocketServer
import optparse
import resource
import socket, tempfile
import time
from threadedqueue import QueueServer
version = '0.1'
SERVER_PORT=6666
SERVER_SOCKET='server_socket'
SERVER_TYPE=SocketServer.UnixStreamServer
ServerBase = SERVER_TYPE
if ServerBase == SocketServer.UnixStreamServer:
server_address = os.path.join(tempfile.gettempdir(), SERVER_SOCKET)
SERVER_LOG=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'convertCentral.log')
logging.basicConfig(format='[%(asctime)s.%(msecs).03d] %(message)s', datefmt='%Y-%m-%d %H:%M:%S', filename=SERVER_LOG, level=logging.INFO)
class RequestHandler(SocketServer.StreamRequestHandler):
"""Request handler
An instance of this class is created for each connection made
by a client. The Server class invokes the instance's
setup(), handle(), and finish() methods.
The template implementation here simply reads a single line from
the client, breaks that up into whitespace-delimited words, and
then uses the first word as the name of a "command." If there is
a method called "do_COMMAND", where COMMAND matches the
commmand name, then that method is invoked. Otherwise, an error
message is returned to the client.
"""
def handle(self):
"""Service a newly connected client.
The socket can be accessed as 'self.connection'. 'self.rfile'
can be used to read from the socket using a file interface,
and 'self.wfile' can be used to write to the socket using a
file interface.
When this method returns, the connection will be closed.
"""
# Read a single request from the input stream and process it.
request = self.rfile.readline()
if request:
self.server.log('request %s: %s',
self.connection.getpeername(), request.rstrip())
try:
self.process_request(request)
except Exception, e:
self.server.log('exception: %s' % str(e))
self.wfile.write('Error: %s\n' % str(e))
else:
self.server.log('error: unable to read request')
self.wfile.write('Error: unable to read request')
def process_request(self, request):
"""Process a request.
This method is called by self.handle() for each request it
reads from the input stream.
This implementation simply breaks the request string into
words, and searches for a method named 'do_COMMAND',
where COMMAND is the first word. If found, that method is
invoked and remaining words are passed as arguments.
Otherwise, an error is returned to the client.
"""
words = request.split()
if len(words) == 0:
self.server.log('error: empty request')
self.wfile.write('Error: empty request\n')
return
command = words[0]
args = words[1:]
methodname = 'do_' + command
if not hasattr(self, methodname):
self.server.log('error: invalid command')
self.wfile.write('Error: "%s" is not a valid command\n' % command)
return
method = getattr(self, methodname)
method(*args)
def do_stop(self, *args):
self.wfile.write('Stopping server\n')
self.server.stop()
"""Process an 'echo' command"""
def do_echo(self, *args):
self.wfile.write(' '.join(args) + '\n')
"""Process a 'convert' command"""
def do_convert(self, video):
self.wfile.write('Converting %s\n' % (video))
try:
self.server.addVideo(video)
except Exception as e:
logging.info("ERROR: %s" % e)
class Server(ServerBase):
def __init__(self, server_address):
self.__daemonize()
self.pool = QueueServer()
if ServerBase == SocketServer.UnixStreamServer:
# Delete the socket file if it already exists
if os.access(server_address, 0):
os.remove(server_address)
ServerBase.__init__(self, server_address, RequestHandler)
def addVideo(self, video):
self.pool.add(video)
def log(self, format, *args):
try:
message = format % args
logging.info("%s" % message)
except Exception, e:
print str(e)
def serve_until_stopped(self):
self.log('started')
self.__stopped = False
while not self.__stopped:
self.handle_request()
self.log('stopped')
def stop(self):
self.__stopped = True
def __daemonize(self):
UMASK = 0
WORKDIR = '/'
MAXFD = 1024
if hasattr(os, 'devnull'):
REDIRECT_TO = os.devnull
else:
REDIRECT_TO = '/dev/null'
try :
if os.fork() != 0:
os._exit(0)
os.setsid()
if os.fork() != 0:
os._exit(0)
os.chdir(WORKDIR)
os.umask(UMASK)
except OSError, e:
self.log('exception: %s %s', e.strerror, e.errno)
raise Exception, "%s [%d]" % (e.strerror, e.errno)
except Exception, e:
self.log('exception: %s', str(e))
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if maxfd == resource.RLIM_INFINITY:
maxfd = MAXFD
for fd in range(0, maxfd):
try:
os.close(fd)
except OSError:
pass
os.open(REDIRECT_TO, os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)
""" Run a server as a daemon """
def run_server(options, args):
print("convertCentral running on %s" % server_address)
svr = Server(server_address)
svr.serve_until_stopped()
svr.server_close()
"""Send request to the server and process response."""
def do_request(options, args):
if ServerBase == SocketServer.UnixStreamServer:
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
# Send request
s.connect(server_address)
s.sendall(' '.join(args) + '\n')
# Print response
sfile = s.makefile('rb')
line = sfile.readline()
while line:
print line,
line = sfile.readline()
#######################################################################
#######################################################################
if __name__ == '__main__':
optparser = optparse.OptionParser(usage=usage,
version=version)
(options, args) = optparser.parse_args()
if len(args) == 0:
optparser.print_help()
sys.exit(-1)
if args[0] == 'start':
run_server(options, args[1:])
else:
do_request(options, args)
Then, the queue (threadedqueue.py - sorry about the name, not feeling particularly creative):
#! /usr/bin/python
# -*- coding: utf-8 -*-
import os, time, threading, psutil, resource, logging, subprocess as sp, sys
from Queue import Queue
from threading import Thread
SERVER_LOG=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'convertCentral.log')
logging.basicConfig(format='[%(asctime)s.%(msecs).03d] %(message)s', datefmt='%Y-%m-%d %H:%M:%S', filename=SERVER_LOG, level=logging.INFO)
class QueueServer(object):
current_video_queue = Queue(maxsize=0)
N_WORKER_THREADS = 1
counter = 0
def __init__(self):
logging.info("[QueueServer] Initializing the video conversion queue")
for i in range(self.N_WORKER_THREADS):
logging.info("Firing thread")
t = Thread(target=self.worker)
t.daemon = True
t.start()
''' Converts the video using Handbrake via subprocess'''
def convertVideo(self, video):
logging.info("Now converting %s" % video)
fileName, fileExtension = os.path.splitext(video)
payload = "nice -n 15 HandBrakeCLI -i %s -e x264 -q 15 -o %s.mp4" % (video, fileName)
pr = sp.Popen(payload, shell=True, stdout=open('/dev/null', 'w'), stderr=sp.STDOUT)
logging.info('Started handbrake')
pr.wait()
logging.info("EXIT CODE: %s " % pr.returncode)
self.counter = self.counter + 1
logging.info("Conversion's done. %d" % self.counter)
''' A worker thread '''
def worker(self):
while True:
logging.info("Getting one")
item = self.current_video_queue.get()
logging.info("Firing conversion: %s" % item)
self.convertVideo(item)
self.current_video_queue.task_done()
logging.info("All done")
''' Adds a video to the video conversion queue '''
def add(self, video):
logging.info("* Adding %s to the queue" % video)
self.current_video_queue.put(video)
logging.info("* Added %s to the queue" % video)
time.sleep(3)
Here's the deal: if I run the threadedqueue on its own, it works great.
However, if I run it using server.py, the conversion never happens because Handbrake crashes.
Here are the logs:
Hal#ubuntu:~/Desktop/convertCentral$ python server.py start
convertCentral running on /tmp/server_socket
Hal#ubuntu:~/Desktop/convertCentral$ python server.py convert UNKNOWN_PARAMETER_VALUE.WMV
[2014-04-17 18:05:44.793] request : convert UNKNOWN_PARAMETER_VALUE.WMV
Converting UNKNOWN_PARAMETER_VALUE.WMV
[2014-04-17 18:05:44.793] * Adding UNKNOWN_PARAMETER_VALUE.WMV to the queue
[2014-04-17 18:05:44.793] * Added UNKNOWN_PARAMETER_VALUE.WMV to the queue
[2014-04-17 18:05:44.793] Firing conversion: UNKNOWN_PARAMETER_VALUE.WMV
[2014-04-17 18:05:44.794] Now converting UNKNOWN_PARAMETER_VALUE.WMV
[2014-04-17 18:05:44.796] Started handbrake
[2014-04-17 18:05:45.046] Exit code: 0
[2014-04-17 18:05:45.046] Conversion's done. 1
[2014-04-17 18:05:45.046] All done
[2014-04-17 18:05:45.047] Getting one
I logged the subprocess's output to a file.
Here's what I got:
[18:05:44] hb_init: starting libhb thread
HandBrake 0.9.9 (2013051800) - Linux x86_64 - http://handbrake.fr
4 CPUs detected
Opening UNKNOWN_PARAMETER_VALUE.WMV...
[18:05:44] hb_scan: path=UNKNOWN_PARAMETER_VALUE.WMV, title_index=1
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening UNKNOWN_PARAMETER_VALUE.WMV/BDMV/index.bdmv
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening UNKNOWN_PARAMETER_VALUE.WMV/BDMV/BACKUP/index.bdmv
libbluray/bluray.c:1725: nav_get_title_list(UNKNOWN_PARAMETER_VALUE.WMV) failed (0x7fd44c000900)
[18:05:44] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 4.1.3
libdvdread: Encrypted DVD support unavailable.
libdvdread: Can't stat UNKNOWN_PARAMETER_VALUE.WMV
No such file or directory
libdvdnav: vm: failed to open/read the DVD
[18:05:44] dvd: not a dvd - trying as a stream/file instead
[18:05:44] hb_stream_open: open UNKNOWN_PARAMETER_VALUE.WMV failed
[18:05:44] scan: unrecognized file type
[18:05:44] libhb: scan thread found 0 valid title(s)
No title found.
HandBrake has exited.
So, we can attest that the script can indeed fire up Handbrake, but the logs indicate that the Handbrake won't recognize the file format and dies on the spot. Again, this doesn't happen if I run the threadedqueue.py script on its own.
I'm guessing that Handbrake is not loading its libraries somehow.
Is this the reason the code won't work? How can I get Handbrake to work?

(darned browser lost my previous answer, hope this does not become a dup)
Both of these errors Can't stat UNKNOWN_PARAMETER_VALUE.WMV and
open UNKNOWN_PARAMETER_VALUE.WMV failed suggest file-not-found errors, BUT it does look like the file name is getting passed through to the handbrake command line.
So in order to be sure that the right files are being used: In the python code convert whatever filenames the user provides to server.py into absolute path names (see os.path.*). Use those absolute path names everywhere (esp. in the handbrake command/Popen)