We Keep Coding

`RuntimeError: Timeout context manager should be used inside a task` in flask api application

I'm working on exposing the results of a web scraper through a REST API running on a flask server.
Without going into too much detail, I have the following 2 files that call the same function that scrapes data from a site and returns it as json :
Working example
# testing_scraper_functions.py
scraper = Scraper(loginparams)
def run_task(task):
try:
loop = asyncio.get_event_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
return loop.run_until_complete(task())
# Simulating api call response
def searchChild(eiNumber):
return run_task(lambda: scraper.searchChild(eiNumber=eiNumber))
childSearchData = searchChild("776105")
scraper.quit()
Broken flask server example
app = Flask(__name__)
port = 5100
if sys.argv.__len__() > 1:
port = sys.argv[1]
scraper = Scraper(loginparams)
def run_task(task):
try:
loop = asyncio.get_event_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
return loop.run_until_complete(task())
#app.route('/api/search_ei/<eiNumber>')
def searchChild(eiNumber):
return run_task(lambda: hardWorker.searchChild(eiNumber=eiNumber))
if __name__ == '__main__':
app.run(host="0.0.0.0", port=port, debug=True)
scraper.quit()
They look like they should work the same but when I do the network request to the API I get the following Runtime Error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flask/app.py", line 2095, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flask/app.py", line 2080, in wsgi_app
response = self.handle_exception(e)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flask/app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flask/app.py", line 1525, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flask/app.py", line 1523, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flask/app.py", line 1509, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/Webapp/server.py", line 56, in searchChild
return run_task(lambda: hardWorker.searchChild(eiNumber=eiNumber))
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/Webapp/server.py", line 39, in run_task
return loop.run_until_complete(task())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/NYEIS_Scraper/NYEIS_API_V2/NYEIS_Worker.py", line 47, in wrapper
return await func(self, *args, **kwargs)
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/NYEIS_Scraper/NYEIS_API_V2/NYEIS_Worker.py", line 95, in searchChild
return (await self.searchChilds(eiNumber=eiNumber,
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/NYEIS_Scraper/NYEIS_API_V2/NYEIS_Worker.py", line 70, in searchChilds
html = await self._htmlFetcher.searchChild(eiNumber=eiNumber)
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/NYEIS_Scraper/NYEIS_API_V2/HTML_Fetcher.py", line 108, in searchChild
return await (await self._sessionHandler.fetch(method="post", url=url, payload=payload)).text()
File "/Users/c4q/Desktop/Pycharm Projects/NYEISNYEISBABY/NYEIS_Scraper/NYEIS_API_V2/SessionManager.py", line 56, in fetch
response = await self._session.post(url=url, headers=self.headers, data=payload)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/client.py", line 466, in _request
with timer:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/helpers.py", line 701, in __enter__
raise RuntimeError(
RuntimeError: Timeout context manager should be used inside a task
In the error, NYEIS_worker is the Scraper
This scraping function is from an object that has a reference to an aiohttp ClientSession(), which is created when I initialize the object.
The only thing I can even see as a difference is that in the working example I call scraper.quit() which closes the ClientSession(), but I do that in the serverapp too, just after closing the server. Is the issue that the client session is open too long in the server example?
Should I create a scraper object and close it on every api request? I feel like this would be a lot slower. Should I reopen the clientsession() on every api call but keep the same scraper object? I'm not sure how to open a session again after closing, I would just be making a new one.
I'm not even sure what a Timeout context manager is. Can I use one without using async with? I've written everything using just await and running ClientSession().close() when I'm finished.
Any guidance here is appreciated.
I've kept most of the code out to keep my question shorter, but I can add more snippets if needed

Using requests_html in Google Cloud Functions or Run

I have a fairly basic scraping application that i want to run in a Google Cloud environment, i am using the requests_html Async library and it works fine in my local environment, however i cannot for the life of me figure out how to run it in Google Cloud having fiddled with it for days now.
The purpose of the application is to simply render some javascript pages (contained in the urls array) using html.arender and then with BeautifulSoup extract the content of some specific tags (from the tags array).
The error message i keep getting is:
"signal only works in main thread of the main interpreter"
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 99, in view_func
return function(request._get_current_object())
File "/workspace/main.py", line 53, in main
results = asyncio.run(collect(urls,tags))
File "/opt/python3.9/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/python3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/workspace/main.py", line 32, in collect
return await asyncio.gather(*tasks)
File "/workspace/main.py", line 18, in getPage
await r.html.arender(timeout=40,sleep=1)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests_html.py", line 615, in arender
self.browser = await self.session.browser
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/requests_html.py", line 714, in browser
self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pyppeteer/launcher.py", line 307, in launch
return await Launcher(options, **kwargs).launch()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pyppeteer/launcher.py", line 159, in launch
signal.signal(signal.SIGINT, _close_process)
File "/opt/python3.9/lib/python3.9/signal.py", line 56, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter
This is my code below:
from requests_html import AsyncHTMLSession
import asyncio
from bs4 import BeautifulSoup as bs
urls = ['list of urls']
tags = ['p','h1']
async def getPage(s, url):
r = await s.get(url)
await r.html.arender(timeout=60,sleep=3, scrolldown=2)
p = bs(r.html.html, "html.parser")
elmList = []
elmList.append(url)
for t in tags:
elements = p.findAll(t)
for e in elements:
elmList.append(e.text)
return elmList
async def collect(urls):
s = AsyncHTMLSession()
tasks = (getPage(s,url) for url in urls)
return await asyncio.gather(*tasks)
results = asyncio.run(collect(urls))
I have also attempted this same thing using the "non async" HTMLSession and rewriting the code to do one URL at a time, but i get the exact same error message in this case with the "signal only works in the main thread".
I have also tried to run this both in Cloud Functions and Cloud Run environment, with the same results.
Additionally, after trawling forums for advice have experimented with manually setting the loop like so, but this had no effect.
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
If anyone has ideas for how to accomplish this with other libraries/methods please do let me know, it does not necessarily even have to be asynchronously, only requirement is javascript rendering of the pages.

Have you tried keep_page=True as a parameter for r.html.arender?
Why I'm asking: The error seems to happen when the browser that is used for rendering the JS is being shut down. Maybe keep_page=True avoids this.

flask: RunTimeError: There is no current event loop in thread

I am making a simple flask app that returns results from an external website. The user enters data into my site. This data is used to load another site. Data is extracted and returned in a list. This program works independently, but will not work as part of the flask app. I have tried using using requests_HTML library using it's async tools, and i have tried using Pyro4, to send the request to an external process. But i always come up with some version of this error:
RunTimeError: There is no current event loop in thread ........
It seems to me that when one of my imported modules runs a thread Flask doesn't like it.
I would love to know why does this happen, is their something in flasks inner workings that means that it won't work well with threads or async or something?
Any direction to some extra resources or info would be very appreciated
This is my Flask App:
from flask import Flask, render_template
from requests_html import AsyncHTMLSession
app = Flask(__name__)
asession = AsyncHTMLSession()
URLS=["https://stackoverflow.com/questions/",
"https://stackoverflow.com/questions/",
"https://stackoverflow.com/questions/"]
#app.route("/", methods=('GET'))
def index():
results = asession.run(run_requests)
return render_template('index_page.html', results_list=results)
async def run_requests():
results_list=[]
for url in URLS:
results_list.append(await get_and_render(url))
return results_list
async def get_and_render(url):
r = await asession.get(url)
await r.html.arender(sleep=0.75)
summary = r.html.find(".question-hyperlink", first=True)
return summary.text
This will get the stackoverflow questions page and grab the summary of the last posted question. I have tried this code in a standalone python file and it works fine. (ie outside of a flask app, just printing the results to the command line)
This is by traceback from the flask debugger:
'Traceback (most recent call last):
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\flask\app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\......\StackOverflowExample.py", line 11, in index
results = asession.run(run_requests)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\requests_html.py", line 771, in run
tasks = [
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\requests_html.py", line 772, in <listcomp>
asyncio.ensure_future(coro()) for coro in coros
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\asyncio\tasks.py", line 660, in ensure_future
loop = events.get_event_loop()
File "C:\Users\Alex\AppData\Local\Programs\Python\Python38-32\Lib\asyncio\events.py", line 639, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-4'.
Can anyone explain why flask has trouble completing these tasks?
cheers
PS
here is my simple HTML (although app has not been able to render this yet.):
<ul>
{% for result in results_list %}
<li>{{result}}</li>
{% endfor %}
</ul>
Here is my Script outside the flask app(it works):
from requests_html import AsyncHTMLSession
asession = AsyncHTMLSession()
URLS=["https://stackoverflow.com/questions/",
"https://stackoverflow.com/questions/",
"https://stackoverflow.com/questions/"]
def index():
results = asession.run(run_requests)
for result in results:
print(result)
async def run_requests():
results_list=[]
for url in URLS:
results_list.append(await get_and_render(url))
return results_list
async def get_and_render(url):
r = await asession.get(url)
await r.html.arender(sleep=0.75)
summary = r.html.find(".question-hyperlink", first=True)
print(summary.text)
if __name__ == "__main__":
index()

WinError 10057 when connecting using websocket-client

I'm making a Flask webapp and I'm using Flask-Socketio. For various reasons, I have a need to also use the websocket-client package. Everything is working as intended, except that when I tried running the app on a different computer on a different network, I get the following error:
"""
Traceback (most recent call last):
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "[Path to venv]\venv\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "[Path to venv]\venv\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "[Path to venv]\venv\lib\site-packages\flask\app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "[Path to app]\app\views.py", line 7, in index
sio.connect("http://localhost:80/", transports=['websocket', 'polling'])
File "[Path to venv]\venv\lib\site-packages\socketio\client.py", line 262, in connect
engineio_path=socketio_path)
File "[Path to venv]\venv\lib\site-packages\engineio\client.py", line 170, in connect
url, headers, engineio_path)
File "[Path to venv]\venv\lib\site-packages\engineio\client.py", line 346, in _connect_websocket
cookie=cookies)
File "[Path to venv]\venv\lib\site-packages\websocket\_core.py", line 514, in create_connection
websock.connect(url, **options)
File "[Path to venv]\venv\lib\site-packages\websocket\_core.py", line 223, in connect
options.pop('socket', None))
File "[Path to venv]\venv\lib\site-packages\websocket\_http.py", line 120, in connect
sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
File "[Path to venv]\venv\lib\site-packages\websocket\_http.py", line 189, in _open_socket
raise error
File "[Path to venv]\venv\lib\site-packages\websocket\_http.py", line 172, in _open_socket
sock.connect(address)
OSError: [WinError 10057] A request to send or receive data was disallowed because
the socket is not connected and (when sending on a datagram socket using a sendto call)
no address was supplied
"""
I've boiled down my code as much as possible to the following, which still works on my computer, but gives the same error on the other:
|start.py
|app
|__init__.py
|views.py
|templates
|index.html
# __init__.py
from flask import Flask
from flask_socketio import SocketIO
from gevent import monkey
monkey.patch_all()
APP = Flask(__name__)
SOCKETIO = SocketIO(APP, engineio_logger=True, logger=True)
from . import views
# views.py
from app import APP
from socketio import Client
from flask import render_template
#APP.route('/', methods=['GET', 'POST'])
def index():
sio = Client()
sio.connect("http://localhost:80", transports=['websocket', 'polling']) # Error being caused here
return render_template('index.html')
# start.py
from app import APP, SOCKETIO
if __name__ == "__main__":
SOCKETIO.run(APP, debug=True, port=80, host='0.0.0.0')
index.html is just a basic 'Hello World' html page.
What kind of stuff might give me this error on one computer/network an not another, especially just running it on localhost:80? I really don't know what to try here.
EDIT: Added traceback data to error
EDIT 2: In my actual code, the websocket.Client is being run inside a Celery task. I didn't include it here because the error is reproduceable without going into that much complexity.

The problem is when you try to run -(Flask-webapp, websocket-client)- both of them simultaneously, only one of them will be running.
Update:
Here are some things to be noted:
If you changing machine, make sure system-softwares are already up-to-date.
Check your system is not behind firewall but when you're running Flask webapp and websocket-client on the same machine then there is not need of this step.
Try to restart your machine after update your OS.
Use multiprocessing instead of Threads as they won't work here because the famous Python-Global-Interpreter or GIL. According to which Only one thread can be in a state of execution at any point in time.
The impact of the GIL isn’t visible to developers who execute
single-threaded programs, but it can be a - Performance-bottleneck - in
CPU-bound and multi-threaded code.
Then Check: How to run flask on Port:80. For development purposes Port:5000 is mostly recommended.
In order to perform "Error Handling" visit the flask-socketio documentation, there is nice tips to to deal with exceptions.
You can go as follow:
#socketio.on_error_default # handles all namespaces without an explicit error handler
def default_error_handler(e):
pass
And finally, to handle socket.timeout, you may also use this following code to handle the final error: socket timed out:
try:
socketio.run(app,...
except socket.error as socketerror:
print("Error: ", socketerror)
It would be an better approach to use postman.For learning and testing purpose, try to use POSTMAN which is a quick and easy platform for sending REST, SOAP, and GraphQL requests directly within Postman. Visit reference: getting-started-postman.
Here is an example program:
import multiprocessing
# Assume for flask-app
def start_flask_app:
app.run()
# Assume for websocket-client-app
# Assume basic events you will handle in your client e.g. on_message, on_error, on_close
def start_ws:
ws = websocket.WebSocketApp(WS_URI, on_message= on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
# use of sub-processes instead of threads.
p_flask = multiprocessing.Process(target=start_flask_app)
p_ws = multiprocessing.Process(target=start_ws)
p_ws.start()
p_flask.start()
p_ws.join()
p_flask.join()

NameError: name 'user_id' is not defined. For no apparent reason [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
Error description
I would fire up the server program, and fire a request (GET) at the path /test_connection which is the path used by the client to test the server IP address btw. The server would then respond by printing out the traceback below in the console, the one line in particular that I can't get my head around is NameError: name 'user_id' is not defined as the variable is not even included in the response function for the path /test_connection (connection_test()). I know the error is on the server side as it will print out 'INTERNAL SERVER ERROR: 500' on the client code, so I didn't feel the necessity to include the client code.
Additional info:
Python version: 3.7
OS: Windows 10
Server Library: Flask
Traceback from error:
Traceback (most recent call last):
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\sccre\AppData\Local\Programs\Python\Python37-32\lib\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "E:\USB Stick Files\Programming\Python\Chat Room Thingy\server.py", line 19, in connection_test
message_dict[user_id]
NameError: name 'user_id' is not defined
Server code
from flask import Flask, request
from threading import Thread
import json
app = Flask(__name__)
message_dict = {}
ids = -1
#app.route('/test_connection')
def connection_test():
global ids
print('Connection tested from: {}'.format(request.remote_addr))
ids += 1
id_sent = str(ids)
return '{}'.format(id_sent)
#app.route('/new_message', methods=['POST'])
def new_message():
global message_dict
message_text = request.form['message']
user_id = request.form['id']
try:
message_dict[user_id]['message'] = message_text
except:
message_dict[user_id] = None
message_dict[user_id]['message'] = message_text
return '0'
#app.route('/chat')
def chat():
return json.dumps(message_dict)
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
Any help with this would be greatly appreciated.
Oscar.

Okay, The code is working completely fine to me.
Try one of the following :
1.Change your current File ( copy the contents to another )
2.Make the variable global.
3.Run the file directly from command prompt/shell (sometimes helpful)
4.Make sure your interpreter is ok