We Keep Coding

error when running python asyncio socks proxy server in a thread

I am using python3 asyncio_socks_server library to run a socks proxy, my program is supposed to run a socks proxy in a thread and an HTTP server on another thread then continue execution.
Here is my socks_proxy.py:
import socket
from asyncio_socks_server.__version__ import __version__
from asyncio_socks_server.app import SocksServer
class SocksProxy:
def __init__(self, local_ip, port):
args = []
config_args = {
"LISTEN_HOST": local_ip,
"LISTEN_PORT": port,
"AUTH_METHOD": 0,
"ACCESS_LOG": 0,
"DEBUG": 1,
"STRICT": 0,
}
app = SocksServer(
config=None,
env_prefix="AIOSS_",
**{k: v for k, v in config_args.items() if v is not None},
)
app.run()
def startSocksProxy(local_ip, port):
s = SocksProxy(local_ip, port)
main.py:
import threading
from modules.util.socks_proxy import startSocksProxy
from modules.logger.logger import Logger
if "__main__" == __name__:
logger = Logger()
remoteLog = threading.Thread(target=startLoggerServer)
remoteLog.start()
socksProxy = threading.Thread(target=startSocksProxy, args=(LOCAL_IP, 7000))
socksProxy.start()
The HTTP server starts fine, however, I am getting the following error:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "j:\WormScanner\modules\util\socks_proxy.py", line 35, in startSocksProxy
s = SocksProxy(local_ip, port)
File "j:\WormScanner\modules\util\socks_proxy.py", line 26, in __init__
app = SocksServer(
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\site-packages\asyncio_socks_server\app.py", line 23, in __init__
self.loop = asyncio.get_event_loop()
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\asyncio\events.py", line 639, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-2'.
and the socks proxy doesn't start.

To anyone who ever faces this problem, it worked fine when I used multiprocessing:
if "__main__" == __name__:
multiprocessing.freeze_support()
remoteLog = multiprocessing.Process(target=startLoggerServer)
remoteLog.start()
socksProxy = multiprocessing.Process(target=startSocksProxy, args=(LOCAL_IP, 7000))
socksProxy.start()

Asyncio run_until_complete gives RunTimeError

I am trying to run a simple asyncio loop that I want to run continuously until Control + C is pressed, which should stop the main function and run a function to shutdown the loop. I am using the following code:
import asyncio
async def run():
while True:
print("Running")
async def shutdown():
print("Starting to shutdown")
await asyncio.sleep(1)
print("Finished sleeping")
try:
loop = asyncio.get_event_loop()
loop.create_task(run())
loop.run_forever()
except KeyboardInterrupt:
loop.stop()
loop.run_until_complete(asyncio.gather(*[shutdown()]))
When running this code and pressing Control + C the script doesn't shutdown gracefully as I expected (also based on this stackoverflow answer but I get the following error:
Traceback (most recent call last):
File "test.py", line 25, in <module>
loop.run_until_complete(asyncio.gather(*[shutdown()]))
File "/usr/lib/python3.7/asyncio/base_events.py", line 585, in run_until_complete
raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at test.py:8> exception=KeyboardInterrupt()>
Traceback (most recent call last):
File "test.py", line 22, in <module>
loop.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
self._run_once()
File "/usr/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
handle._run()
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "test.py", line 10, in run
print("Running")
KeyboardInterrupt
Can anyone tell me where I am going wrong and how I should change my code to solve the issue?

Run processes in parallel with python on windows asyncio

I have a code that only runs on Linux with asyncio, but I need it to run on Windows, my knowledge is little in multiprocessing, what would be the way to run on Windows? I saw that there is a limitation of asyncio to run on windows: https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess
My code:
import sys
import asyncio
scriptspy = [
'scrp1.py',
'scrp2.py',
'scrp3.py',
'scrp4.py',
'scrp5.py',
'scrp6.py',
'scrp7.py',
'scrp8.py',
'scrp9.py',
'scrp10.py',
'scrp11.py',
'scrp12.py',
]
async def main():
tarefas_rodando = set()
while scriptspy:
# start up to 6 scripts
while len(tarefas_rodando) < 6 and scriptspy:
script = scriptspy.pop()
p = await asyncio.create_subprocess_exec(sys.executable, script)
tarefa = asyncio.create_task(p.wait())
tarefas_rodando.add(tarefa)
# wait for one of the scripts to end
tarefas_rodando, finalizadas = await asyncio.wait(tarefas_rodando, return_when=asyncio.FIRST_COMPLETED)
# finished, wait for the rest to finish
await asyncio.wait(tarefas_rodando, return_when=asyncio.ALL_COMPLETED)
asyncio.run(main())
This code runs fine on linux but not windows. I need to run it on windows. https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess
Exception run in Windows:
Traceback (most recent call last):
File "scripts.py", line 53, in <module>
asyncio.run(main())
File "C:\Python\Python37\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete
return future.result()
File "scripts.py", line 44, in main
p = await asyncio.create_subprocess_exec(sys.executable, script)
File "C:\Python\Python37\lib\asyncio\subprocess.py", line 217, in create_subprocess_exec
stderr=stderr, **kwds)
File "C:\Python\Python37\lib\asyncio\base_events.py", line 1533, in subprocess_exec
bufsize, **kwargs)
File "C:\Python\Python37\lib\asyncio\base_events.py", line 463, in
_make_subprocess_transport
raise NotImplementedError NotImplementedError

According to the documentation, to run subprocesses on Windows, you need to switch to the proactor event loop:
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())

How to use telethon in a thread

I want to run a function in background. so I use Threading in my code.
but return error ValueError: signal only works in main thread and don't know about two things:
what is the main thread
how to solve this problem :)
views.py
def callback(update):
print('I received', update)
def message_poll_start():
try:
client = TelegramClient('phone', api_id, api_hash,
update_workers=1, spawn_read_thread=False)
client.connect()
client.add_update_handler(callback)
client.idle()
except TypeNotFoundError:
pass
def message_poll_start_thread(request):
t = threading.Thread(target=message_poll_start, args=(), kwargs={})
t.setDaemon(True)
t.start()
return HttpResponse("message polling started")
urls.py
urlpatterns = [
path('message_poll_start', messagemanager_views.message_poll_start_thread, name="message_poll_start"),
]
trace
[12/Jan/2018 11:24:38] "GET /messages/message_poll_start HTTP/1.1" 200 23
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/teletogram/telethogram/messagemanager/views.py", line 123, in message_poll_start
client0.idle()
File "/home/teletogram/.env/lib/python3.5/site-packages/telethon/telegram_bare_client.py", line 825, in idle
signal(sig, self._signal_handler)
File "/usr/lib/python3.5/signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

1) A python script runs in the main thread by default. If you spawn a new thread using threading.Thread, that will create a new thread which runs separately from the main one. When I began learning about threading I spent a lot of time reading before it started to click. The official threading docs are decent for basic functionality, and I like this tutorial for a deeper dive.
2) The internals of Telethon rely on asyncio. In asyncio each thread needs its own asynchronous event loop, and thus spawned threads need an explicitly created event loop. Like threading, asyncio is a large topic, some of which is covered in the Telethon docs.
Something like this should work:
import asyncio
def message_poll_start():
try:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
client = TelegramClient('phone', api_id, api_hash, loop=loop)
client.connect()
client.add_update_handler(callback)
client.idle()
except TypeNotFoundError:
pass

Program with flask-socketio and multiprocessing thorws 'LoopExit: This operation would block forever'

first: I am an absolute beginner in python, I used to write PHP before, so if I am getting something complitly wrong please let me know.
I am writing an app. It should serve its information via websockets. I choosed flask-socketio for this. In the background I want to process the data. Because I would like to have the app small I decided against a solution like Celery.
I have shortened the code to:
# -*- coding: utf8 -*-
from flask import Flask, jsonify, abort, make_response, url_for, request, render_template
from flask.ext.socketio import SocketIO, emit
from multiprocessing import Pool
from multiprocessing.managers import BaseManager
import time
import os
def background_stuff(args):
while True:
try:
print args
time.sleep(1)
except Exception as e:
return e
thread = None
_pool = None
app = Flask(__name__)
app.debug = True
socketio = SocketIO(app)
#app.route('/', methods=['GET'])
def get_timers():
return 'timer'
if __name__=='__main__':
_pool = Pool(1)
if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
workers = _pool.apply_async(
func=background_stuff,
args=('do background stuff',),
)
socketio.run(app)
# app.run()
When starting this, i get the following messages:
python test/multitest.py
* Running on http://127.0.0.1:5000/
* Restarting with stat
do background stuff
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 336, in _handle_tasks
for taskseq, set_length in iter(taskqueue.get, None):
File "/usr/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File "/usr/lib/python2.7/threading.py", line 340, in wait
waiter.acquire()
File "gevent/_semaphore.pyx", line 112, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:3386)
File "/home/phil/work/ttimer/server/local/lib/python2.7/site-packages/gevent/hub.py", line 338, in switch
return greenlet.switch(self)
LoopExit: This operation would block forever
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
127.0.0.1 - - [2015-09-30 00:06:23] "GET / HTTP/1.1" 200 120 0.001860
do background stuff
do background stuff
do background stuff
do background stuff
^CProcess PoolWorker-1:
Process PoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
result = (True, func(*args, **kwds))
File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
File "test/multitest.py", line 14, in background_stuff
KeyboardInterrupt
time.sleep(1)
KeyboardInterrupt
return recv()
KeyboardInterrupt
So the background process is working and it answers http requests (127.0.0.1 - - [2015-09-30 00:06:23] "GET / HTTP/1.1" 200 120 0.001860). But just ignoring an error because it seems to work does not seem to be a solution for me. Does anyone can tell my what I am doing wrong here?
If you say I can't do it that way can you tell me why? I would like to learn and understand what I am doing wrong.
I read something about monkepatching, but everything suggested threw just more or other errors. I think it is better to work on the first error instead of blindly trying fixes.
python -V
Python 2.7.9
Greetings
update
I added the 2 lines for monkeypatching, this is what I got:
$python multitest2.py
^CProcess PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
return recv()
KeyboardInterrupt
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 380, in _handle_results
task = get()
KeyboardInterrupt
* Running on http://127.0.0.1:5000/
* Restarting with stat
^CProcess PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 380, in _handle_results
task = get()
KeyboardInterrupt
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
return recv()
KeyboardInterrupt
do background stuff
FAILED to start flash policy server: [Errno 98] Address already in use: ('127.0.0.1', 10843)
$do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
do background stuff
On start there is no output at all. After hittinc ctrl-c several times, i get the background stuff output. This continues until i kill the python process via SIGKILL
update 2
what I expect to see is
* Running on http://127.0.0.1:5000/
* Restarting with stat
do background stuff
do background stuff
do background stuff
right after the running of the script. But before I press ctrl-c nothing is happening.

First of all, you need to be aware that the version of Flask-SocketIO that you are using requires gevent, which is a coroutine framework. Using the asynchronous coroutines of gevent with a multiprocessing pool is a strange combination. You are using gevent, so what would make the most sense is to use the gevent pool functionality so that everything is consistent.
Now regarding the problem, I think it is likely due to not having the standard library monkey patched at an early stage. I recommend that you add the following lines at the very top of your script (above your imports, make them lines 1 and 2):
from gevent import monkey
monkey.patch_all()
These will ensure that any calls into standard library for things such as threads, semaphores, etc. go to the gevent implementations.
Update: I tried your example. The original version, without monkey-patching, works fine for me, I do not see the LoopExit error that you reported. Adding the monkey patching prevents the background processes from running, as you reported.
In any case, I converted your script to use gevent.pool and that works reliably for me. Here is the edited script:
from flask import Flask, jsonify, abort, make_response, url_for, request, render_template
from flask.ext.socketio import SocketIO, emit
from gevent.pool import Pool
import time
import os
def background_stuff(args):
while True:
try:
print args
time.sleep(1)
except Exception as e:
return e
thread = None
_pool = None
app = Flask(__name__)
app.debug = True
socketio = SocketIO(app)
#app.route('/', methods=['GET'])
def get_timers():
return 'timer'
if __name__=='__main__':
_pool = Pool(1)
workers = _pool.apply_async(
func=background_stuff,
args=('do background stuff',),
)
socketio.run(app)
Hope this helps!

I read a tutorial about gevent and fount a solution which is simple and clean for my needs:
# -*- coding: utf8 -*-
from flask import Flask
from flask.ext.socketio import SocketIO
import gevent
import os
def background_stuff():
while True:
try:
print 'doing background work ... '
gevent.sleep(1)
except Exception as e:
return e
app = Flask(__name__)
app.debug = True
socketio = SocketIO(app)
#app.route('/', methods=['GET'])
def get_timers():
return 'timer'
if __name__=='__main__':
if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
gevent.spawn(background_stuff)
socketio.run(app)
The tutorial can be found here: http://sdiehl.github.io/gevent-tutorial/#long-polling
It even tells about problems with gevent and multiprocesing: http://sdiehl.github.io/gevent-tutorial/#subprocess , but because I found a simple solution fitting to my needs I didn't try anything else.