I am writing an application using pycurl, and need to make it work in Twisted. I've been searching for either making pycurl somehow compatible with Twisted framework, or using an existing Twisted library. I am suggested Twisted web, but there is no direct map of functions from pycurl to Twisted web. Can anyone point me to the right direction?
Edit: One solution is run pycurl in another thread, but preferably, I wish to use Twisted framework or pycurl that is nonblocking, so I don't have to create another thread.
For any blocking function, if there's no other asynchronous alternative, Twisted lets you run it on another thread but treat it as a Deferred.
from twisted.internet import threads
d = threads.deferToThread(pycurl.some_function)
d.addCallback(callback)
See Integrating blocking code with Twisted in Generatin Deferreds.
Have you considered using Twisted web.client.Agent? It is very basic agent, but integrates very well with the Twisted event loop.
Related
I would like to load-test a SignalR service using Locust. I found that the following library can send and receive SignalR requests: https://pypi.org/project/signalrcore/
Now, according to the Locust docs, the next step would be to write a custom client for Locust that can send SignalR requests. But there is the following warning:
Any protocol libraries that you use must be gevent-friendly (use the
Python socket module or some other standard library function like
subprocess), or your calls are likely to block the whole Locust/Python
process.
Some C libraries cannot be monkey patched by gevent, but allow for
other workarounds. For example, if you want to use psycopg2 to
performance test PostgreSQL, you can use psycogreen
I am a beginner in Python so I don't understand exactly what it means. The library "signalrcore" I am using is 100% synchronous. Does it means I can't use it with Locust?
I found an a fork of signalrcore that uses asyncio. Should I use that fork instead and just make sure all my signalr calls are non blocking?
Thanks!
SignalRCore seems to use requests and websocket-client under the hood, both of which are gevent-friendly. I cant say for sure, but I’d give it 90% probability that it will work ”out of the box” :)
If you do use the asyncio one you’d need to do some magic yourself. At least I have never combined that with gevent.
I have a running CLI application in Python that uses threads to execute some workers. Now I am writing a GUI using electron for this application. For simple requests/responses I am using gRPC to communicate between the Python application and the GUI.
I am, however, struggling to find a proper publishing mechanism to push data to the GUI: gRPCs integrated streaming won't work since it uses generators; as already mentioned my longer, blocking tasks are executed using threads (subclasses of threading.Thread). Also I'd like to emit certain events (e.g., the progress) from within those threads.
Then I've found the Flasks SocketIO implementation, which is, however, a blocking execution and thus not really suited for what I have in mind - I'd have to again execute two processes (Flask and my CLI application)...
Another package I've found is websockets but I can't get my head around how I could implement this producer() function that they mention in the patterns.
My last idea would be to deploy a broker-based message system like Redis or simply fall back to the brokerless zmq, which is a bit of a hassle to setup for the GUI application.
So the simple question:
Is there any easy framework that allows to create a server-"task" in a Python that I can pass messages to publish to?
For anyone struggling with concurrency in python:
No, there isn't any simple framework. IMHO pythons' concurrency handling is a bit of a mess (compared to other languages like golang, where concurrency is built in). There's multiple major packages implementing this, one of them asyncio, but most of them are incompatible. I've ended up using a similar solution like proposed in this question.
Is there any way to make DNS resolving using Python with asyncore?
I can not install adns and I do not like to use gevent library.
(for URL downloading gevent give me slow performance than syncore.loop)
I asked google your question, and it told me about http://subvert-rpki.hactrn.net/trunk/rpkid/rpki/adns.py which seems to be a DNS client implementation based on asyncore. So, there you have it, there is a way.
I'm trying to make a cherrypy application with a wxpython ui. The problem is both libraries use closed loop event handlers. Is there a way for this to work? If I have the wx ui start cherrypy is that going to lock up the ui?
See my answer at CherryPy interferes with Twisted shutting down on Windows
In short, CherryPy handles the main loop by default, but it definitely doesn't need to. Stop using quickstart and call engine.start without engine.block, and CP will run in its own threads and leave the main thread for your other framework to control.
If you use threading, you should be able to start up the CherryPy server in one thread and run wxPython in the other. This article (http://wiki.wxpython.org/LongRunningTasks) on the wxPython wiki has some info on threading, and the CherryPy server source code (http://www.cherrypy.org/browser/trunk/cherrypy/wsgiserver/__init__.py) has some documentation on how the server works, and possibly how you could get it to interact with threads.
One way to decouple them would be to start them up as two separate processes and have them communicate via some kind of IPC mechanism. You might have to write a small adaptor to have them speak a common protocol.
Since you're doing CherryPy, you might also be able to expose a control interface via HTTP which the wx GUI can use to drive your server.
I would encourage you to take a look at the Calibre (e-book manager) source. It is written in PyQT, but uses CherryPy to allow people to view their library from outside their LAN.
I have a simple pygtk app using urllib2, what changes should I make to add working twisted code?
The pbgtk2.py example it's confusing
You switch from using the gtk mainloop to the right Twisted reactor. Or you decide to run Twisted in a separate thread using reactor.run(installSignalHandlers=0), and stay with the gtk mainloop.
You decide if you want to defer the urllib2 call to its own thread, or if you want to rewrite that code using Twisted's HTTP client libraries.
You go to the Twisted mailing list or IRC channel and ask for help.