I am currently trying to create a throttler decorator to limit my api calls accross all my app.
My idea is to use threading.semaphore and/or threading.timer to track and block the request which are above the rate limit.
I already have something working, however my concern is that with this solution i will start a thread per request. So lets say i'm making 5000 calls per minute, could this cause any issue / is bad practice ?
Good practice is to use something that already exists in the framework you are using. There are ready-made solutions for Django and DRF, for example. Something similar is bound to exist for the framework you are using. If you are developing a web api, that is.
Related
I have a blog and an application that gives the number of comments and posts on my blog by using my blog's API.
The issue I'm having is that I want to have my application receive new comments from my application in real-time.
My solution:
I can have my application calling the API every 30 seconds or so to check whether there is a response (i.e. whether there is a new comment).
I think the best solution is to use something called Long Polling to get updates. Its a technique in programming to handle requests with less resources such as CPU being used over time. For a detailed solution for your case search for
Long Polling in Flask application
I wanted to ask you about microservices in Python. As of writing this, i got pretty good with writing well structured flask-restful APIs and I wanted to go and learn about microservices in python.
Right now I have read up quite a lot of info regarding this and even searched online to find examples for this (1 example here) but I am not really sure exactly where to start as I don't want to invest too much time in a inefficient pattern.
So I wanted to know if anyone know any courses with examples for Python on building high load services. My only hints so far as asyncio and aiohttp for request handling and i'm not sure if using a message broker (such as zeromq or rabbitmq) would be a good idea as from what I read, it adds request lag.
Any advice would be great.
PS: The current pattern I'm stuck on is the API Gateway pattern and I would also want to know if it is a good direction as a start.
There are plenty of microservice frameworks in Python that can handle high load and get you a long way towards following best practices.
Try for example pymacaron (http://pymacaron.com/). Pymacaron is basically a flask app whose endpoints are auto-spawn from a swagger specification. To write a pymacaron microservice, you mostly have to:
(1) write a swagger specification for your api (which is always a good starting point, whatever language you are using). Your swagger file describes the get/post/etc calls of your api and which objects (json dicts) they get and return, but also which python method in your code that implement the endpoint.
(2) and implement your endpoints' methods.
Once you have done that, you get loads of things for free: you can package your code as a docker container, deploy it to amazon beanstalk, start asynchronous tasks from within your api calls, or get the api documentation with no extra work.
Here is an example of an helloworld api implemented with pymacaron: https://github.com/pymacaron/pymacaron-helloworld
I have a small Django app that is highly dependent on other services. In one situation, the client makes one request to our app, and the Django app makes 3 or 4 requests to another server, aggregates the data, and saves it. This is a slow process as I need to make these calls synchronously although there is no requirement for the requests to be made sequentially.
Given the above, this particular set of calls seem like a good candidate for async programming and the asyncio package is provided with newer Python versions.
I’ve implemented what I have described using loop.run_until_complete and hooked it into a view and it seems to work fine, as well as provide a significant performance increase.
However, I am worried that I am missing something in the magic. Doing multiple google searches on whether this is a good idea in Django has given me a mixed bag of answers and I’m still unclear on the negatives of doing so.
I realize this could all be done by JavaScript client in a well supported manner, I’m just trying to reduce complexity on that side, and am more interested in the implications of doing this running Django in prod environment with WSGI.
Here's the setup: On a single-board computer with a very rudimentary linux I'm running a Django app. This app is, when a button is pressed or as a response to the data described below, supposed to call either a function from a library written in C, or a compiled C program, to write data to system memory at a specified address, poke/peek like. (Python doesn't seem to be able to do that natively).
The Django app should also display data, continuously, which is being read from the memory from the same library / program.
My question now is how to even begin with setting up the scenario described above. Is this even possible with a web app? Is a Django or more fundamentally any web framework even the right approach here? I'm at a bit of a loss here, since I've spent quite a few hours now trying to figure out how to do this while not getting the most basic starting point...
Disclaimer: I'm pretty new to the entire web framework thing, and more importantly web development in general, so sorry if this is a bad question as in, I could have easily found information on this topic online, but I couldn't really find a good starting point on this.
I wanted to add a comment but not enough space... anyway
You can write a native extension in C for Python that could do what you need, check this.
Now for the fact of displaying data continuously this is kind of vague, if this C library is switching this hypothetical address, very often and very fast you have to update a browser client as fast as possible.
I think websockets would do the trick but they are js related, so I think NodeJs would be a better candidate for the server side of your application instead of Django.
If you want to stick to Django you can also expose an URL with the generated address value and have a webpage continuously (with a little Interval) checking that URL using a simple ajax call, kind of ugly and inefficient but would work.
Anyway IMHO your best bet is for websockets because with them you have a fullduplex communication between client and server.
Good Luck with your project.
Info:
Websockets in Django with socket.io
Nodejs socket.io
I have a Python-driven web interface powered by Apache 2.2 with mod_python and Python 2.4. I need to make an asynchronous process appear synchronous to users of this web interface.
When users access one module on this website:
An external SOAP interface will be contacted with a unique identifier and will respond with a number N
The external interface will respond asynchronously by contacting a SOAP server on my machine between 1 and 10 times (the number N tells us how many responses we will receive)
I need to somehow aggregate these responses and pass them to the original module which will display the information back to the user. The goal is to make the process appear synchronous to the user.
What is the best way to handle this synchronization issue? Is this something Twisted would be well-suited for?
I am not restricting myself to Python for the solution, though it is preferred because everything else on the server is in Python. I prefer a solution that is both scalable and will take a minimal amount of programming time (though I understand that these attributes are somewhat at odds).
Maybe you can use Orbited to get ajax push with long-lived HTTP connections to your web clients. Orbited is based on Twisted, so I think it makes sense to look at if you already know Twisted. Have a look at this tutorial to get started.