We Keep Coding

API Rate limiting Python - Design

I am currently trying to create a throttler decorator to limit my api calls accross all my app.
My idea is to use threading.semaphore and/or threading.timer to track and block the request which are above the rate limit.
I already have something working, however my concern is that with this solution i will start a thread per request. So lets say i'm making 5000 calls per minute, could this cause any issue / is bad practice ?

Good practice is to use something that already exists in the framework you are using. There are ready-made solutions for Django and DRF, for example. Something similar is bound to exist for the framework you are using. If you are developing a web api, that is.

What's the best way to get continuous data from another program in Django?

Here's the setup: On a single-board computer with a very rudimentary linux I'm running a Django app. This app is, when a button is pressed or as a response to the data described below, supposed to call either a function from a library written in C, or a compiled C program, to write data to system memory at a specified address, poke/peek like. (Python doesn't seem to be able to do that natively).
The Django app should also display data, continuously, which is being read from the memory from the same library / program.
My question now is how to even begin with setting up the scenario described above. Is this even possible with a web app? Is a Django or more fundamentally any web framework even the right approach here? I'm at a bit of a loss here, since I've spent quite a few hours now trying to figure out how to do this while not getting the most basic starting point...
Disclaimer: I'm pretty new to the entire web framework thing, and more importantly web development in general, so sorry if this is a bad question as in, I could have easily found information on this topic online, but I couldn't really find a good starting point on this.

I wanted to add a comment but not enough space... anyway
You can write a native extension in C for Python that could do what you need, check this.
Now for the fact of displaying data continuously this is kind of vague, if this C library is switching this hypothetical address, very often and very fast you have to update a browser client as fast as possible.
I think websockets would do the trick but they are js related, so I think NodeJs would be a better candidate for the server side of your application instead of Django.
If you want to stick to Django you can also expose an URL with the generated address value and have a webpage continuously (with a little Interval) checking that URL using a simple ajax call, kind of ugly and inefficient but would work.
Anyway IMHO your best bet is for websockets because with them you have a fullduplex communication between client and server.
Good Luck with your project.
Info:
Websockets in Django with socket.io
Nodejs socket.io

reload image/page after computation is complete from the server side

I have a python code that performs some fairly intense computations, and then generates a plot (png file) for display on the web. I'm using python, flask, mod_wsgi, and apache. Since the computation takes several seconds (around 10 seconds), I'd like to display a "LOADING" type image while the computation is happening so that the viewer doesn't think the server is messed up, and then the actual image when computations are complete. How can I do this from the server side (not using javascript in the web browser)? In the past I remember seeing a lot of web pages where it seemed like the server was pushing a new page to the browser (from what a recall most it was search engines on message forums). The answer to this question I believe is really an http related question, so it doesn't necessarily have to be specific to serving an image (it could be an html page), or using python, flask, mod_wsgi, or apache, but it would be great if an example could be giving for just that configuration.

Before Javascript I did implement this by generating a page that had a refresh in the HTML header, with a delay of 2-3 seconds.
This page would redisplay itself until the code generating that page noticed that the 'result' was finished then generating different HTML code (without the refresh).
<HEAD>
<META http-equiv="refresh" content="3">
</HEAD>

I'm aware that this question is a bit old now, but I feel like there is not enough information available on this topic. I was struggling with this myself to find a relevant information, so I hope this will help:
Suggested solution
There are different technologies that can be used here, but I believe the simplest would be Server Sent Events. The implementation for Flask can be found here. The last part of the documentation is really important:
Subscribers will connect and block for a long time, so you should seriously consider running under an asynchronous WSGI server, such as gunicorn+gevent
So make sure fulfil the requirement. Also, it's pretty important to understand that this approach is good if you want to send the messages from your server to the client. In case you have an external worker that does the calculations for you this method will only make it more complicated for you, since your server will have to play the role of a middle man between the browser and the worker machine. On some hostings it may even not work as expected (e.g. Heroku - still not sure why it misbehaves, looks like too many updates from the worker and are not propagated properly to the client). In case you use the same host for your app and the workers, you should have no problem though.
Alternate solution
In my opinion this type of calculations belong to background, so this solution assumes that we have some kind of workers doing the job for you (like I had when I first encountered the problem). Note that this solution is not a server->client communication, but it's based on polling. I think this may be the only option if you don't run on the asynchronous server in production.
So let's assume you have a worker which status you can check, for example Iron Worker. The user visits your page and this invokes the calculations on the worker. From this point on you should use AJAX calls to get the status update directly from your worker. What I did in my app, I used jQuery to poll the worker web api and learn about it's status. After you discover that your worker is done, you can just reload the page or just the image or whatever else you need.
Additional information
If you need to update many places at the same time (not only the browser), you can use queue services, for example ironMQ, which allows you to propagate your messages to a special queue, and then subscribe to this queue with a client and receive the messages from it. This is what I did before I discovered I can query the worker directly for it's status.

Recommendation for click/event tracking mechanisms (python, django, celery, mongo etc)

I'm looking into way to track events in a django application (events would generally be clicks tied to a specific unique user id).
These events would essentially contain an event type like "click" and then each click event would be assigned to a unique id (many events can go to one id) and each event would have a data set including items like referrer etc...
I have tried mixpanel, but for now the data api they are offering seems too limiting as I can't seem to find a way to get all of my data out by a unique id (apart from the event itself).
I'm looking into using django-eventracker, but curious about any others thought on the best way to do this. Mongo or CouchDb seem like a great choice here, but the celery/rabbitmq looks really attractive with mongo. Pumping these events into the existing applications db seems limiting at this point.
Anyways, this is just a thread to see what others thoughts are on this and how they have implemented something like this...
shoot

I am not familiar with the pre-packaged solutions you mention. Were I to design this from scratch, I'd have a simple JS collecting info on clicks and posting it back to the server via Ajax (using whatever JS framework you're already using), and on the server side I'd simply append that info to a log file for later "offline" processing -- so that would be independent of django or other server-side framework, essentially.
Appending to a log file is a very light-weight action, while DBs for web use are generally way optimized for read-intensive (not write-intensive) operation, so I agree with you that force fitting that info (as it trickes in) into the existing app's DB is unlikely to offer good performance.

You probably want to keep a flexible format for your logs to anticipate future needs or changes. In this sense, the schema-less document-oriented databases are nice. One advantage is that the structure of your data will be close to your application needs for whatever analyses you perform later (so, avoiding some of the inevitable parsing/data munging work).
If you're thinking about using mysql, postgresql or such, then you should look into something like rsyslog for buffering writes and avoiding the performance penalty with heavy logging. (I can't say much about celery and other queueing mechanisms for this type of thing, but they sound promising.)
Mongodb has a some nice features that make it amenable to logging such as capped collections. A summary can be found in this post.

If by click, you mean a click on a link that loads a new page (or performs an AJAX request), then what you aim to do is fairly straightforward. Web servers tend to keep plain-text logs about requests - with information about the user, time/date, referrer, the page requested, etc. You could examine these logs and mine the statistics you need.
On the other hand, if you have a web application where clicks don't necessarily generate server requests, then collecting click information with javascript is your best bet.

Streaming the result of a command back to the browser using Twisted and Comet

I'm writing an application that streams the output (by this I mean both sys.stdout and sys.stderr) of a python script excited on the server, in real time to the browser.
The users on the site will be allowed to select the script to run, excite and kill their chosen script, and change some parameters, so I will need a different thread per user on the site (user A can start, stop and change a script, whilst user B can do the same with a different script).
I know I need to use comet for the web clients, and seeing as the rest of the project is written in python, I'd like to use twisted for the server, however I'm not really sure of what I need to do next!
There are a daunting number of options (Divmod Mantissa, Divmod Nevow, twisted.web, STOMP, etc), and some are better documented that others, making the whole thing rather tricky!
I have a working demo using stompservice on orbited, using Orbited.TCPSocket for the javascript side of things, however I'm starting to think that STOMPs channel model isn't going to work for multithreading, multi-running scripts (unless I open a new channel per run, but that seems like the wrong use of the channel model).
Can anyone point me in the right direction, or some sample code I can learn from?
Thanks!

Nevow Athena is a framework specifically for AJAX and COMET applications and in theory is exactly the sort of thing you are looking for.
However, I am not sure that it is well used or supported at this time - looking at mailing list traffic and google search results suggests that it may not be.
There are a couple of tutorials you could look at to help you decide on it:
one on the 'official' site: http://divmod.org/trac/wiki/DivmodNevow/Athena/Tutorials/LiveElement
and one other that I found:
http://divmodsphinx.funsize.net/nevow/chattutorial/part01/index.html
The code for the latter seems to be included in the Nevow distribution when you download it under /doc/listings/partxx (I think...)

You can implement a very simple "HTTP streaming" by keeping the http connection open and appending javascript chunks that update the dom contents. This works since the browser evaluates the "script" chunks as they arrive.
I wrote a blog entry a while ago with a running example using twisted and very few lines of javascript: Simple HTTP streaming with Twisted & Javascript
You can easily mix this pattern with a publisher/subscriber pattern to make it multiuser, etc. I use this pattern to watch live log streams via web.

An example of serving for long-polling clients with Twisted is slosh. This might not be what you want, but because it's not a large framework, it can help you figure out how to use Twisted.