Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed last month.
Improve this question
I use an API that has ~30 endpoints and I have settings how often I need to send request to each endpoint. For some endpoints it's seconds and for some hours. I want to implement python app that will call each API endpoint (and execute some code) after every N seconds where N can be different for each endpoint. If one call is still in progress when second one kicks in, then that one should be added to queue (or something similar) and executed after the first one finishes.
What would be the correct way to implement this using python?
I have some experience with RabbitMQ but I think that might be overkill for this problem.
You said "executed after the first one finishes", so it's a single thread program.
Just use def() to create some functions and then execute them one by one.
For example
import time
def task1(n):
print("Task1 start")
time.sleep(n)
print("Task1 end ")
def task2(n):
print("Task2 start")
time.sleep(n)
print("Task2 end ")
task1(5) #After 5sec, task1 end and execute task2
task2(3) #task2 need 3sec to execute.
You could build your code in this way:
store somewhere the URL, method and parameters for each type of query. A dictionary would be nice: {"query1": {"url":"/a","method":"GET","parameters":None} , "query2": {"url":"/b", "method":"GET","parameters":"c"}} but you can do this any way you want, including a database if needed.
store somewhere a relationship between query type and interval. Again, you could do this with a case statement, or with a dict (maybe the same you previously used), or an interval column in a database.
Every N seconds, push the corresponding query entry to a queue (queue.put)
an HTTP client library such as requests runs continuously, removes an element from the queue, runs the HTTP request and when it gets a result it removes the following element.
Of course if your code is going to be distributed across multiple nodes for scalability or high availability, you will need a distributed queue such as RabbitMQ, Ray or similar.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
I'm working with an API and the documentation doesn't state the exact limits on the requests I make, this causes my app to suddenly stop working because of long waiting periods and eventually timeouts.
Is there a way to find out what the API limits are and build a workaround? such as "if API limits are 5 requests per minute then wait a minute before sending the 6th request" or so ...
The API I'm talking about here is the TD Ameritrade API, documentation:
https://developer.tdameritrade.com/home
I'm coding with Python.
Thanks for anybody who helps.
Edit: Problem was solved, the API can handle 120 calls per minute.
Yes, there is a limit every minute. So, it's says at the bottom of this page : https://developer.tdameritrade.com/content/authentication-faq
All non-order based requests by personal use non-commercial applications are throttled to 120 per minute. Exceeding this throttle limit will provide a response with a 429 error code to inform you that the throttle limit has been exceeded.
API calls, especially private accounts are restricted to be able to preserve processing power to people who pay for the service, like companies do.
For about 2 minutes of searching in the documentation, I managed to find this line:
All private, non-commercial use apps are currently limited to 120 requests per minute on all APIs except for Accounts & Trading
Please, read the docs carefully before posting here!
By the way, you can calculate that you have 120 calls / 60 seconds, which means 1 call / 0.5 second.
You can simply sleep for that amount of time, or delay the call of a new thread, if your app is designed that way.
Since you did not provided any code, I will show you a basic example using sleep.
import time
while True: #main loop
apicall() #apicall here
time.sleep(1) #sleep 1 second after each call
But I strongly suggest adding your code to the question, so people can provide you better solutions.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I'm relatively new to Python and requests, so I'm not sure the best way to go about this.
I need to send a large amount of POST requests to a URL. Right now, I'm simply using a loop and sending the request, which yields roughly 100 posts every 10 - 30 seconds, depending on the internet. I'm looking for a way to do this faster and with more posts. Multiprocessing was recommended to me, but my knowledge here is very lacking (I've already frozen my computer trying to spawn too many processes).
How can I effectively implement multiprocessing to increase my results?
Here is a code sample taken from http://skipperkongen.dk/2016/09/09/easy-parallel-http-requests-with-python-and-asyncio/ which may solve your problem. It uses the requests library to make the request and asyncio for the asynchronous calls. The only change you'd have to make is from a GET call to a POST call.
This was written in Python 3.5 (as expressed in the article)
# Example 2: asynchronous requests
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
futures = [
loop.run_in_executor(
None,
requests.get,
'http://example.org/'
)
for i in range(20)
]
for response in await asyncio.gather(*futures):
pass
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
I would also recommend reading the entire article as it shows time comparisons when using lots of threads.
There's no reason to use multiprocessing here. Making requests of HTTP servers is almost entirely I/O-bound, not CPU-bound, so threads work just fine.
And the very first example of using ThreadPoolExecutor in the stdlib's concurrent.futures documentation does exactly what you're asking for, except with urllib instead of requests.
If you're doing anything complicated, look at requests-futures.
If you really do need to use multiprocessing for some reason (e.g., you're doing a whole lot of text processing on each result, and you want to parallelize that along with the requesting), you can just switch the ThreadPoolExecutor to a ProcessPoolExecutor and change nothing else in your code.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
Hi I have a Server/client model using SocketServer module. The server job is to receive test name from the clients and launch the test.
the test is launched using subprocess module.
I would like the server to keep answering clients and any new jobs to be stacked on a list or queue and launch one after the other, the only restriction I have is the server should not launch the test unless currently running one is completed.
Thanks
You can use the module multiprocessing for starting new processes. On the server-side, you would have a variable which refers to the current running process. You can still have your SocketServer running and accepting requests and storing them in a list. Every second (or whatever you want), in another thread, you would check if the current process is dead or not by calling isAlive(). If it is dead, then just simply run the next test on the list.
Another way to do it (better), is that on the third thread (the one that checks), you call .join() from the process so that it will only call the next line of code once the current process is dead. That way you don't have to keep checking every second or whatever and it is more efficient.
What you might want to do is:
Get test name in server socket, put it in a Queue
In a separate thread, read test names from the Queue one by one
Execute the process and wait for it to end using communicate()
Keep polling Queue for new tests, repeat steps 2, 3 if test names are available
Meanwhile server continues receiving and putting test names in Queue
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am using Python 2.7.6 and the threading module.
I am fairly new to python threading. I am trying to write a program to read files from a filesystem and store some hashes in my database. That are a lot of files and I would like to do it in threads. Like one thread for every folder that starts with a, one thread for every folder that starts with b. Since I want to use a database connection in the threads I don't want to generate 26 threads at once. So I would like to have 10 threads running and always if one of them finishes I want to start a new thread.
The main program should hold a list of threads with a specified max
amount of threads (e.g. 10)
The main program should start 10 threads
The main program should be notified when one thread finished
If a thread is finished start a new one
And so on ... until the job is done and every thread is finished
I am not quite sure how the main program has to look like. How can I manage this list of threads without a big overhead?
I'd like to indicate you that python doesn't manage well multi-threading : As you might know (or not) python comes with a Global Interpreter Lock (GIL), that doesn't allow real concurrency : Indeed, only one thread will execute at a time. (However you will not see the execution as a sequential one, thanks to the process scheduler of your machine)
Take a look here for more information : http://www.dabeaz.com/python/UnderstandingGIL.pdf
That said, if you still want to do it this way, take a look at semaphores : every thread will have to acquire it, and if you initialize this lock to 10, only 10 thread at a time will be able to acquire it.
https://docs.python.org/2/library/threading.html#threading.Semaphore
Hope it helps
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am working a lot with texts in Python, but im kinda new to the language and don't yet know how to employ multi-threading in Py.
My usecase is the following:
Single producer P (database/XML) which generates texts T_s.
Each of the texts in T_s could be processed independently. Processed texts compose T_p set.
The resulting set is written to a text-file/XML/database by a single thread S.
Data volumes are huge and all the processing couldn't keep anything except for the current data in the memory.
I would organize the process as the following:
Producer put the texts into Q_s queue.
There are a set of workers and a manager that gets texts from the queue and distributes between workers.
Each worker puts the processed text to the Q_p.
Sink process reads processed texts from Q_p and persists them.
Beyound all that Producer should be able to communicate that it ended reading the input data source to the manager and the sink.
Summary. I learned so far, that there is a nice lib/solution for each of the typical tasks in Py. Is there any for my current task?
Due to the nature of CPython (see gil), you will need to use multiple processes rather than threads if your tasks are CPU and not I/O bound. Python comes with the multiprocessing module that has everything you need to get the job done. Specifically, it has pools and thread-safe queues.
In your case, you need an input and output queues that you pass to each worker and they asynchronously read from the input queue and write to the output queue. The single threaded producers/consumers just operate on their respective queues, keeping only what's necessary in memory. The only potential quirk here is that order of outputs may not correlate with the order of the inputs.
Note: you can communicate status with the JoinableQueue class.