We Keep Coding

Is there an alternative process to threading for async process in python?

Please tell me if there is a better way to do this in python.
I have an endpoint in the flask app that takes in the request, validate, and starts a thread and returns a response saying request if valid and a process is started.
At the end of the thread, I will send a request to the call back URL saying the threading process has completed.
is there a way to do this without threading?
what are the other options to do an asynchronous endpoint call in python where the client doesn't till the process is complete.

You can try using features from Python 3.4+'s asyncio. This will change the working of the program to cooperative multitasking, so need to be a bit careful. Some answers on how to do that here.
Maybe websockets would work for you. The Starlette framework is async-first and has inbuilt websocket support.

Scraping website using Celery

Currently, my structure is Flask, Redis, RabbitMQ and Celery. In my scraping, I am using requests and BeautifulSoup.
My flask is running on apache and wsgi. This is on prod. With app.run(threaded=True)
I have 25 APIs. 10 are to scrape the URL like headers, etc. , and the rest is to use a 3rd party API for that URL.
I am using chord for processing my APIs and getting data from the APIs using requests.
For my chord header I have 3 workers, while on my callback I only have 1.
I am having a bottleneck issue of having ConnectTimeoutError and MaxRetryError. As I read some thread it said to do a timeout for every process, because having this error means you are overloading the remote server.
The problem is since I am using a chord there is no sense to use a time sleep since the 25 API call will be run at the same time. Have anyone encountered this? Or am I doing this wrong?
The thread I read seem to be saying to change the requests to pycurl or use Scrapy. But I dont think that's the case since ConnectTimeoutError is about my host overloading a specific URLs server.
My chord process:
callback = create_document.s(url, company_logo, api_list)
header = [api_request.s(key) for key in api_list.keys()]
result = chord(header)(callback)
In api_request task requests is used.

If you're wanting to limit the number of scrapes running at the same time you can create an enqueue task that checks to see if another task is running that shares the same properties as the task you are wanting to run. If the task is running you tell it to sleep for a few seconds and check again. When it sees that one is not running you can then queue the task you want to run. This will allow you to have sleeps with asynchronous tasks. You can even count the tasks and run more if only a certain number are running. With this you can run 5 at a time and see if it is throttled enough then queue another when you see one has finished etc.
::EDIT::
Documentation for Celery Inspect

Sleep in async application, without blocking event loop

I have a problem with one my async tests: tornado seems not to be up and running (I get 599s), even though it is started as a fixture. I would like to verify that it is indeed running, by doing the following:
Start the test
Before the requests are actually sent to tornado, I would like my code to "sleep" for a long time so that I can manually check with the browser that tornado is indeed running.
How can I tell python to sleep without blocking the async loop? If I do time.sleep, the whole process (and thus the loop) is not responsive for the duration of the sleep.

Both of your questions are answered in Tornado's FAQ:
http://www.tornadoweb.org/en/latest/faq.html
Use yield gen.sleep(1) to wait 1 second while letting other tasks run on the event loop, or use IOLoop.current().add_timeout(1, callback) to run a callback one second later.
Don't use your browser to verify that your application handles multiple requests concurrently: the browser itself may not allow concurrent requests. Use curl instead.

You may check your code whether your request method is POST or PUT, I get the 599 Response Code when I use method POST and didn't post a data, and it gones when I added data.
error:
res = self.fetch('/', method='POST')
fixed:
res = self.fetch('/', method='POST', data={})

How do I return an HTTP response in a callback in Flask, or does it even matter?

I am familiar with evented servers but not threaded ones. A common feature of REST APIs implemented in evented systems like node.js+express or tornado is, in an handler, to do some I/O work asynchronously in a callback, then return the actual HTTP response in the callback. In Express, we have things like:
app.post('/products', function (req, res) {
Product.create(req.body, function (err, product) {
if (err) return res.json(400, err);
res.send(201, product);
});
});
where Post.create hits the database and calls the callback after the product is persisted. The response, whether 201 or 400 is sent in the callback. This keeps the server freed up to do other things while the database is working, but from the point of view of the client, the request seems to take a while.
Suppose I wanted to do the same thing in Flask (which is not an evented server). If I have a POST handler to create several objects that needs to make several database writes that could take several seconds to complete, it seems I have two choices:
I could immediately return a 202 ACCEPTED but then this burdens the client with having to check back to see whether all the writes were committed.
I could just implement all the database writes directly inside the handler. The client will have to wait the few seconds for the reply, but it's synchronous and simple from the client perspective.
My question is whether if I do #2, is Flask smart enough to block the current request thread so that other requests can be handled during the database writes? I would hope the server doesn't block here.
BTW I have done long polling, but this is for a public REST API where clients expect simple requests and responses, so I think either approach 1 or 2 is best. Option 1 seems rare to me, but I am worried about #2 blocking the server? Am I right to be concerned, or is Flask (and threaded servers) just smart so I need not worry?

Blocking vs. non-blocking
Flask itself (much like express) is not inherently blocking or non-blocking - it relies on the underlying container to provide the features necessary for operation (reading data from the user and writing responses to the user). If the server does not provide an event loop (e. g. mod_wsgi) then Flask will block. If the server is a non-blocking one (e. g. gunicorn) then Flask will not block.
On the other end of things, if the code that you write in your handlers is blocking Flask will block, even if it is run on a non-blocking container.
Consider the following:
app.post('/products', function (req, res) {
var response = Product.createSync(req.body);
// Event loop is blocked until Product is created
if (response.isError) return res.json(400, err);
res.send(201, product);
});
If you run that on a node server you will quickly bring everything to a screeching halt. Even though node itself is non-blocking your code is not and it blocks the event loop preventing you from handling any other request from this node until the loop is yielded at res.json or res.send. Node's ecosystem makes it easy to find non-blocking IO libraries - in most other common environments you have to make a conscious choice to use non-blocking libraries for the IO you need to do.
Threaded servers and how they work
Most non-evented containers use multiple threads to manage the workload of a concurrent system. The container accepts requests in the main thread and then farms off the handling of the request and the serving of the response to one of its worker threads. The worker thread executes the (most often blocking) code necessary to handle the request and generate a response. While the handling code is running that thread is blocked and cannot take on any other work. If the request rate exceeds the total thread pool count then clients start backing up, waiting for a thread to complete.
What's the best thing to do with a long-running request in a threaded environment?
Knowing that blocking IO blocks one of your workers, the question now is "how many concurrent users are you expecting to have?" (Where concurrent means "occur over the span of time it takes to accept and process one request") If the answer is "less than the total number of threads in my worker thread pool" then you are golden - your server can handle the load and it's non-blocking nature is in no way a threat to stability. Choosing between #1 and #2 is largely a matter of taste.
On the other hand, if the answer to the above question is "more than the total number of works in my thread pool" then you will need to handle the requests by passing off the user's data to another worker pool (generally via a queue of some kind) and responding to the request with a 202 (Option #1 in your list). That will enable you to lower the response time, which will, in turn, enable you to handle more users.
TL;DR
Flask is not blocking or non-blocking as it does no direct IO
Threaded servers block on the request / response handling thread, not the accept request thread
Depending on the expected traffic you will almost certainly want to go go with option #1 (return a 202 and push the work into a queue to be handled by a different thread pool / evented solution).

Can AppEngine python threads last longer than the original request?

We're trying to use the new python 2.7 threading ability in Google App Engine and it seems like the created thread is getting killed before it finishes running. Our scenario:
User sends a message to the server
We update the user's data
We spawn a thread to do some more heavy duty processing
We return a response to the user before waiting for the heavy duty processing to finish
My assumption was that the thread would continue to run after the request had returned, as long as it did not exceed the total request time limit. What we're seeing though is that the thread is randomly killed partway through it's execution. No exceptions, no errors, nothing. It just stops running.
Are threads allowed to exist after the response has been returned? This does not repro on the dev server, only on live servers.
We could of course use a task queue instead, but that's a real pain since we'd have to set up a url for the action and serialize/deserialize the data.

The 'Sandboxing' section of this page:
http://code.google.com/appengine/docs/python/python27/using27.html#Sandboxing
indicates that threads cannot run past the end of the request.

Deferred tasks are the way to do this. You don't need a URL or serialization to use them:
from google.appengine.ext import deferred
deferred.defer(myfunction, arg1, arg2)