This is the requirement from the API's documentation
A HTTPS connection is needed to use the API. This means that you will require a secure SSL/TLS connection to be able to communicate with our API Server.
This is the Curl command of getting the clients in their documentation
curl -i -X GET -H "X-KEYALI-API:{API_KEY}" -u {API_USERNAME}:{API_PASSWORD} https://aliphia.com/v1/api_public/clients/
So, I need to implement the same thing in Python
import requests
headers = {
'X-KEYALI-API': '{API_KEY}',
}
response = requests.get('https://aliphia.com/v1/api_public/clients/', headers=headers, auth=('{API_USERNAME}', '{API_PASSWORD}'))
I have a Django rest service running on virutal environment on gunicorn server with the following .wsgi file:
import os, sys import site
site.addsitedir('/opt/valuation/env/lib/python2.7/site-packages')
sys.stdout = sys.stderr
os.environ['DJANGO_SETTINGS_MODULE'] = 'valuation.valuationcont.valuation.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
When I do curl POST call the service works perfectly:
curl -H "Content-Type: application/json" -X POST -d '{...}' -u username:password http://localhost:8000/valuation/predict/
But when I do the same request on API gateway using axios, Django service responds my custom GET response ("GET not supported, try POST").
axios({
method: 'post',
url:'http://localhost:8000/valuation/predict',
headers:{
"Content-Type":"application/json",
"Authorization":"Basic [BASE64 ENCODING]"
},
data:{
...
}
}).then(response=>{
console.log(response.data)
}).catch(err=>{
console.log(err.toString())
})
The request is transformed from GET to POST.
This only happens with the django/gunicorn service.
Since I am new to django/gunicorn I think there is something wrong with the .wsgi file. But how come the curl call then works?
Any help appreciated, been struggling with this for a week now.
Edit:
Managed to recreate the same problem in my local machine. axios POST requests using its API are translated into GET.
Using the axios.post(...) method I managed to get 403 and 201. All while POSTMAN works fine.
I have a suspicion that since the POST fails axios API has a default fallback to GET which then doesn't fail and service responds normally ("GET not supported" as is should).
New step to debug this would be to ask, how do I recreate POSTMAN POST call as close as possible in javascript since POSTMAN is working and it is obviously axios that is causing the problems.
You're not using the same URL. In the curl snippet you request http://localhost:8000/valuation/predict/ but in the second you request http://localhost:8000/valuation/predict - without the final slash.
Django by default redirects URLs that don't end in a slash to one that does, and a redirect is always a GET.
How do I save the data uploaded using a curl command like curl localhost:5000/upload/test.bin --upload-file tmp/test.bin in a Flask-RESTful PUT handler method?
The code in Ron Harlev's answer to Flask-RESTful - Upload image works with the POST request from curl -F "file=#tmp/test.bin" localhost:5000/upload/test.bin (slightly modified below):
def post(self, filepath):
parse = reqparse.RequestParser()
parse.add_argument('file', type=werkzeug.datastructures.FileStorage, location='files')
args = parse.parse_args()
upload_file = args['file']
upload_file.save("/usr/tmp/{}".format(filepath))
return ({'filepath': filepath}, 200)
But if I try to use the code to handle the PUT request from curl --upload-file (changing post to put above, of course) I get: "'NoneType' object has no attribute 'save'". This refers to the second-last line in the above code.
How do I get a handle to the file data uploaded using curl --upload-file, so I can save it to a local file?
Update: This works around the problem: curl --request PUT -F "file=#tmp/test.bin" localhost:5000/upload/test.bin, but I still don't have an answer to my question.
curls docs define --upload-file as a PUT http request https://curl.haxx.se/docs/manpage.html#-T.
I'm not sure of the need to handle this through a restful API and I'm pretty sure that's where curl is causing problems, perhaps the assumption that this must be done through flask-restful is holding you back?
maybe try building this as a vanilla flask endpoint instead, this code should work for you.
from flask import Flask, request, jsonify
...
#app.route('/simpleupload/<string:filepath>', methods=['POST','PUT'])
def flask_upload(filepath):
with open("/tmp/{}".format(filepath), 'wb') as file:
file.write(request.stream.read())
return (jsonify({'filepath': filepath}), 200)
I wanted to know if airflow tasks can be executed upon getting a request over HTTP. I am not interested in the scheduling part of Airflow. I just want to use it as a substitute for Celery.
So an example operation would be something like this.
User submits a form requesting for some report.
Backend receives the request and sends the user a notification that the request has been received.
The backend then schedules a job using Airflow to run immediately.
Airflow then executes a series of tasks associated with a DAG. For example, pull data from redshift first, pull data from MySQL, make some operations on the two result sets, combine them and then upload the results to Amazon S3, send an email.
From whatever I read online, you can run airflow jobs by executing airflow ... on the command line. I was wondering if there is a python api which can execute the same thing.
Thanks.
The Airflow REST API Plugin would help you out here. Once you have followed the instructions for installing the plugin you would just need to hit the following url: http://{HOST}:{PORT}/admin/rest_api/api/v1.0/trigger_dag?dag_id={dag_id}&run_id={run_id}&conf={url_encoded_json_parameters}, replacing dag_id with the id of your dag, either omitting run_id or specify a unique id, and passing a url encoded json for conf (with any of the parameters you need in the triggered dag).
Here is an example JavaScript function that uses jQuery to call the Airflow api:
function triggerDag(dagId, dagParameters){
var urlEncodedParameters = encodeURIComponent(dagParameters);
var dagRunUrl = "http://airflow:8080/admin/rest_api/api/v1.0/trigger_dag?dag_id="+dagId+"&conf="+urlEncodedParameters;
$.ajax({
url: dagRunUrl,
dataType: "json",
success: function(msg) {
console.log('Successfully started the dag');
},
error: function(e){
console.log('Failed to start the dag');
}
});
}
A new option in airflow is the experimental, but built-in, API endpoint in the more recent builds of 1.7 and 1.8. This allows you to run a REST service on your airflow server to listen to a port and accept cli jobs.
I only have limited experience myself, but I have run test dags with success. Per the docs:
/api/experimental/dags/<DAG_ID>/dag_runs creates a dag_run for a given dag id (POST).
That will schedule an immediate run of whatever dag you want to run. It does still use the scheduler, though, waiting for a heartbeat to see that dag is running and pass tasks to the worker. This is exactly the same behavior as the CLI, though, so I still believe it fits your use-case.
Documentation on how to configure it is available here: https://airflow.apache.org/api.html
There are some simple example clients in the github, too, under airflow/api/clients
You should look at Airflow HTTP Sensor for your needs. You can use this to trigger a dag.
Airflow's experimental REST API interface can be used for this purpose.
Following request will trigger a DAG:
curl -X POST \
http://<HOST>:8080/api/experimental/dags/process_data/dag_runs \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{"conf":"{\"START_DATE\":\"2018-06-01 03:00:00\", \"STOP_DATE\":\"2018-06-01 23:00:00\"}'
Following request retrieves a list of Dag Runs for a specific DAG ID:
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET http://<HOST>:8080/api/experimental/dags/process_data/dag_runs
For the GET API to work set rbac flag to True at airflow.cfg.
Following are the list of APIs available: here & there.
UPDATE: stable Airflow REST API released:
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
Almost everything stays the same, except API URL change.
Also "conf" is now required to be an object, so I added additional wrapping:
def trigger_dag_v2(self, dag_id, run_id=None, conf=None, execution_date=None):
endpoint = '/api/v1/dags/{}/dagRuns'.format(dag_id)
url = urljoin(self._api_base_url, endpoint)
data = self._request(url, method='POST',
json={
"run_id": run_id,
"conf": {'conf': json.dumps(event)},
"execution_date": execution_date,
})
return data['message']
OLD ANSWER:
Airflow has REST API (currently experimental) - available here:
https://airflow.apache.org/api.html#endpoints
If you do not want to install plugins as suggested in other answers - here is code how you can do it directly with the API:
def trigger_dag(self, dag_id, run_id=None, conf=None, execution_date=None):
endpoint = '/api/experimental/dags/{}/dag_runs'.format(dag_id)
url = urljoin(self._api_base_url, endpoint)
data = self._request(url, method='POST',
json={
"run_id": run_id,
"conf": conf,
"execution_date": execution_date,
})
return data['message']
More examples working with airflow API in python are available here:
https://github.com/apache/airflow/blob/master/airflow/api/client/json_client.py
I found this post while trying to do the same, after further investigation, I switch to ArgoEvents. It is basically the same but based on event-driven flows so it is much more suitable for this use case.
Link:
https://argoproj.github.io/argo
Airflow now has support for stable REST API. Using stable REST API, you can trigger DAG as:
curl --location --request POST 'localhost:8080/api/v1/dags/unpublished/dagRuns' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
"dag_run_id": "dag_run_1",
"conf": {
"key": "value"
}
}'
I have a web application that serves post requests, and handles cases when no data or empty data are given. When running the application manually, it behaves properly. But when serving it through uwsgi, a specific case fails, POST requests with no data (empty data works):
$ curl -X POST http://www.example.com/search
curl: (52) Empty reply from server
$ curl -d "" http://www.example.com/search
[{...}]
Is there any uwsgi option I missed that could fix this behavior? How can I get uwsgi to properly forward those requests to my web application?
As #roberto stated, it was a problem in my code. I had this piece of code:
When a user does a post query without data, wheezy will try building self.request.form using environ variables like CONTENT_LENGTH, which here doesn't exist and raises an error. Somehow, the try/except did not work with uwsgi, so I changed:
try:
self.request.form
except:
self.request.form = {}
self.request.files = {}
As follow:
if not self.request.environ.get('CONTENT_LENGTH', False):
self.request.form = {}
self.request.files = {}
And it works fine now