We Keep Coding

notebook to execute Databricks job

Is there an api or other way to programmatically run a Databricks job. Ideally, we would like to call a Databricks job from a notebook. Following just gives currently running job id but that's not very useful:
dbutils.notebook.entry_point.getDbutils().notebook().getContext().currentRunId().toString()

To run a databricks job, you can use Jobs API. I have a databricks job called for_repro which I ran using the 2 ways provided below from databricks notebook.
Using requests library:
You can create an access token by navigating to Settings -> User settings. Under Access token tab, click generate token.
Use the above generated token along with the following code.
import requests
import json
my_json = {"job_id": <your_job-id>}
auth = {"Authorization": "Bearer <your_access-token>"}
response = requests.post('https://<databricks-instance>/api/2.0/jobs/run-now', json = my_json, headers=auth).json()
print(response)
The <databricks-instance> value from the above code can be extracted from your workspace URL.
Using %sh magic command script:
You can also use magic command %sh in your python notebook cell to run a databricks job.
%sh
curl --netrc --request POST --header "Authorization: Bearer <access_token>" \
https://<databricks-instance>/api/2.0/jobs/run-now \
--data '{"job_id": <your job id>}'
The following is my job details and run history for reference.
Refer to this Microsoft documentation to know all other operations that can be achieved using Jobs API.

Python requests.post timeing out on localhost

I'm having trouble getting the Python requests module to post to an endpoint on the same server.
The server is running flask, and the route I want to post to is /new_applet.
My code is below. It is inside a different flask route function, which could be an issue?
url = "http://0.0.0.0:5000/new_applet"
data = {
"action": "add",
"plugin": plugin,
"version": version,
"component": component
}
resp = requests.post(url, data=data)
However it hangs while trying to make the post request. Debugging shows that the request never reaches the flask route function.
The request works if I run the following command on the server:
curl --location --request POST '0.0.0.0:5000/new_applet' \
--form 'action=add' \
--form 'plugin=up_spotify' \
--form 'version=0.1' \
--form 'component=inputs'
Why doesn't the Python request work, when the curl request does?

I believe flask runs a single thread by default.
According to your explanation, if the request is made from flask then it blocks the thread waiting for a reply (same thread that is supposed to handle the request.post).
you could try to use multiple thread as explained here: Can I serve multiple clients using just Flask app.run() as standalone?

Flask session doesn't update consistently with parallel requests

I'm noticing that when requests running in parallel modify Flask's session, only some keys are recorded. This happens both with Flask's default cookie session and with Flask-Session using the Redis backend. The project is not new, but this only became noticeable once many requests were happening at the same time for the same session.
import time
from flask import Flask, session
from flask_session import Session
app = Flask(__name__)
app.secret_key = "example"
app.config["SESSION_TYPE"] = "redis"
Session(app)
#app.route("/set/<value>")
def set_value(value):
"""Simulate long running task."""
time.sleep(1)
session[value] = "done"
return "ok\n"
#app.route("/keys")
def keys():
return str(session.keys()) + "\n"
The following shell script demonstrates the issue. Notice that all the requests complete, but only one key is present in the final listing, and it's different between test runs.
#!/bin/bash
# set session
curl -c 'cookie' http://localhost:5007/keys
# run parallel
curl -b 'cookie' http://localhost:5007/set/key1 && echo "done1" &
curl -b 'cookie' http://localhost:5007/set/key2 && echo "done2" &
curl -b 'cookie' http://localhost:5007/set/key3 && echo "done3" &
wait
# get result
curl -b 'cookie' http://localhost:5007/keys
$ sh test.sh
dict_keys(['_permanent'])
ok
ok
ok
done3
done1
done2
dict_keys(['_permanent', 'key2'])
$ sh test.sh
dict_keys(['_permanent'])
ok
done3
ok
ok
done2
done1
dict_keys(['_permanent', 'key1'])
Why aren't all the keys present after the requests finish?

Cookie-based sessions are not thread safe. Any given request only sees the session cookie sent with it, and only returns the cookie with that request's modifications. This isn't specific to Flask, it's how HTTP requests work.
You issue three requests in parallel. They all read the initial cookie that only contains the _permanent key, send their requests, and get a response that sets a cookie with their specific key. Each response cookie would have the _permanent key and the key_keyN key only. Whichever request finishes last writes to the file, overwriting previous data, so you're left with its cookie only.
In practice this isn't an issue. The session isn't really meant to store data that changes rapidly between requests, that's what a database is for. Things that modify the session, such as logging in, don't happen in parallel to the same session (and are idempotent anyway).
If you're really concerned about this, use a server-side session to store the data in a database. Databases are good at synchronizing writes.
You're already using Flask-Session and Redis, but digging into the Flask-Session implementation reveals why you have this issue. Flask-Session doesn't store each session key separately, it writes a single serialized value with all the keys. So it suffers the same issue as cookie-based sessions: only what was present during that request is put back into Redis, overwriting what happened in parallel.
In this case, it will be better to write your own SessionInterface subclass to store each key individually. You would override save_session to set all keys in session and delete any that aren't present.

Django service on gunicorn POST request is recieved as GET?

I have a Django rest service running on virutal environment on gunicorn server with the following .wsgi file:
import os, sys import site
site.addsitedir('/opt/valuation/env/lib/python2.7/site-packages')
sys.stdout = sys.stderr
os.environ['DJANGO_SETTINGS_MODULE'] = 'valuation.valuationcont.valuation.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
When I do curl POST call the service works perfectly:
curl -H "Content-Type: application/json" -X POST -d '{...}' -u username:password http://localhost:8000/valuation/predict/
But when I do the same request on API gateway using axios, Django service responds my custom GET response ("GET not supported, try POST").
axios({
method: 'post',
url:'http://localhost:8000/valuation/predict',
headers:{
"Content-Type":"application/json",
"Authorization":"Basic [BASE64 ENCODING]"
},
data:{
...
}
}).then(response=>{
console.log(response.data)
}).catch(err=>{
console.log(err.toString())
})
The request is transformed from GET to POST.
This only happens with the django/gunicorn service.
Since I am new to django/gunicorn I think there is something wrong with the .wsgi file. But how come the curl call then works?
Any help appreciated, been struggling with this for a week now.
Edit:
Managed to recreate the same problem in my local machine. axios POST requests using its API are translated into GET.
Using the axios.post(...) method I managed to get 403 and 201. All while POSTMAN works fine.
I have a suspicion that since the POST fails axios API has a default fallback to GET which then doesn't fail and service responds normally ("GET not supported" as is should).
New step to debug this would be to ask, how do I recreate POSTMAN POST call as close as possible in javascript since POSTMAN is working and it is obviously axios that is causing the problems.

You're not using the same URL. In the curl snippet you request http://localhost:8000/valuation/predict/ but in the second you request http://localhost:8000/valuation/predict - without the final slash.
Django by default redirects URLs that don't end in a slash to one that does, and a redirect is always a GET.

How to auth into BigQuery on Google Compute Engine?

What's the easiest way to authenticate into Google BigQuery when on a Google Compute Engine instance?

Make sure that your instance has the scope to access BigQuery first of all - you can decide this only at creation time.
in a bash script, get a oauth token by calling :
ACCESSTOKEN=`curl -s "http://metadata/computeMetadata/v1/instance/service-accounts/default/token" -H "X-Google-Metadata-Request: True" | jq ".access_token" | sed 's/"//g'`
echo "retrieved access token $ACCESSTOKEN"
now let's say you want a list of the data sets in a project :
CURL_URL="https://www.googleapis.com/bigquery/v2/projects/YOURPROJECTID/datasets"
CURL_OPTIONS="-s --header 'Content-Type: application/json' --header 'Authorization: OAuth $ACCESSTOKEN' --header 'x-goog-project-id:YOURPROJECTID' --header 'x-goog-api-version:1'"
CURL_COMMAND="curl --request GET $CURL_URL $CURL_OPTIONS"
CURL_RESPONSE=`eval $CURL_COMMAND`
the response in JSON format can be found in the variable CURL_RESPONSE
PS: I realize now that this question is tagged as Python, but same principles apply.

In Python:
AppAssertionCredentials is a python class that allows a Compute Engine instance to identify itself to Google and other OAuth 2.0 servers, withour requiring a flow.
https://developers.google.com/api-client-library/python/
The project id can be read from the metadata server, so it doesn't need to be set as a variable.
https://cloud.google.com/compute/docs/metadata
The following code gets a token using AppAssertionCredentials, the project id from the metadata server, and instantiates a BigqueryClient with this data:
import bigquery_client
import urllib2
from oauth2client import gce
def GetMetadata(path):
return urllib2.urlopen(
'http://metadata/computeMetadata/v1/%s' % path,
headers={'Metadata-Flavor': 'Google'}
).read()
credentials = gce.AppAssertionCredentials(
scope='https://www.googleapis.com/auth/bigquery')
client = bigquery_client.BigqueryClient(
credentials=credentials,
api='https://www.googleapis.com',
api_version='v2',
project_id=GetMetadata('project/project-id'))
For this to work, you need to give the GCE instance access to the BigQuery API when creating it:
gcloud compute instances create <your_instance_name> --scopes storage-ro bigquery