First time trying to use Python 3.6 requests library's get() function with data from quandl.com and load and dump json.
import json
import requests
request = requests.get("https://www.quandl.com/api/v3/datasets/CHRIS/MX_CGZ2.json?api_key=api_keyxxxxx", verify=False)
request_text=request.text()
data = json.loads(request_text)
data_serialized = json.dumps(data)
print(data_serialized)
I have an account at quandl.com to access the data. The error when python program is run in cmd line says "cannot connect to HTTPS URL because SSL mode not available."
import requests
import urllib3
urllib3.disable_warnings()
r = requests.get(
"https://www.quandl.com/api/v3/datasets/CHRIS/MX_CGZ2.json?api_key=api_keyxxxxx").json()
print(r)
Output will be the following since i don't have API Key
{'quandl_error': {'code': 'QEAx01', 'message': 'We could not recognize your API key. Please check your API key and try again.'}}
You don't need to import json module as requests already have it.
Although I have verified 'quandl_api_keys received the following error when trying to retrieve data with 'print' data 'json.dumps' function: "quandl_error": {"code": "QEAx01" ... discovered that the incorrect fonts on the quotation marks around key code in the .env file resulted in this error. Check language settings and fonts before making requests after the error mssg.
I am trying to access a certain 'third-party' database via a URL, using the urllib.request library but I get this error instead:
<urlopen error [Errno 2] No such file or directory>
I have been able to access the link several times with the same script some time before though.
import urllib.request
def read_temp():
url = "https://maturity.000webhostapp.com/api/temp/read_all.php"
response = urllib.request.urlopen(url).read()
json_obj = str(response, 'utf-8')
data = json.loads(json_obj)
I'd expected that the script should work effortlessly but sometimes it fails to connect.
The website has been taken down by the owner.
I'm trying to connect to the Etherscan API in order download my transaction history across my cryptocurrency wallets. I've used the following Python code, which works just fine with the sample Etherscan gives in their API documentation.
import pandas as pd
import requests
address = '0xfbb1b73c4f0bda4f67dca266ce6ef42f520fbb98' #wallet address
url = 'http://api.etherscan.io/api?module=account&action=txlist&address='+address+'&startblock=0&endblock=99999999&sort=asc&apikey=my_api_key'
r = requests.get(url)
r.status_code #200 indicates success
json_output =r.json()
json_output["result"]
data = pd.DataFrame(json_output["result"])
I consistently get no data returned although the status code tells me the query is a success. Any thought? How stupid am I being?
I only manage to use the Emotion API subscription key for pictures but never for videos. It makes no difference whether I use the API Testing Console or try to call the Emotion API by Pathon 2.7. In both cases I get a response status 202 Accepted, however when opening the Operation-Location it says
{ "error": { "code": "Unauthorized", "message": "Access denied due to
invalid subscription key. Make sure you are subscribed to an API you are
trying to call and provide the right key." } }
On the Emotion API explanatory page it says that Response 202 means that
The service has accepted the request and will start the process later.
In the response, there is a "Operation-Location" header. Client side should further query the operation status from the URL specified in this header.
Then there is Response 401, which is exactly what my Operation-Location contains. I do not understand why I'm getting a response 202 which looks like response 401.
I have tried to call the API with Python using at least three code versions that I found on the Internet that
all amount to the same, I found the code here :
Microsoft Emotion API for Python - upload video from memory
python-upload-video-from-memory
import httplib
import urllib
import base64
import json
import pandas as pd
import numpy as np
import requests
_url = 'https://api.projectoxford.ai/emotion/v1.0/recognizeInVideo'
_key = '**********************'
_maxNumRetries = 10
paramsPost = urllib.urlencode({'outputStyle' : 'perFrame', \
'file':'C:/path/to/file/file.mp4'})
headersPost = dict()
headersPost['Ocp-Apim-Subscription-Key'] = _key
headersPost['content-type'] = 'application/octet-stream'
jsonGet = {}
headersGet = dict()
headersGet['Ocp-Apim-Subscription-Key'] = _key
paramsGet = urllib.urlencode({})
responsePost = requests.request('post', _url + "?" + paramsPost, \
data=open('C:/path/to/file/file.mp4','rb').read(), \
headers = headersPost)
print responsePost.status_code
videoIDLocation = responsePost.headers['Operation-Location']
print videoIDLocation
Note that changing _url = 'https://api.projectoxford.ai/emotion/v1.0/recognizeInVideo' to _url =
'https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognizeInVideo' doesn't help.
However, afterwards I wait and run every half an hour:
getResponse = requests.request('get', videoIDLocation, json = jsonGet,\
data = None, headers = headersGet, params = paramsGet)
print json.loads(getResponse.text)['status']
The outcome has been 'Running' for hours and my video is only about half an hour long.
Here is what my Testing Console looks like Testing Console for Emotion API, Emotion Recognition in Video
Here I used another video that is about 5 minutes long and available on the internet. I found the video in a different usage example
https://benheubl.github.io/data%20analysis/fr/
that uses a very similar code, which again gets me a response status 202 Accepted and when opening the Operation-Location the subscription key is wrong
Here the code:
import httplib
import urllib
import base64
import json
import pandas as pd
import numpy as np
import requests
# you have to sign up for an API key, which has some allowances. Check the
API documentation for further details:
_url = 'https://api.projectoxford.ai/emotion/v1.0/recognizeinvideo'
_key = '*********************' #Here you have to paste your
primary key
_maxNumRetries = 10
# URL direction: I hosted this on my domain
urlVideo = 'http://datacandy.co.uk/blog2.mp4'
# Computer Vision parameters
paramsPost = { 'outputStyle' : 'perFrame'}
headersPost = dict()
headersPost['Ocp-Apim-Subscription-Key'] = _key
headersPost['Content-Type'] = 'application/json'
jsonPost = { 'url': urlVideo }
responsePost = requests.request( 'post', _url, json = jsonPost, data = None,
headers = headersPost, params = paramsPost )
if responsePost.status_code == 202: # everything went well!
videoIDLocation = responsePost.headers['Operation-Location']
print videoIDLocation
There are further examples on the internet and they all seem to work but replicating any of them never worked for me. Does anyone have any idea what could be wrong?
The Video Feature of Emotion API retires October 30th, so maybe you should change your procedure to screenshots anyways.
But for your question: The API returns you an URL where your results are accessible. You cannot open this URL in your browser, this will give you the notice of "invalid key", instead you need to call over python again this URL including your key.
I will post you my code how to get the score, I am using Python 3, so there might be some adjustments necessary. Only "tricky" point is getting the Operation ID, which is just the ID in the URL ( =location in my case) which leads to your request. Rest of the parameters like subscription key etc. is as before.
#extract operation ID from location-string
OID = location[67:]
bod = ""
try:
conn =
http.client.HTTPSConnection('westus.api.cognitive.microsoft.com')
conn.request("GET", "/emotion/v1.0/operations/"+OID+"?%s" %params, bod, headers)
response = conn.getresponse()
data = response.read()
print(data)
conn.close()
except Exception as e:
print("[Errno {0}] {1}".format(e.errno, e.strerror))
Did you verify your API call is working using curl? Always prototype calls using curl first. If it works in curl but not in Python, use Fiddler to observe the API request and response.
I also found an answer in the following link, all steps are explained:
https://gigaom.com/2017/04/10/discover-your-customers-deepest-feelings-using-microsoft-facial-recognition/
I Need to create cronjob to test the website whether the data is retrieved for every one hour.
Initially have tried by pasted the json data into text file and validated the data by encoding and decoding it. Now i need the real time data(json data) to get loaded on every time running the cron job. used urllib2 but it is not getting the request response from the url.
Url -> on loading -> through firebug gives url to execute and to json data from that. how can i import or parse such url into python. Please get me with an example.
my steps:
create shedule
1.45 08 * * 1-5 /home/user/myfile/daily_verifydata.sh >> /home/user/cronlog.log
daily_verifydata.sh
#!/bin/sh
python /home/user/path/Dashboard_test.py
Dashboard_test.py
import json
import urllib2
f = open('test.txt','r') # open in read mode
data = f.read()
print data
# How to Parse the json from the URL to python
data_string = json.dumps(data)
print '\n''ENCODED:', data_string
decoded = json.loads(data_string)
print '\n''DECODED:', decoded
# Validating data through decoded output.
If possible parsing through curl, need to know the syntax
Thanks, vijay
For retrieveing your JSON in bash script — you can use nice tool httpie
If you'd like to pull JSON from python script — best option is requests lib
And for validation, it it's complex — JSONSchema
i recommend using requests
import requests
import simplejson
session = requests.session()
# I presume your site has authentication
response = session.post(URL_TO_LOGIN, {
'username': username,
'password': password
})
response = session.get(URL_TO_JSON)
if response.ok:
simplejson.loads(response.text)