We Keep Coding

How to test python's http.client.HTTPResponse?

I'm trying to work with a third party API and I am having problems with sending the request when using the requests or even urllib.request.
Somehow when I use http.client I am successful sending and receiving the response I need.
To make life easier for me, I created an API class below:
class API:
def get_response_data(self, response: http.client.HTTPResponse) -> dict:
"""Get the response data."""
response_body = response.read()
response_data = json.loads(response_body.decode("utf-8"))
return response_data
The way I use it is like this:
api = API()
rest_api_host = "api.app.com"
connection = http.client.HTTPSConnection(rest_api_host)
token = "my_third_party_token"
data = {
"token":token
}
payload = json.loads(data)
headers = {
# some headers
}
connection.request("POST", "/some/endpoint/", payload, headers)
response = connection.getresponse()
response_data = api.get_response_data(response) # I get a dictionary response
This workflow works for me. Now I just want to write a test for the get_response_data method.
How do I instantiate a http.client.HTTPResponse with the desired output to be tested?
For example:
from . import API
from unittest import TestCase
class APITestCase(TestCase):
"""API test case."""
def setUp(self) -> None:
super().setUp()
api = API()
def test_get_response_data_returns_expected_response_data(self) -> None:
"""get_response_data() method returns expected response data in http.client.HTTPResponse"""
expected_response_data = {"token": "a_secret_token"}
# I want to do something like this
response = http.client.HTTPResponse(expected_response_data)
self.assertEqual(api.get_response_data(response), expected_response_data)
How can I do this?
From the http.client docs it says:
class http.client.HTTPResponse(sock, debuglevel=0, method=None, url=None)
Class whose instances are returned upon successful connection. Not instantiated directly by user.
I tried looking at socket for the sock argument in the instantiation but honestly, I don't understand it.
I tried reading the docs in
https://docs.python.org/3/library/http.client.html#http.client.HTTPResponse
https://docs.python.org/3/library/socket.html
Searched the internet on "how to test http.client.HTTPResponse" but I haven't found the answer I was looking for.

how do I make API request after deploying uvicorn via fastapi in mylaptop

I am new to FastAPI/model deployment, I managed to created a model which can return a result using the FastAPI docs path in my local server/laptop.
My question is, how can i access this model in jupyter notebook?
Trying to relate into other API request such as Foursquare API. How do I wrap a params dictionary
Tried the following however it is not working
import requests
url = 'http://127.0.0.1:8000/predict'
params = { 'title' : ['Product Operations Software Engineer (DevOps / SRE)']}
result = requests.post(url, params)
<Response [422]>
result.text
'{"detail":[{"loc":["body"],"msg":"value is not a valid dict","type":"type_error.dict"}]}'
Input
Output
My Code
from fastapi import FastAPI
import numpy as np
import dill
import pandas as pd
import uvicorn
import re
from pydantic import BaseModel
app = FastAPI()
class JobTitle(BaseModel):
title: list
#load model
with open("logisticmodel", "rb") as to_load:
logistic_model = dill.load(to_load)
#app.get("/")
def home():
return 'welcome'
#app.post("/predict")
def predict(request: JobTitle):
X = pd.Series(request.title)
result = logistic_model.predict(X)
return {'Type' : list(result) }

Your code here indicates that your data should be put inside the HTTP request body
#app.post("/predict")
def predict(request: JobTitle):
...
But when you issue the HTTP request with the requests module,
you didn't put anything to your request body.
You should use json argument so the data can be put to the right place:
result = requests.post(
url,
json={'title': ['Product Operations Software Engineer (DevOps / SRE)']}
)
You can check this for sending an HTTP request with request body using requests module,
and this for building an API with a request body.

You need to use .dict() to get the dictionary of BaseModel instance. In your case, request.dict()
Also, Define your model like this,
class JobTitle(BaseModel):
title: List[str]

This works:
>>> result = requests.post(url, json=params)
>>> result
<Response [200]>

Iinvalid subscription key when using Microsoft Emotion API for Video in API Testing Console and in Python 2.7

I only manage to use the Emotion API subscription key for pictures but never for videos. It makes no difference whether I use the API Testing Console or try to call the Emotion API by Pathon 2.7. In both cases I get a response status 202 Accepted, however when opening the Operation-Location it says
{ "error": { "code": "Unauthorized", "message": "Access denied due to
invalid subscription key. Make sure you are subscribed to an API you are
trying to call and provide the right key." } }
On the Emotion API explanatory page it says that Response 202 means that
The service has accepted the request and will start the process later.
In the response, there is a "Operation-Location" header. Client side should further query the operation status from the URL specified in this header.
Then there is Response 401, which is exactly what my Operation-Location contains. I do not understand why I'm getting a response 202 which looks like response 401.
I have tried to call the API with Python using at least three code versions that I found on the Internet that
all amount to the same, I found the code here :
Microsoft Emotion API for Python - upload video from memory
python-upload-video-from-memory
import httplib
import urllib
import base64
import json
import pandas as pd
import numpy as np
import requests
_url = 'https://api.projectoxford.ai/emotion/v1.0/recognizeInVideo'
_key = '**********************'
_maxNumRetries = 10
paramsPost = urllib.urlencode({'outputStyle' : 'perFrame', \
'file':'C:/path/to/file/file.mp4'})
headersPost = dict()
headersPost['Ocp-Apim-Subscription-Key'] = _key
headersPost['content-type'] = 'application/octet-stream'
jsonGet = {}
headersGet = dict()
headersGet['Ocp-Apim-Subscription-Key'] = _key
paramsGet = urllib.urlencode({})
responsePost = requests.request('post', _url + "?" + paramsPost, \
data=open('C:/path/to/file/file.mp4','rb').read(), \
headers = headersPost)
print responsePost.status_code
videoIDLocation = responsePost.headers['Operation-Location']
print videoIDLocation
Note that changing _url = 'https://api.projectoxford.ai/emotion/v1.0/recognizeInVideo' to _url =
'https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognizeInVideo' doesn't help.
However, afterwards I wait and run every half an hour:
getResponse = requests.request('get', videoIDLocation, json = jsonGet,\
data = None, headers = headersGet, params = paramsGet)
print json.loads(getResponse.text)['status']
The outcome has been 'Running' for hours and my video is only about half an hour long.
Here is what my Testing Console looks like Testing Console for Emotion API, Emotion Recognition in Video
Here I used another video that is about 5 minutes long and available on the internet. I found the video in a different usage example
https://benheubl.github.io/data%20analysis/fr/
that uses a very similar code, which again gets me a response status 202 Accepted and when opening the Operation-Location the subscription key is wrong
Here the code:
import httplib
import urllib
import base64
import json
import pandas as pd
import numpy as np
import requests
# you have to sign up for an API key, which has some allowances. Check the
API documentation for further details:
_url = 'https://api.projectoxford.ai/emotion/v1.0/recognizeinvideo'
_key = '*********************' #Here you have to paste your
primary key
_maxNumRetries = 10
# URL direction: I hosted this on my domain
urlVideo = 'http://datacandy.co.uk/blog2.mp4'
# Computer Vision parameters
paramsPost = { 'outputStyle' : 'perFrame'}
headersPost = dict()
headersPost['Ocp-Apim-Subscription-Key'] = _key
headersPost['Content-Type'] = 'application/json'
jsonPost = { 'url': urlVideo }
responsePost = requests.request( 'post', _url, json = jsonPost, data = None,
headers = headersPost, params = paramsPost )
if responsePost.status_code == 202: # everything went well!
videoIDLocation = responsePost.headers['Operation-Location']
print videoIDLocation
There are further examples on the internet and they all seem to work but replicating any of them never worked for me. Does anyone have any idea what could be wrong?

The Video Feature of Emotion API retires October 30th, so maybe you should change your procedure to screenshots anyways.
But for your question: The API returns you an URL where your results are accessible. You cannot open this URL in your browser, this will give you the notice of "invalid key", instead you need to call over python again this URL including your key.
I will post you my code how to get the score, I am using Python 3, so there might be some adjustments necessary. Only "tricky" point is getting the Operation ID, which is just the ID in the URL ( =location in my case) which leads to your request. Rest of the parameters like subscription key etc. is as before.
#extract operation ID from location-string
OID = location[67:]
bod = ""
try:
conn =
http.client.HTTPSConnection('westus.api.cognitive.microsoft.com')
conn.request("GET", "/emotion/v1.0/operations/"+OID+"?%s" %params, bod, headers)
response = conn.getresponse()
data = response.read()
print(data)
conn.close()
except Exception as e:
print("[Errno {0}] {1}".format(e.errno, e.strerror))

Did you verify your API call is working using curl? Always prototype calls using curl first. If it works in curl but not in Python, use Fiddler to observe the API request and response.

I also found an answer in the following link, all steps are explained:
https://gigaom.com/2017/04/10/discover-your-customers-deepest-feelings-using-microsoft-facial-recognition/

Create a functioning Response object

For testing purposes I'm trying to create a Response() object in python but it proves harder then it sounds.
i tried this:
from requests.models import Response
the_response = Response()
the_response.code = "expired"
the_response.error_type = "expired"
the_response.status_code = 400
but when I attempted the_response.json() i got an error because the function tries to get len(self.content) and a.content is null.
So I set a._content = "{}" but then I get an encoding error, so I have to change a.encoding, but then it fails to decode the content....
this goes on and on. Is there a simple way to create a Response object that's functional and has an arbitrary status_code and content?

That because the _content attribute on the Response objects (on python3) has to be bytes and not unicodes.
Here is how to do it:
from requests.models import Response
the_response = Response()
the_response.code = "expired"
the_response.error_type = "expired"
the_response.status_code = 400
the_response._content = b'{ "key" : "a" }'
print(the_response.json())

Create a mock object, rather than trying to build a real one:
from unittest.mock import Mock
from requests.models import Response
the_response = Mock(spec=Response)
the_response.json.return_value = {}
the_response.status_code = 400
Providing a spec ensures that the mock will complain if you try to access methods and attributes a real Response doesn't have.

Just use the responses library to do it for you:
import responses
#responses.activate
def test_my_api():
responses.add(responses.GET, 'http://whatever.org',
json={}, status=400)
...
This has the advantage that it intercepts a real request, rather than having to inject a response somewhere.

Another approach by using the requests_mock library, here with the provided fixture:
import requests
def test_response(requests_mock):
requests_mock.register_uri('POST', 'http://test.com/', text='data', headers={
'X-Something': '1',
})
response = requests.request('POST', 'http://test.com/', data='helloworld')
...

I keep getting this error the JSON object must be str, not 'bytes' [duplicate]

I've been trying to update a small Python library called libpynexmo to work with Python 3.
I've been stuck on this function:
def send_request_json(self, request):
url = request
req = urllib.request.Request(url=url)
req.add_header('Accept', 'application/json')
try:
return json.load(urllib.request.urlopen(req))
except ValueError:
return False
When it gets to this, json responds with:
TypeError: the JSON object must be str, not 'bytes'
I read in a few places that for json.load you should pass objects (In this case an HTTPResponse object) with a .read() attached, but it doesn't work on HTTPResponse objects.
I'm at a loss as to where to go with this next, but being that my entire 1500 line script is freshly converted to Python 3, I don't feel like going back to 2.7.

Facing the same problem I solve it using decode()
...
rawreply = connection.getresponse().read()
reply = json.loads(rawreply.decode())

I recently wrote a small function to send Nexmo messages. Unless you need the full functionality of the libpynexmo code, this should do the job for you. And if you want to continue overhauling libpynexmo, just copy this code. The key is utf8 encoding.
If you want to send any other fields with your message, the full documentation for what you can include with a nexmo outbound message is here
Python 3.4 tested Nexmo outbound (JSON):
def nexmo_sendsms(api_key, api_secret, sender, receiver, body):
"""
Sends a message using Nexmo.
:param api_key: Nexmo provided api key
:param api_secret: Nexmo provided secrety key
:param sender: The number used to send the message
:param receiver: The number the message is addressed to
:param body: The message body
:return: Returns the msgid received back from Nexmo after message has been sent.
"""
msg = {
'api_key': api_key,
'api_secret': api_secret,
'from': sender,
'to': receiver,
'text': body
}
nexmo_url = 'https://rest.nexmo.com/sms/json'
data = urllib.parse.urlencode(msg)
binary_data = data.encode('utf8')
req = urllib.request.Request(nexmo_url, binary_data)
response = urllib.request.urlopen(req)
result = json.loads(response.readall().decode('utf-8'))
return result['messages'][0]['message-id']

I met the problem as well and now it pass
import json
import urllib.request as ur
import urllib.parse as par
html = ur.urlopen(url).read()
print(type(html))
data = json.loads(html.decode('utf-8'))

Since you are getting a HTTPResponse, you can use Tornado.escape and its json_decode() to convert the JSON string into a dictionary:
from tornado import escape
body = escape.json_decode(body)
From the manual:
tornado.escape.json_decode(value)
Returns Python objects for the given JSON string.