I'm using the client.Client class from gremlin_python.driver to connect to AWS Neptune. See the following
def _prepare_request(method, url, *, data=None, params=None, headers=None, service='neptune-db'):
_ = requests.Session()
request = requests.Request(method=method, url=url, data=data, params=params, headers=headers)
credentials = Session().get_credentials()
frozen_creds = credentials.get_frozen_credentials()
req = AWSRequest(method=method, url=url, data=data, params=params, headers=headers)
SigV4Auth(frozen_creds, service, os.environ['AWS_REGION']).add_auth(req)
prepared_iam_req = req.prepare()
request.headers = dict(prepared_iam_req.headers)
return request.prepare()
# https
http_protocol = 'https'
uri = f'{http_protocol}://{self.host}:{self.port}/gremlin'
request = _prepare_request('GET', uri)
# wss
ws_url = 'wss://{}:{}/gremlin'.format(self.host, self.port)
ws_request = httpclient.HTTPRequest(ws_url, headers=dict(request.headers))
self.conn = client.Client(ws_request, 'g')
Now my question how can I used the client.Client object from above to get a traversal object "g".
There is a similar example at https://pypi.org/project/gremlinpython/#description showing this. But I can't use the DriverRemoteConnection in the above code.
>>> from gremlin_python.process.anonymous_traversal import traversal
>>> from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
>>> g = traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))
Adding an answer in case others find this discussion.
When working with Amazon Neptune using Python, there is a nice open source set of utilities located here that make it easy to create SigV4 signed requests and perform many other common tasks.
Several of the examples are also useful with any Gremlin Server and not just Neptune.
Related
I'm trying to work with a third party API and I am having problems with sending the request when using the requests or even urllib.request.
Somehow when I use http.client I am successful sending and receiving the response I need.
To make life easier for me, I created an API class below:
class API:
def get_response_data(self, response: http.client.HTTPResponse) -> dict:
"""Get the response data."""
response_body = response.read()
response_data = json.loads(response_body.decode("utf-8"))
return response_data
The way I use it is like this:
api = API()
rest_api_host = "api.app.com"
connection = http.client.HTTPSConnection(rest_api_host)
token = "my_third_party_token"
data = {
"token":token
}
payload = json.loads(data)
headers = {
# some headers
}
connection.request("POST", "/some/endpoint/", payload, headers)
response = connection.getresponse()
response_data = api.get_response_data(response) # I get a dictionary response
This workflow works for me. Now I just want to write a test for the get_response_data method.
How do I instantiate a http.client.HTTPResponse with the desired output to be tested?
For example:
from . import API
from unittest import TestCase
class APITestCase(TestCase):
"""API test case."""
def setUp(self) -> None:
super().setUp()
api = API()
def test_get_response_data_returns_expected_response_data(self) -> None:
"""get_response_data() method returns expected response data in http.client.HTTPResponse"""
expected_response_data = {"token": "a_secret_token"}
# I want to do something like this
response = http.client.HTTPResponse(expected_response_data)
self.assertEqual(api.get_response_data(response), expected_response_data)
How can I do this?
From the http.client docs it says:
class http.client.HTTPResponse(sock, debuglevel=0, method=None, url=None)
Class whose instances are returned upon successful connection. Not instantiated directly by user.
I tried looking at socket for the sock argument in the instantiation but honestly, I don't understand it.
I tried reading the docs in
https://docs.python.org/3/library/http.client.html#http.client.HTTPResponse
https://docs.python.org/3/library/socket.html
Searched the internet on "how to test http.client.HTTPResponse" but I haven't found the answer I was looking for.
I am trying to port the "Prepared Request" example from this link: https://docs.python-requests.org/en/latest/user/advanced/ using httpx AsnycClient.
from requests import Request, Session
s = Session()
req = Request('GET', url, data=data, headers=headers)
prepped = s.prepare_request(req)
# do something with prepped.body
prepped.body = 'Seriously, send exactly these bytes.'
# do something with prepped.headers
prepped.headers['Keep-Dead'] = 'parrot'
resp = s.send(prepped,
stream=stream,
verify=verify,
proxies=proxies,
cert=cert,
timeout=timeout
)
print(resp.status_code)
But I could not find any method to prepare the request in the httpx async lib.
In particular, it seems that "prepare_request" method is not implemented.
Can anyone tell me how do I prepare requests using AsyncClient?
Any hint is much appreciated.
One could use httpx.Request class to write an prepare_request factory.
import httpx
def prepare_request(method, url, **kwargs):
# a very simple factory
return httpx.Request(method, url, **kwargs)
request = prepare_request("GET", "https://www.google.com")
client = httpx.Client()
response = client.send(request)
print(response, response.num_bytes_downloaded )
used sync client for demonstration only
I have successfully implemented OAuth1 with the regular requests module like this:
import requests
from requests_oauthlib import OAuth1
oauth = OAuth1(client_key=oauth_cred["consumer_key"], client_secret=oauth_cred["consumer_secret"], resource_owner_key=oauth_cred["access_token"], resource_owner_secret=oauth_cred["access_token_secret"])
session = requests.Session()
session.auth = oauth
When trying to transfer this to aiohttp, I have not been able to get it to work. Substituting aiohttp.ClientSession() for requests.Session() gives me {'errors': [{'code': 215, 'message': 'Bad Authentication data.'}]}.
I have looked at some solutions on the internet like https://github.com/klen/aioauth-client, but this seems to be a different approach. I just want it to function exactly like in my example above.
I tried
import aiohttp
from aioauth_client import TwitterClient
oauth = TwitterClient(consumer_key=oauth_cred["consumer_key"], consumer_secret=oauth_cred["consumer_secret"], oauth_token=oauth_cred["access_token"], oauth_token_secret=oauth_cred["access_token_secret"])
session = aiohttp.ClientSession()
session.auth = oauth
but I got the same error.
How can I get this to work?
Using oauthlib:
import oauthlib.oauth1, aiohttp, asyncio
async def main():
# Create the Client. This can be reused for multiple requests.
client = oauthlib.oauth1.Client(
client_key = oauth_cred['consumer_key'],
client_secret = oauth_cred['consumer_secret'],
resource_owner_key = oauth_cred['access_token'],
resource_owner_secret = oauth_cred['access_token_secret']
)
# Define your request. In my code I'm POSTing so that's what I have here,
# but if you're doing something different you'll need to change this a bit.
uri = '...'
http_method = 'POST'
body = '...'
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
# Sign the request data. This needs to be called for each request you make.
uri,headers,body = client.sign(
uri = uri,
http_method = http_method,
body = body,
headers = headers
)
# Make your request with the signed data.
async with aiohttp.ClientSession() as session:
async with session.post(uri, data=body, headers=headers, raise_for_status=True) as r:
...
# asyncio.run has a bug on Windows in Python 3.8 https://bugs.python.org/issue39232
#asyncio.run(main())
asyncio.get_event_loop().run_until_complete(main())
The oauthlib.oauth1.Client constructor takes a bunch more parameters too if you need them (for basic use you don't). The official documentation isn't very thorough, but the doc comment on the method itself is pretty good.
The doc comment on the Client.sign method has more information about the parameters it takes.
How do you use Bitbucket's 2.0 API to decline a pull request via Python?
According to their documentaion, it should be something like:
import requests
kwargs = {
'username': MY_BITBUCKET_ACCOUNT,
'repo_slug': MY_BITBUCKET_REPO,
'pull_request_id': pull_request_id
}
url = 'https://api.bitbucket.org/2.0/repositories/{username}/{repo_slug}/pullrequests/{pull_request_id}/decline'.format(**kwargs)
headers = {'Content-Type': 'application/json'}
response = requests.post(url, auth=(USERNAME, PASSWORD), headers=headers)
However, this fails with response.text simply saying "Bad Request".
This similar code works for me with their other API endpoints, so I'm not sure why the decline method is failing.
What am I doing wrong?
You have to authenticate with Oath. I wrote a wrapper for making these requests. Here is a simple example that works. The only thing I couldn't figure out was how to add a reason it was declined. I ended up making a request before I declined the PR that added a comment on why it was declined.
import os
from oauthlib.oauth2 import BackendApplicationClient
from requests_oauthlib import OAuth2Session
class Bitbucket(object):
def __init__(self, client_id, client_secret, workplace, repo_slug):
self.workplace = workplace # username or company username
self.repo_slug = repo_slug
self.token_url = 'https://bitbucket.org/site/oauth2/access_token'
self.api_url = 'https://api.bitbucket.org/2.0/'
self.max_pages = 10
self.client = BackendApplicationClient(client_id=client_id)
self.oauth = OAuth2Session(client=self.client)
self.oauth.fetch_token(
token_url=self.token_url,
client_id=client_id,
client_secret=client_secret
)
def get_api_url(self, endpoint):
return f'{self.api_url}repositories/{self.workplace}/{self.repo_slug}/{endpoint}'
bitbucket = Bitbucket(os.environ['BITBUCKET_KEY'], os.environ['BITBUCKET_SECRET'], workplace='foo', repo_slug='bar')
pr_id = 1234
resp = bitbucket.oauth.post(f"{bitbucket.get_api_url('pullrequests')}/{pr_id}/decline")
if resp.status_code == 200:
print('Declined')
else:
print('Someting went wrong.')
I'm little confused about requests module, especially proxies.
From documentation:
PROXIES
Dictionary mapping protocol to the URL of the proxy (e.g. {‘http’:
‘foo.bar:3128’}) to be used on each Request.
May there be more proxies of one type in the dictionary? I mean is it possible to put there list of proxies and requests module will try them and use only those which are working?
Or there can be only one proxy address for example for http?
Using the proxies parameter is limited by the very nature of a python dictionary (i.e. each key must be unique).
import requests
url = 'http://google.com'
proxies = {'https': '84.22.41.1:3128',
'http': '185.26.183.14:80',
'http': '178.33.230.114:3128'}
if __name__ == '__main__':
print url
print proxies
response = requests.get(url, proxies=proxies)
if response.status_code == 200:
print response.text
else:
print 'Response ERROR', response.status_code
outputs
http://google.com
{'http': '178.33.230.114:3128', 'https': '84.22.41.1:3128'}
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."
...more html...
As you can see, the value of the http protocol key in the proxies dictionary corresponds to the last encountered in its assignment (i.e. 178.33.230.114:3128). Try swapping the http entries around.
So, the answer is no, you cannot specify multiple proxies for the same protocol using a simple dictionary.
I have tried using an iterable as a value, which would make sense to me
proxies = {'https': '84.22.41.1:3128',
'http': ('178.33.230.114:3128', '185.26.183.14:80', )}
but with no luck, it produces an error
Well, actually you can, I've done this with a few lines of code and it works pretty well.
import requests
class Client:
def __init__(self):
self._session = requests.Session()
self.proxies = None
def set_proxy_pool(self, proxies, auth=None, https=True):
"""Randomly choose a proxy for every GET/POST request
:param proxies: list of proxies, like ["ip1:port1", "ip2:port2"]
:param auth: if proxy needs auth
:param https: default is True, pass False if you don't need https proxy
"""
from random import choice
if https:
self.proxies = [{'http': p, 'https': p} for p in proxies]
else:
self.proxies = [{'http': p} for p in proxies]
def get_with_random_proxy(url, **kwargs):
proxy = choice(self.proxies)
kwargs['proxies'] = proxy
if auth:
kwargs['auth'] = auth
return self._session.original_get(url, **kwargs)
def post_with_random_proxy(url, *args, **kwargs):
proxy = choice(self.proxies)
kwargs['proxies'] = proxy
if auth:
kwargs['auth'] = auth
return self._session.original_post(url, *args, **kwargs)
self._session.original_get = self._session.get
self._session.get = get_with_random_proxy
self._session.original_post = self._session.post
self._session.post = post_with_random_proxy
def remove_proxy_pool(self):
self.proxies = None
self._session.get = self._session.original_get
self._session.post = self._session.original_post
del self._session.original_get
del self._session.original_post
# You can define whatever operations using self._session
I use it like this:
client = Client()
client.set_proxy_pool(['112.25.41.136', '180.97.29.57'])
It's simple, but actually works for me.