Request not working for Python AWS Lambda - python

on a AWS Lambda function, I have a simple HTTP Request
import requests
def lambda_handler(event, context):
request_headers = {
'X-API-KEY': 'somekey',
'Content-Type':content,
'Host':'somehost',
}
resp = requests.get(some_url_here, headers=request_headers)
api_response = resp.json()
If I set the request requests.get(some_url_here, headers=request_headers, verify=False) I got no issues with the code and it works perfectly, but when I removed it it yields
Error encountered: HTTPSConnectionPool(host='x.xxx.xxx.xx', port=443): Max retries exceeded with url:some_url_here (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)')))
how to fix this issue?

Related

Problem while implementing my own OpenID provider and client

I am using authlib library(python) which is for client side and Oauth2 for the serverside.
after resp = oauth.get('userinfo') step my client side code stops and I get this error shown in the browser:
{ "message": "HTTPSConnectionPool(host='test.*****.com', port=8000): Max retries exceeded with url: /sample/user (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)')))" }
and here is my code for Oauth client side:
oauth = oauth.register(
'oauth',
client_id=env.get('client_id'),
client_secret=env.get('client_secret'),
access_token_url=env.get('access_token_url'),
access_token_params = None,
authorize_url=env.get('authorize_url'),
authorize_params=None,
api_base_url=env.get('api_base_url'),
userinfo_endpoint = env.get('userinfo_endpoint'),
client_kwargs={
'scope': 'openid',
},
)
#app.route('/login')
def login():
return oauth.authorize_redirect(redirect_uri='http://localhost:3000/callback')
#app.route('/callback')
def callback_handling():
oauth.authorize_access_token()
resp = oauth.get('userinfo')
userinfo = resp.json()
print('resp: ',vars(resp))
return redirect('http://localhost:3000/dashboard')
What could be the possible problem? where should I check? server side or client side? Do you have any clue?

Python Geopy with Google Maps API SSL Error

I found the following link:
https://towardsdatascience.com/pythons-geocoding-convert-a-list-of-addresses-into-a-map-f522ef513fd6
It shows a quick walk through on how to use Google Maps API to get latitude/longitude. However, when I use the provided code I get an SSL error. I have a working API key as I can get the URL to work that is produced from the second code set below.
Code:
from geopy.geocoders import GoogleV3
AUTH_KEY = "HIDDEN"
geolocator = GoogleV3(api_key=AUTH_KEY)
print(geolocator.geocode("1 Apple Park Way, Cupertino, CA").point) #Apple
Error:
HTTPSConnectionPool(host='maps.googleapis.com', port=443): Max retries exceeded with url: /maps/api/geocode/json?address=1+Apple+Park+Way%2C+Cupertino%2C+CA&key=HIDDEN (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)')))
I also tried using the following:
Code:
import requests
import json
import urllib
# https://developers.google.com/maps/documentation/geocoding/intro
base_url= "https://maps.googleapis.com/maps/api/geocode/json?"
AUTH_KEY = HIDDEN
# set up your search parameters - address and API key
parameters = {"address": "1 Apple Park Way, Cupertino, CA",
"key": AUTH_KEY}
# urllib.parse.urlencode turns parameters into url
print(f"{base_url}{urllib.parse.urlencode(parameters)}")
r = requests.get(f"{base_url}{urllib.parse.urlencode(parameters)}")
I get the exact same error. Oddly though the URL produced by print(f"{base_url}{urllib.parse.urlencode(parameters)}") is usable when I click on it.

Python3 reques with cert and key "verified fail"

I'm trying to request data from endpoints, I can do that with curl -k --key a-key.pem --cert a.pem https://<endpoint>
But when I using python3 to do that, I failed every time
Examples:
With curl:
root#control-plane-0:~# curl -k --key /etc/kubernetes/a-key.pem --cert /etc/kubernetes/a.pem https://127.0.0.1:6443/api/
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.0.31.2:6443"
}
]
}
With python:
(code):
from flask import Flask, render_template
import requests
from ast import literal_eval
app = Flask(__name__)
#app.route('/metrics')
def metrics():
data = requests.get("https://127.0.0.1:6443/api/, cert=('/etc/kubernetes/a.pem', '/etc/kubernetes/a-key.pem'))
print(data)
return data
if __name__ == '__main__':
app.run(host='0.0.0.0',port="5001", debug=True)
Result:
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=6443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1121)')))
Are there any problems with requests libs ?
I can't find a way to make it work
Have you got your certificates verified using
openssl verify -CAfile your-cert.pm
I got the same Problem and solved by using full-chain certificates.please see your certificate contains fullchain(root,intermediate).
And you can try like
import requests
test=request.get("url",verify="certificate-with-path")

How can I retrieve this file online with python? SSL error

So I have been trying to download the dataset from this page with python program.
The method I have tried using were requests and urllib.request.
A page I used as reference to solve the SSL error but didnt work...
My code here:
import pandas as pd
import requests
import shutil
# 2017 School Quality Report
FileLink = 'https://data.cityofnewyork.us/api/views/cxrnzyvb/files/35e2893e-75ed-4449-8e7e-d6360a3386a1?download=true&filename=2017_School_Quality_Report_DD.xlsx'
requests.packages.urllib3.disable_warnings()
response = requests.get(FileLink,verify='gd_bundle-g2-g1.crt', auth=('user', 'pass'),stream = True)
response.raw.decode_content = True
with open("2017_School_Quality_Report_DD.xlsx", 'wb') as f:
shutil.copyfileobj(response.raw, f)
#import urllib.request
#urllib.request.urlretrieve(FileLink, '2017_School_Quality_Report_DD.xlsx')
data = pd.read_excel('2017_School_Quality_Report_DD.xlsx')
print(data.sheet_names)
There is this error message which I don't know what to do to solve:
SSLError: HTTPSConnectionPool(host='data.cityofnewyork.us',
port=443): Max retries exceeded with url: /api/views/cxrn-
zyvb/files/35e2893e-75ed-4449-8e7e-d6360a3386a1?
download=true&filename=2017_School_Quality_Report_DD.xlsx (Caused by
SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:777)'),))
Please kindly let me know how I can solve the error or show me how you would do this task. I am fairly new to python. Thank you.
NOTE: found solution on this page which worked for me

python Requests SSL ERROR (certificate verify failed)

I have generated following self-signed certificates for my server and client.
I have created ca.crt & ca.key. Using ca.crt & ca.key, I have created server.crt, server.key for server and client.crt, client.key for client respectively.
I am using python requests library as client. Below is the code snippet:
import json
import requests
cert = ("/home/tests/certs/client.crt",
"/home/tests/certs/client.key")
class TestCart():
def test_cart(self, **kwargs):
url = "https://192.168.X.Y/cart"
cart_data = {
'id': kwargs.get('id'),
'items': kwargs.get('items')
}
req_data = json.dumps(cart_data)
resp = requests.post(url,
data=req_data,
verify="/home/certs/ca.cert",
cert=cert)
print resp.text
if __name__ == '__main__':
t_cart = TestCart()
data = {'id': 'ba396e79-0f0f-4952-a931-5a528c9ff72c', 'items': []}
t_cart.test_cart(**data)
This gives exception:
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.X.Y',
port=443): Max retries exceeded with url: /cart (Caused by
SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed (_ssl.c:590)'),))
If I use verify=False, code works, but I want to verify. What should be the value of verify in my request ?
It is highly recommended to have a deeper look at the excellent documentation for requests. It has a special chapter about SSL Cert Validation which explains:
You can pass verify the path to a CA_BUNDLE file or directory with certificates of trusted CAs:
>>> requests.get('https://github.com', verify='/path/to/certfile')
Assuming that your server certificate was signed by your ca.crt you should use this for the verify parameter.
EDIT: based on the discussion it looks like that CA and server certificate used the same subject. This means that the certificate validation assumes that this is a self-signed certificate which thus results in an certificate validation error.

Categories