We Keep Coding

urllib.error.URLError: <urlopen error Tunnel connection failed: 403 URLBlocked>

i am trying to scrap weather and pollution data from openweathermap through python,
i am using urllib and the URL i try to access is HTTPS, here is my code :
api_key = self.dict_config["api_key"]
urlweather = self.dict_config["weather_url"] % (lat, lon, dt, api_key)
request=urllib.request.Request(urlweather)
request.add_header('client_id','xxxxxxxxxxxxx')
request.add_header('client_secret','xxxxxxxxxxxxx')
request.add_header('User-Agent', 'Mozilla/6.0')
proxy='xxxxxxxxxxxxx:8080'
request.set_proxy(proxy, 'https')
with urllib.request.urlopen(request, context=self.gcontext) as response:
weather = response.read().decode()
return weather
but with this code i have a urllib.error.URLError: <urlopen error Tunnel connection failed: 403 URLBlocked>
i've tried a to do the same but with urllib.request.opener not using a proxy try to pass my headers in an other way, not using user agent etc.. but nothing works.
interesting point : when i run this shell command on the same server it works :
curl -k --location --request GET 'https://xxxxxxxxxxxxx/owm/v1/data/2.5/onecall/timemachine?lat=51.2&lon=6.4&dt=1667390504&appid=xxxxxxxxxxxxx&units=metric' --header 'client_id: xxxxxxxxxxxxx' --header 'client_secret: xxxxxxxxxxxxx'
(i've obviously tested if my URL was correct when building it in the code)
So if you have any ideas please let me know !

Certificates won't work in Python requests

I am trying to create a little Python script to send data to a server using the requests module in Python. To make it a bit more secure i want to use self signed certificates made in a program called XCA. When using the certificates in the browser everything works and is secure. When using Postman to send a request with the certificates it works as well. But when i created the Python script it seems to not work or can't get to the certificate.
I have tried to include the CA with the 'Verify' command as well as adding the certificates with the 'cert' command.
When using the 'Verify' command (as seen down below) with the CA i get the error message: Remote end closed connection without response' This message seems to appear everytime i add the CA to this script somehow.
When i use the cert command i get this error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))). I have searched for a sollution to this problem and it seemed that the CA should be include in the certifi 'cacert.pem' file. I have done this but then i also get the 'Remote disconect' message.
Code with 'verify':
import requests
import json
url = "https://ip/tapi"
payload = json.dumps ({"command" : "GetUserList"})
headers = {'content-type': 'application/json',
'X-TAPI': '',
'Authorization': 'Basic ',
'connection': 'keep-alive'}
r = requests.request("POST", url, headers=headers, data= payload, verify= 'Tbox_CA.crt')
print(r.text)
Code with 'cert':
import requests
import json
cert_file_path = 'HTTPS_client.crt'
key_file_path = 'HTTPS_client_key.pem'
url = "https://ip/tapi"
payload = json.dumps ({"command" : "GetUserList"})
headers = {'content-type': 'application/json',
'X-TAPI': '',
'Authorization': 'Basic ',
'connection': 'keep-alive'}
cert = cert_file_path, key_file_path
r = requests.request("POST", url, headers=headers, data= payload, cert=cert)
print(r.text)

When using the 'Verify' command (as seen down below) with the CA i get the error message: Remote end closed connection without response' This message seems to appear everytime i add the CA to this script somehow.
Which means that SSL works, the HTTP request was sent, but then the server closes the connection without sending a response. SSL is not the problem here and the certificate validation worked.
When i use the cert command i get this error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))).
The cert argument is not useful here since it is about providing a client certificate if the server requests it. Only, your server does not request a client certificate so providing this argument is basically the same as not providing it. Since no useful CA is given the certificate validation works. Since SSL thus already failed it cannot even send the real HTTP request inside the SSL connection, i.e. the error you've received when giving a CA is just masked because this error comes earlier.
Why the request failed from Python but not from the browser or Postman is unknown. There might be some server side bot protection implemented, but this is just a guess since there is nothing known about the actual URL you access.

Client certificate works in curl but not in Python

Using curl I can connect to a server that needs specific certificate.
curl -E ./file.crt.pem --key ./file.key.pem -k https://server.url
curl version: 7.29.0
But when using Python's requests library, I get an error:
import requests
cert_file_path = "file.crt.pem"
key_file_path = "file.key.pem"
cert = (cert_file_path, key_file_path)
url = 'https://server.url'
r = requests.post(url, cert=cert, verify=False)
Error:
SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert unknown ca')])"))
Python version: v3.7
What am I missing?

A comment on this answer helped me figure this out.
Update your code as such:
import requests
cert_file_path = "file.crt.pem"
key_file_path = "file.key.pem"
cert = (cert_file_path, key_file_path)
url = 'https://server.url'
r = requests.post(url, cert=cert, verify="path/to/ca_public_keys.pem") # replace with your file
I'm assuming you're using a self-signed certificate, so you need to specify the .pem file containing the public certificates of the CA that issued your self-signed certificate. Make sure to include the intermediate certificates, otherwise the requests library will throw the tlsv1 alert unknown ca error.
You can check the issuer of your client certificate by typing openssl x509 -noout -in file.crt.pem -issuer in a terminal.

Request module checks environmental variable REQUESTS_CA_BUNDLE for cert file. So just do this
export REQUESTS_CA_BUNDLE=/absolute/path/to/your/file.crt.pem
Your python code will simply be:
import requests
url = 'https://server.url'
r = requests.post(url)
print(r.text)

python Requests SSL ERROR (certificate verify failed)

I have generated following self-signed certificates for my server and client.
I have created ca.crt & ca.key. Using ca.crt & ca.key, I have created server.crt, server.key for server and client.crt, client.key for client respectively.
I am using python requests library as client. Below is the code snippet:
import json
import requests
cert = ("/home/tests/certs/client.crt",
"/home/tests/certs/client.key")
class TestCart():
def test_cart(self, **kwargs):
url = "https://192.168.X.Y/cart"
cart_data = {
'id': kwargs.get('id'),
'items': kwargs.get('items')
}
req_data = json.dumps(cart_data)
resp = requests.post(url,
data=req_data,
verify="/home/certs/ca.cert",
cert=cert)
print resp.text
if __name__ == '__main__':
t_cart = TestCart()
data = {'id': 'ba396e79-0f0f-4952-a931-5a528c9ff72c', 'items': []}
t_cart.test_cart(**data)
This gives exception:
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.X.Y',
port=443): Max retries exceeded with url: /cart (Caused by
SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed (_ssl.c:590)'),))
If I use verify=False, code works, but I want to verify. What should be the value of verify in my request ?

It is highly recommended to have a deeper look at the excellent documentation for requests. It has a special chapter about SSL Cert Validation which explains:
You can pass verify the path to a CA_BUNDLE file or directory with certificates of trusted CAs:
>>> requests.get('https://github.com', verify='/path/to/certfile')
Assuming that your server certificate was signed by your ca.crt you should use this for the verify parameter.
EDIT: based on the discussion it looks like that CA and server certificate used the same subject. This means that the certificate validation assumes that this is a self-signed certificate which thus results in an certificate validation error.

python SSL error when using request

I need to write a simple test script for rest get using python. What I have is:
import request
url = 'http://myurl......net'
headers ={'content-type':'application/xml'}
r= requests.get(url,headers=headers)
so this give me the following SSL error:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed(_ssl.c:590)
So, i did some research and add " verify = False" to the end of my last line of code, but not I am stuck with: : InsecureRequestsWarning, Unverified request is been made. Addig certificate verification is strongly advised."
What to do to get rid of this message?