I need to write a simple test script for rest get using python. What I have is:
import request
url = 'http://myurl......net'
headers ={'content-type':'application/xml'}
r= requests.get(url,headers=headers)
so this give me the following SSL error:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed(_ssl.c:590)
So, i did some research and add " verify = False" to the end of my last line of code, but not I am stuck with: : InsecureRequestsWarning, Unverified request is been made. Addig certificate verification is strongly advised."
What to do to get rid of this message?
Related
I am trying to create a little Python script to send data to a server using the requests module in Python. To make it a bit more secure i want to use self signed certificates made in a program called XCA. When using the certificates in the browser everything works and is secure. When using Postman to send a request with the certificates it works as well. But when i created the Python script it seems to not work or can't get to the certificate.
I have tried to include the CA with the 'Verify' command as well as adding the certificates with the 'cert' command.
When using the 'Verify' command (as seen down below) with the CA i get the error message: Remote end closed connection without response' This message seems to appear everytime i add the CA to this script somehow.
When i use the cert command i get this error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))). I have searched for a sollution to this problem and it seemed that the CA should be include in the certifi 'cacert.pem' file. I have done this but then i also get the 'Remote disconect' message.
Code with 'verify':
import requests
import json
url = "https://ip/tapi"
payload = json.dumps ({"command" : "GetUserList"})
headers = {'content-type': 'application/json',
'X-TAPI': '',
'Authorization': 'Basic ',
'connection': 'keep-alive'}
r = requests.request("POST", url, headers=headers, data= payload, verify= 'Tbox_CA.crt')
print(r.text)
Code with 'cert':
import requests
import json
cert_file_path = 'HTTPS_client.crt'
key_file_path = 'HTTPS_client_key.pem'
url = "https://ip/tapi"
payload = json.dumps ({"command" : "GetUserList"})
headers = {'content-type': 'application/json',
'X-TAPI': '',
'Authorization': 'Basic ',
'connection': 'keep-alive'}
cert = cert_file_path, key_file_path
r = requests.request("POST", url, headers=headers, data= payload, cert=cert)
print(r.text)
When using the 'Verify' command (as seen down below) with the CA i get the error message: Remote end closed connection without response' This message seems to appear everytime i add the CA to this script somehow.
Which means that SSL works, the HTTP request was sent, but then the server closes the connection without sending a response. SSL is not the problem here and the certificate validation worked.
When i use the cert command i get this error message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))).
The cert argument is not useful here since it is about providing a client certificate if the server requests it. Only, your server does not request a client certificate so providing this argument is basically the same as not providing it. Since no useful CA is given the certificate validation works. Since SSL thus already failed it cannot even send the real HTTP request inside the SSL connection, i.e. the error you've received when giving a CA is just masked because this error comes earlier.
Why the request failed from Python but not from the browser or Postman is unknown. There might be some server side bot protection implemented, but this is just a guess since there is nothing known about the actual URL you access.
Whenever am calling an API from python am getting error :
'self._sslobj.do_handshake()
ss.SSLCertVerificationError:[SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get Local issuer certificate.
During handling of the above exception, another exception occurred:'
Here is the code:
import requests
url = 'some URL'
response = requests.request('GET', url, verify=False)
print(response.text.encode('utf8'))
The same I am doing from postman initially I got error as 'No required SSL Certificates was sent', later on I have added the certificate to Postman then am able to hit the API and successfully getting the response.
Could anyone please what I need to do to overcome the error.
I'm getting a
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>
When I try to validate captchas in my django project.
This is how i do it:
recaptcha_response = request.POST.get('g-recaptcha-response')
print(recaptcha_response)
url = 'https://www.google.com/recaptcha/api/siteverify'
values = {
'secret': settings.CAPTCHA_SECRET_KEY,
'response': recaptcha_response
}
data = urllib.parse.urlencode(values).encode()
req = urllib.request.Request(url, data=data)
response = urllib.request.urlopen(req) # It fails here
result = json.loads(response.read().decode())
print(result)
The site has a valid certificate, and it works on local. In the log i'm getting this:
Request Method: POST
Request URL: http://prod.xxxx.com/evalua
Which is weird because the site works in https. Its on kubernetes, could that be the problem? I really have no idea what the problem IS? I have the captcha keys correctly set up in de recaptcha admin console. And the certificate are not autosign. I use lets encrypt
Check how you build the container image for your app and if it has very old CA certificates in it. You can use something like ADD https://curl.haxx.se/ca/cacert.pem /etc/ssl/certs/cacert.pem to ensure you have the latest standard bundle. You can also switch to Requests and Certifi instead of urllib, as that embeds a copy of the current cert bundle and ensures it is used.
I want to crawl a given list by the Top-1-Million from Alexa, to check which website still offers acces via http:// an do not redirect to https://.
If the webpage does not redirect to a https:// Domain, it should be written into a csv file.
The Problem occurs, when I am adding a bunch of multiple URLs. Than I get two errors:
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1056
or
requests.exceptions.ConnectionError: HTTPConnectionPool(host='17ok.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed')
I have tried the opportunities mentioned in the following threads and documentation:
https://2.python-requests.org//en/latest/user/advanced/#ssl-cert-verification
Edit: the sample url: https://requestb.in raises a 404 error actually, probably does not exist even more (?)
Python Requests throwing SSLError
Python Requests: NewConnectionError
requests.exceptions.SSLError: HTTPSConnectionPool: (Caused by SSLError(SSLError(336445449, '[SSL] PEM lib (_ssl.c:3816)')))
and some other delivered solutions.
The option to set verify=False helps, when using it for few URLs, but not when using a List > 10 URLs, the program brakes. I tried my program on a Win10 machine as well as on Ubuntu 16.04.
As expected, its the same issue. I also tried the option using Sessions and installed the certificate library as sugested.
If I am just calling three pages like 'http://www.example.com', 'https://www.github.com' and 'http://www.python.org', its not a big deal and the delivered solutions. The Headache starts, when using a bunch of URLs from the Alexa List.
Here is my code, which is working, when using it for only 3-4 urls:
import requests
from requests.utils import urlparse
urls = ['http://www.example.com',
'http://bloomberg.com',
'http://github.com',
'https://requestbin.fullcontact.com/']
with open('G:\\Request_HEADER_Suite/dummy/http.csv', 'w') as f:
for url in urls:
r = requests.get(url, stream=True, verify=False)
parsed_url = urlparse(r.url)
print("URL: ", url)
print("Redirected to: ", r.url)
print("Status Code: ", r.status_code)
print("Scheme: ", parsed_url.scheme)
if parsed_url.scheme == 'http':
f.write(url + '\n')
I expect to crawl at least a list with 100 URLs. The code should write URLs which are accessible by http:// and do not redirect to https:// into a csv file or complementary database and ignore all URLs with https://.
Because it is working for few URLs, I would expectd a stable opportunity for a larger scan.
But 2 errors araise and break the program. Is it worthy to try a workaround using pytest? Any other suggestions? Thanks in advance.
EDIT:
This is a list, which will raise errors. Only for clarification, this list from a study based on the Alexa-Top-1-Million.
urls = ['http://www.example.com',
'http://bloomberg.com',
'http://github.com',
'https://requestbin.fullcontact.com/',
'http://51sole.com',
'http://58.com',
'http://9gag.com',
'http://abs-cbn.com',
'http://academia.edu',
'http://accuweather.com',
'http://addroplet.com',
'http://addthis.com',
'http://adf.ly',
'http://adhoc2.net',
'http://adobe.com',
'http://1688.com',
'http://17ok.com',
'http://17track.net',
'http://1and1.com',
'http://1tv.ru',
'http://2ch.net',
'http://360.cn',
'http://39.net',
'http://4chan.org',
'http://4pda.ru']
I double checked, the last time the errors starts with the url 17.ok.com. But I have also tried different lists with urls. Thanks for your support.
I have generated following self-signed certificates for my server and client.
I have created ca.crt & ca.key. Using ca.crt & ca.key, I have created server.crt, server.key for server and client.crt, client.key for client respectively.
I am using python requests library as client. Below is the code snippet:
import json
import requests
cert = ("/home/tests/certs/client.crt",
"/home/tests/certs/client.key")
class TestCart():
def test_cart(self, **kwargs):
url = "https://192.168.X.Y/cart"
cart_data = {
'id': kwargs.get('id'),
'items': kwargs.get('items')
}
req_data = json.dumps(cart_data)
resp = requests.post(url,
data=req_data,
verify="/home/certs/ca.cert",
cert=cert)
print resp.text
if __name__ == '__main__':
t_cart = TestCart()
data = {'id': 'ba396e79-0f0f-4952-a931-5a528c9ff72c', 'items': []}
t_cart.test_cart(**data)
This gives exception:
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.X.Y',
port=443): Max retries exceeded with url: /cart (Caused by
SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed (_ssl.c:590)'),))
If I use verify=False, code works, but I want to verify. What should be the value of verify in my request ?
It is highly recommended to have a deeper look at the excellent documentation for requests. It has a special chapter about SSL Cert Validation which explains:
You can pass verify the path to a CA_BUNDLE file or directory with certificates of trusted CAs:
>>> requests.get('https://github.com', verify='/path/to/certfile')
Assuming that your server certificate was signed by your ca.crt you should use this for the verify parameter.
EDIT: based on the discussion it looks like that CA and server certificate used the same subject. This means that the certificate validation assumes that this is a self-signed certificate which thus results in an certificate validation error.