I'm having this issue while running the script:
(I'm using Spyder to build my script, but I'm trying at Jupyter Notebook and I'm getting the same error)
#STEP 3.8 - Get the URL request
LIMIT = 100
radius = 50
url = 'https://api.foursquare-com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
CLIENT_ID, CLIENT_SECRET, VERSION, neighbor_lat, neighbor_long, radius, LIMIT)
#STEP 3.9 - Get request and examinate the result
results = requests.get(url).json()
print(results)
ConnectionError: HTTPSConnectionPool(host='api.foursquare-com', port=443): Max retries exceeded with url: /v2/venues/explore?&client_id=xxx&client_secret=xxx&v=20180605&ll=43.806686299999996,-79.19435340000001&radius=500&limit=100 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
Try to add headers parameters in your request.get.
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'}
page = requests.get(url, headers=headers)
Try using the exceptions
results=" "
while results==" ":
try:
results = requests.get(url).json()
except:
time.sleep(50)
continue
This is common error, after doing SSL to your VPS server, it only finds the url with https://domanName.com
in this case check you connection in you code File or .env file and
change connections from
http://domainName.com -- to -- https://domainName.com
I hope this will solve your problem
Thanks.
Related
My code is below:
import urllib.request
import urllib.parse
from lxml import etree
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
url = url + urllib.parse.quote(word)
print('search: ', word)
req = urllib.request.Request(url=url, headers=HEADERS, method='GET')
response = urllib.request.urlopen(req)
text = response.read().decode('utf-8')
This works fine, but after about 400 requests, I got this error:
urllib.error.URLError: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1076)>
What might be the cause of this?
The connection is being rejected during the TLS handshake for some reason. It could just be something transient (and a retry would work) or it may be due to a mismatch due to TLS protocol versions or ciphers. Or it could also be for other reasons that aren't immediately obvious, like a block list, or a broken server etc.
Adding some detection to your code that retries a few times then skips is probably a good general approach. Something like:
for i in range(0,3):
try:
# CONNECT
except:
continue
break
If you want to understand exactly why this particular URL is failing, the easiest solution is to download a copy of wireshark and see what is happening on the wire. There will likely be a TLS error, and possibly an alert code and message that gives more information.
I am try to access a site that is restricted to region using a VPN. The proxy works fine in browser but when proxy is enabled and I try to send request using
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Edg/100.0.1185.50',
}
data = {
'courtType': '',
'countyCode': '',
'cortCode': 'SW',
'caseNumber': '22SL-PR00157',
}
res = requests.post('https://www.courts.mo.gov/cnet/caseNoSearch.do', data=data, headers=headers)
I get following error
requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.courts.mo.gov', port=443): Max retries exceeded with url: /cnet/caseNoSearch.do (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)'))))
Is there any way I can send requests enabling VPN?
I have three versions that i tested: 3.10.2, 3.9.9 and 3.8.10 on different machines and even one online compiler. In all of them i did the following:
import requests
requests.get(url, proxies=proxies, headers=headers)
Testing in each parameter:
url:
"https://www.icanhazip.com"
"http://www.icanhazip.com"
proxies:
{"https": "223.241.0.250:3000", "http": "223.241.0.250:3000"}
{"https": "223.241.0.250:3000", "http": "223.241.0.250:3000"}
headers:
{'User-Agent': 'Chrome'}
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36"}
In all of them but the 3.10.2 I got this error:
ProxyError: HTTPSConnectionPool(host='www.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConn
ectionError('<urllib3.connection.HTTPSConnection object at 0x05694F40>: Failed to establish a new connection: [WinError 10060] A connection attempt failed beca
use the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond')))
On the 3.10.2 i got:
InvalidURL: Proxy URL had no scheme, should start with http:// or https://
But when i tried to put the proxies like this:
{"https": "223.241.0.250:3000", "http": "223.241.0.250:3000"}
It didnt work and it just showed my normal ip.
What am I missing? A normal requests works just fine but when I add the proxies it just doesnt work. This code was working fine a while back and now outputs this error I cant figure out why.
Try adding a scheme in the proxy address
{"https": "http://223.241.0.250:3000", "http": "http://223.241.0.250:3000"}
{"https": "http://223.241.0.250:3000", "http": "http://223.241.0.250:3000"}
I have this function to access a website.
def access_website(link):
try:
cert = requests.certs.where()
page = requests.get(link,
verify=cert,
headers={"User-Agent": "Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36"})
return page
except Exception as x:
print(x)
return ''
access_website('https://www.davita.com/-/media/davita/project/kidneycare/pdf/corporate-governance/dva-pay-equity-disclosure-32119-final.ashx?la=en-us&hash=E5E2F4F69620F3C0BB52FFE818ABCE6CD36BFA12')
But when I do this, I get the following error:
HTTPSConnectionPool(host='www.davita.com', port=443): Max retries exceeded with url: /-/media/davita/project/kidneycare/pdf/corporate-governance/dva-pay-equity-disclosure-32119-final.ashx?la=en-us&hash=E5E2F4F69620F3C0BB52FFE818ABCE6CD36BFA12 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
But I can easily access the site from the browser. All other links that I have do not raise this error.
How can I resolve this? I already tried by providing the certificate, but it still doesn't work
You can set the verify=False argument:
import requests
def access_website(link):
HEADERS = {
"User-Agent": "Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36"
}
try:
page = requests.get(link, verify=False, headers=HEADERS)
return page
except Exception as x:
print(x)
return ""
access_website(
"https://www.davita.com/-/media/davita/project/kidneycare/pdf/corporate-governance/dva-pay-equity-disclosure-32119-final.ashx?la=en-us&hash=E5E2F4F69620F3C0BB52FFE818ABCE6CD36BFA12"
)
I have deployed an AWS ec2 instance to use a proxy. I have edited the security policies and have allowed my machine to have access to the server. I am using port 22 for SSH, and port 4444 for the proxy. For some reason I still can not start a session using the proxy.
The code:
import requests
session = requests.Session()
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36'
headers = {'user-agent' : user_agent}
proxies = {
'http' : 'socks5h://ec2-ip-address-here.us-east-2.compute.amazonaws.com:4444',
'https' : 'socks5h://ec2-ip-address-here.us-east-2.compute.amazonaws.com:4444',
}
print(session.get('https://www.ipchicken.com/', headers=headers, proxies=proxies).content)
The error:
requests.exceptions.ConnectionError: SOCKSHTTPSConnectionPool(host='www.ipchicken.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x107a09048>: Failed to establish a new connection: [Errno 61] Connection refused'))
I'm not sure what I am doing wrong. I followed this video https://www.youtube.com/watch?v=HOL2eg0g0Ng for setting up the server. Thanks to all of those who reply in advance.
You need to be using socks5h:// for your http and https proxies.
I get this error on macOS when using socks5://.