I'm trying to fetch some urls via urllib and mechanize through my proxy.
With mechanize I try the following:
from mechanize import Browser
import re
br = Browser()
br.set_proxies({"http": "MYUSERNAME:*******#itmalsproxy.italy.local:8080"})
br.open("http://www.example.com/")
I get the following error:
httperror_seek_wrapper: HTTP Error 407: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied.
As the proxy, the username and the password are correct, what could be the problem?
Maybe the proxy is using NTLM authentication?
If that is the case, you can try using the NTLM Authorization Proxy Server (see also this answer).
you might get more info from the response headers
print br.response().info()
When your web browser uses proxy server to surf the Web from within your local
network your may be required to authenticate youself to use proxy. Google ntlmaps.
Related
Trying to call a URL and get an HTML table from an internal server on network.
When entering a URL into a browser, the page and table load.
When calling URL in Python using Requests, URLLIB, and Pandas, a Network Error (dns_unresolved_hostname) is returned.
I have updated the hosts file to include Localhost. I unchecked "Don't use proxy server for local addresses". I can ping the server and it replies. I have changed the server name to the IP address and tried removing the port in the URL. No idea here.
import requests
r = requests.get('http://fc12rp01:8000/bin/ReportCall.py?rule=N2')
print(r.content)
Returns:
<HTML><HEAD>...........<big>Network Error (dns_unresolved_hostname)
Proxy Settings:
Host file settings:
I'm trying to fetch some data from Instagram and to prevent IP limitation I want to use AWS API Gateway proxies. I'm using a python lib requests_ip_rotator to manage my gateway and it works well, my requests on few website get a 200. But when I make a request on Instagram, all my requests are redirected with a HTTP 301 Response.
Here is my code, pretty simple, you can remove the mount to check that the request works well without the gateway setup.
import requests
from requests_ip_rotator import ApiGateway
gateway = ApiGateway("https://www.instagram.com/", regions=['eu-west-3'], access_key_id=AWS_ID,
access_key_secret=AWS_KEY)
gateway.start()
session = requests.Session()
session.mount("https://www.instagram.com/", gateway)
response = session.get("https://www.instagram.com/neymarjr/feed")
print(response)
gateway.shutdown()
Hope someone can help me !
If you need more information don't hesitate !
And feel free to give me solution for mass "scrapping" on Instagram =)
I am trying to test the latency of a proxy by pinging a site while using a proxy with a login. I know requests easily supports proxies and was wondering if there was a way to ping/test latency to a site through this. I am open to other methods as well, as long as they support a proxy with a login. Here is an example of my proxy integration with requests
import requests
proxy = {'https' : 'https://USER:PASS#IP:PORT'}
requests.get('https://www.google.com/', proxy=proxy)
How can I make a program to test the latency of a proxy with a login to a site?
I am using python request library to access the soap requests. And it was working fine. As there is change in our domain structure. I could not access the url, it always prompting me to enter the credentials.
I am using below code to access the url earlier using requests.
program_list_response = requests.get(program_list_path,
data=self.body, headers=self.headers)
How to pass the authentication in background using requests?
You can use the Authentication feature for that in order to provide the credentials for the link that you want to access.
For an eg:
You can pass the username and password by using the below format:
requests.get('https://website.com/user', auth=('user', 'pass'))
For more details I would recommend the official docs.
For handling the Windows authentication then I would recommend the Requests-NTLM.
For eg:
import requests
from requests_ntlm import HttpNtlmAuth
requests.get("http://ntlm_protected_site.com",auth=HttpNtlmAuth('domain\\username','password'))
I am using a proxy server to connect to several target servers. Some of the target servers expect http and others expect https. My http requests work swimmingly, but urllib2 ignores the proxy handler on the https requests and sends the requests directly to the target server.
I've tried a number of different things but here is one reasonably concise attempt:
import urllib2
cookie_handler = urllib2.HTTPCookieProcessor (cookielib.LWPCookieJar())
proxies = {'http': 'http://123.456.78.9/',
'https': 'http://123.45.78.9/'}
proxy_handler = urllib2.ProxyHandler (proxies)
url_opener = urllib2.build_opener (proxy_handler, cookie_handler)
request = urllib2.Request ('https://example.com')
response = url_opener.open (request)
I understand that urllib2 has had the ability to send https requests to a proxy server since Python 2.6.3, but I can't seem to get it to work. I'm using 2.7.3.
Thanks for any advice you can offer.
UPDATE: The code above does work. I'm not certain why it wasn't working when I asked this question. Most likely, I had a typo in the https proxy URL.