We Keep Coding

python requests.get(url) times out but works in browser (chrome); how can I tailor the request headers for a certain host?

I am trying to download a file using the python requests module, my code works for some urls/hosts but I've come across one that does not work.
Based on other similar questions it may be related to the User-Agent request header, I have tried to remedy by adding the chrome user-agent but the connection still times out for this particular url (it does work for others).
I have tested opening the url in chrome browser (which works all OK) and inspecting the request headers, but I still can't figure out why my code is failing:
import requests
url = 'http://publicdata.landregistry.gov.uk/market-trend-data/house-price-index-data/Indices-2020-03.csv'
headers = {'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'}
session = requests.Session()
session.headers.update(headers)
response = session.get(url, stream=True)
# !!! code fails here for this particular url !!!
with open('test.csv', "wb") as fh:
for x in response.iter_content(chunk_size=1024):
if x: fh.write(x)
Update 2020-08-14
I have figured out what was wrong; on the instances where the code was working the urls were using https protocol. This url is http protocol, and my proxy settings were not configured for http only https. After providing a http proxy to requests my code did work as written.

The code you posted worked for me, it saved the file (129007 lines). It could be that the host is rate-limiting you, try again later to see if it works.
# count lines
$ wc -l test.csv
129007 test.csv
# inspect headers
$ head -n 4 test.csv
Date,Region_Name,Area_Code,Index
1968-04-01,Wales,W92000004,2.11932727
1968-04-01,Scotland,S92000003,2.108087275
1968-04-01,Northern Ireland,N92000001,3.300419757

You can disable requests' timeouts by passing timeout=None. Here is the official documentation: https://requests.readthedocs.io/en/master/user/advanced/#timeouts

Redirecting a webserver request to Akamaized hosted video url not working

I have a certain akamaized video url of form -
https://gcs-vimeo.akamaized.net/exp=<expiry-in-milliseconds>~acl=%2A%2F<9-digit-video-code>.mp4%2A~hmac=<64-char-alphanumeric>/vimeo-prod-skyfire-std-us/<2-digits>/<4-digits>/<1-digit>/<9-digits>/<same-9-digit-video-code>.mp4
I can download this url via Internet Download Manager (in Browser), curl, wget, and python-requests library. I can even play it directly if I put the url in the search bar.
My video tag looks like this -
<video src='/play/<random-string>' controls></video>
And my apache RewriteRule is this -
RewriteRule ^play/([a-z]+)$ <video-url>
My problem is that despite being able to download this video from IDM, curl, wget and python-requests, I am unable to redirect to this URL successfully. The video works fine if I write it directly in the `src' attribute, but not if I redirect as mentioned above.
The headers of the request when url specified directly in src attribute and video runs -
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Host:gcs-vimeo.akamaized.net
Range:bytes=0-
Referer:http://<domain>/<some-end-point>
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36
On successful load (by directly specifying the URL) I get either 200 OK or 206 Partial Content
On failure (by trying to run video via redirect) I get some (canceled) status.
Someone help.

Specifying proxies for http vs https sites when using Python Requests

I have a Python script to get a page using Requests. I need to use proxies to access the page. When I access an http page, it goes through the proxy but when I access an https page, it does not go through the proxy (I used logs to check this, as explained below). I have checked with the proxy service provider (proxymesh) and they said that their proxies can be used for https pages as well. Is there anything I need to change in the script when accessing https sites vs http sites?
My code is presented below. At the end of this question, I have included the log files generated for the http and https sites, which show that the proxy is used for http but not for https.
Any ideas will be really helpful.
import logging
import requests
#set up logging
logging.getLogger('').handlers = []
logging.basicConfig(
filename = "mylog_with_proxy.log", #in my code, the full path is specified
filemode="w",
level = logging.DEBUG)
#specify proxies and headers
proxies = {'http': 'http://fr.proxymesh.com:31280', 'https': 'http://fr.proxymesh.com:31280'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393',}
#the two URLs that I accessed. One is for an http site and the other one is for an https site. These sites are just examples of sites I need to access.
http_url = "http://docs.python-requests.org/en/master/user/quickstart/"
https_url = "https://www.haskell.org/happy/"
#get the page. I executed the script twice - once for http_url and the second time for https_url. Here, it shows http_url
r = requests.get(http_url, headers=headers, proxies=proxies, timeout=5)
r.raise_for_status()
The log files are as shown below:
When accessing the http site (that is, when running the script with http_url):
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): fr.proxymesh.com
DEBUG:requests.packages.urllib3.connectionpool:"GET http://docs.python-requests.org/en/master/user/quickstart/ HTTP/1.1" 200 None
When accessing the https site (that is, when running the script with https_url)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.haskell.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /happy/ HTTP/1.1" 200 None

pycurl is working in an unexpected way

I have written a code that visits a URL using pycurl. I have tor enabled.
The URL gets redirected to some other url.
Below is the code.
import pycurl
curl = pycurl.Curl()
curl.setopt(pycurl.URL, URL)
curl.setopt(pycurl.PROXY, '127.0.0.1')
curl.setopt(pycurl.PROXYPORT, 9050)
curl.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5_HOSTNAME)
curl.setopt(pycurl.USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0')
curl.perform()
It prints the expected html content.
But whenever there is a visit to a URL, there is an increment to a count somewhere else.
Now, when I run the script, I get the html content, but there is no increment in the count, but when the same html output is run in some online html rendering website(htmledit.squarefree.com/
), the count is incremented.
Any help to increment the count automatically, using the script itself?
Thanks.

Any kind of updation of some data on server when client visit their website is possibly done through javascript.
When some website content is loaded on client machine, it has got some javascript, which gets executed onto client's machine to notify the server. Now when the webpage is visited through browser, javascript are executed(if the browser is enabled to do it). But when the webpage is visited through curl, it can't execute javascript.
I managed to do it using dryscrape. Dryscrape uses http protocol. You can read here for work around to enable socks5 protocol for dryscrape.

How to check Proxy headers to check anonymity?

I'm trying to determine high anonymity proxies. Also called private/elite proxies. From a forum I've read this:
High anonymity Servers don't send HTTP_X_FORWARDED_FOR, HTTP_VIA and
HTTP_PROXY_CONNECTION variables. Host doesn't even know you are using
proxy server and of course it doesn't know your IP address.
A highly anonymous proxy will display the following information:
REMOTE_ADDR = Proxy's IP address
HTTP_VIA = blank
HTTP_X_FORWARDED_FOR = blank
So, how I can check for this headers in Python, to discard them as a HA Proxy ? I have tried to retrieve the headers for 20-30 proxies using the requests package, also with urllib, with the build-in http.client, with urllib2. But I didn't see these headers, never. So I should be doing something wrong...
This is the code I've used to test with requests:
proxies = {'http': 'http://176.100.108.214:3128'}
header = {'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.360',}
s = requests.session()
s.proxies = proxies
r = s.get('http://www.python.org', headers=header)
print(r.status_code)
print(r.request.headers)
print(r.headers)

It sounds like the forum post you're referring to is talking about the headers seen by the server on your proxied request, not the headers seen by the client on the proxied response.
Since you're testing with www.python.org as the server, the only way to see the headers it receives would be to have access to their logs. Which you don't.
But there's a simple solution: run your own HTTP server, make requests against that, and then you can see what it receives. (If you're behind a firewall or NAT that the proxy you're testing won't be able to connect to, you may have to get a free hosted server somewhere; if not, you can just run it on your machine.)
If you have no idea how to set up and configure a web server, Python comes with one of its own. Just run this script with Python 3.2+ (on your own machine, or an Amazon EC2 free instance, or whatever):
from http.server import HTTPServer, SimpleHTTPRequestHandler
class HeaderDumper(SimpleHTTPRequestHandler):
def do_GET(self):
try:
return super().do_GET()
finally:
print(self.headers)
server = HTTPServer(("", 8123), HeaderDumper)
server.serve_forever()
Then run that script with python3 in the shell.
Then just run your client script, with http://my.host.ip instead of http://www.python.org, and look at what the script dumps to the server's shell.