We Keep Coding

Redirect with no auth

According to the docs, it should be as simple as:
data = self.http_pool.urlopen('GET', file_url,
preload_content=False,
retries=max_download_retries)
request.add_unredirected_header(key, header)
Add a header that will not be added to a redirected request.
But I cannot seem to find any examples on how this can be achieved.
I am using the pyupdater to download updates from bitbucket and launch the newest version of exe. I am using this library to create a script that connects to bitbucket fine, but then it later redirects to Amazon with nauthorization: Basic <redacted>\r\n\r\n (this is bitbucket auth) meaning I get 'HTTP/1.1 400 Bad Request\r\n'. Amazon does not support basic auth. This should be easily solvable, but I cannot find much on this issue.
The solutions presented here, require Recreating each redirected request manually. This would become an ever-growing list and get tedious very quickly, if I had to do this for new file I uploaded. It also does not continue the rest of the script, but rather downloads to the same directory.
As this is how Pyupdater handles the downloads this is where the issue would likely be solved.
Line 366 of downloader.py:
data = self.http_pool.urlopen('GET', file_url,
preload_content=False,
retries=max_download_retries)
Any ideas on how to fix this so it no longer creates this error.
Full Error (ctrl f -> 400):
Python main.py
DEBUG:root:Version - 2.5.1
DEBUG:pyupdater.client:PyUpdater Version 2.5.1
Current version is 1.3
{'authorization': 'Basic <redacted>'}
DEBUG:pyupdater.client:Setting up directories...
DEBUG:pyupdater.client:Downloading key file
DEBUG:pyupdater.client.downloader:Url for request: https://api.bitbucket.org/2.0/repositories/ brofewfefwefewef/eee/downloads/keys.gz
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
send: b'GET /2.0/repositories/ brofewfefwefewef/eee/downloads/keys.gz HTTP/1.1\r\nHost: api.bitbucket.org\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET /2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz HTTP/1.1" 302 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://api.bitbucket.org/2.0/repositories/ brofewfefwefewef/eee/downloads/keys.gz -> https://bbuseruploads.s3.amazonaws.com/a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=<redacted>&Expires=1515976464&AWSAccessKeyId=<redacted>&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bbuseruploads.s3.amazonaws.com
header: Server header: Vary header: Content-Type header: X-OAuth-Scopes header: Strict-Transport-Security header: Date header: Location header: X-Served-By header: ETag header: X-Static-Version header: X-Content-Type-Options header: X-Accepted-OAuth-Scopes header: X-Credential-Type header: X-Render-Time header: Connection header: X-Request-Count header: X-Frame-Options header: X-Version header: Content-Length send: b'GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=<redacted>&Expires=1515976464&AWSAccessKeyId=<redacted>&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1\r\nHost: bbuseruploads.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
DEBUG:urllib3.connectionpool:https://bbuseruploads.s3.amazonaws.com:443 "GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=<redacted>&Expires=1515976464&AWSAccessKeyId=<redacted>&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1" 400 None
DEBUG:pyupdater.client.downloader:Resource URL: https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz
DEBUG:pyupdater.client.downloader:Got content length of: None
DEBUG:pyupdater.client.downloader:Content-Length not in headers
DEBUG:pyupdater.client.downloader:Callbacks will not show time left or percent downloaded.
DEBUG:pyupdater.client.downloader:Using file as storage since the file is too large
DEBUG:pyupdater.client.downloader:Block size: 1036
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'downloading', 'percent_complete': '-.-%', 'time': '--:--'}
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'finished', 'percent_complete': '-.-%', 'time': '00:00'}
DEBUG:pyupdater.client.downloader:Download Complete
DEBUG:pyupdater.client.downloader:No hash to verify
WARNING:pyupdater.client.downloader:Downloaded file is very large, reading it in to memory may crash the app
DEBUG:pyupdater.client:Failed to decompress gzip file
DEBUG:pyupdater.client:Version file download failed
header: x-amz-request-id header: x-amz-id-2 header: Content-Type header: Transfer-Encoding header: Date header: Connection header: Server {'authorization': 'Basic <redacted>'}
DEBUG:pyupdater.client:Not a gzipped file (b'<?')
Traceback (most recent call last):
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\pyupdater\client\__init__.py", line 440, in _get_key_data
decompressed_data = _gzip_decompress(data)
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\dsdev_utils\helpers.py", line 58, in gzip_decompress
data = decompressed_file.read()
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 276, in read
return self._buffer.read(size)
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 463, in read
if not self._read_gzip_header():
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 411, in _read_gzip_header
raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'<?')
DEBUG:pyupdater.client:Loading version file...
DEBUG:pyupdater.client:Downloading online version file
DEBUG:pyupdater.client.downloader:Url for request: https://api.bitbucket.org/2.0/repositories/ brofewfefwefewef/eee/downloads/versions.gz
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
send: b'GET /2.0/repositories/ brofewfefwefewef/eee/downloads/versions.gz HTTP/1.1\r\nHost: api.bitbucket.org\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET /2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz HTTP/1.1" 302 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz -> https://bbuseruploads.s3.amazonaws.com/a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/0b04c4a8-dd59-49d2-9cd7-95d22379a5e6/versions.gz?Signature=<redacted>&Expires=1515976465&AWSAccessKeyId=<redacted>&versionId=jLhOcIbVAU4xRghD3kB2NfB4iLqUr7PM&response-content-disposition=attachment%3B%20filename%3D%22versions.gz%22
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bbuseruploads.s3.amazonaws.com
header: Server header: Vary header: Content-Type header: X-OAuth-Scopes header: Strict-Transport-Security header: Date header: Location header: X-Served-By header: ETag header: X-Static-Version header: X-Content-Type-Options header: X-Accepted-OAuth-Scopes header: X-Credential-Type header: X-Render-Time header: Connection header: X-Request-Count header: X-Frame-Options header: X-Version header: Content-Length send: b'GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/0b04c4a8-dd59-49d2-9cd7-95d22379a5e6/versions.gz?Signature=<redacted>&Expires=1515976465&AWSAccessKeyId=<redacted>&versionId=jLhOcIbVAU4xRghD3kB2NfB4iLqUr7PM&response-content-disposition=attachment%3B%20filename%3D%22versions.gz%22 HTTP/1.1\r\nHost: bbuseruploads.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
DEBUG:urllib3.connectionpool:https://bbuseruploads.s3.amazonaws.com:443 "GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/0b04c4a8-dd59-49d2-9cd7-95d22379a5e6/versions.gz?Signature=<redacted>&Expires=1515976465&AWSAccessKeyId=<redacted>&versionId=jLhOcIbVAU4xRghD3kB2NfB4iLqUr7PM&response-content-disposition=attachment%3B%20filename%3D%22versions.gz%22 HTTP/1.1" 400 None
reply: 'HTTP/1.1 400 Bad Request\r\n'
DEBUG:pyupdater.client.downloader:Resource URL: https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz
DEBUG:pyupdater.client.downloader:Got content length of: None
DEBUG:pyupdater.client.downloader:Content-Length not in headers
DEBUG:pyupdater.client.downloader:Callbacks will not show time left or percent downloaded.
DEBUG:pyupdater.client.downloader:Using file as storage since the file is too large
DEBUG:pyupdater.client.downloader:Block size: 1036
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'downloading', 'percent_complete': '-.-%', 'time': '--:--'}
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'finished', 'percent_complete': '-.-%', 'time': '00:00'}
DEBUG:pyupdater.client.downloader:Download Complete
DEBUG:pyupdater.client.downloader:No hash to verify
WARNING:pyupdater.client.downloader:Downloaded file is very large, reading it in to memory may crash the app
DEBUG:pyupdater.client:Failed to decompress gzip file
DEBUG:pyupdater.client:Version file download failed
DEBUG:pyupdater.client:Not a gzipped file (b'<?')
Traceback (most recent call last):
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\pyupdater\client\__init__.py", line 417, in _get_manifest_from_http
decompressed_data = _gzip_decompress(data)
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\dsdev_utils\helpers.py", line 58, in gzip_decompress
data = decompressed_file.read()
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 276, in read
return self._buffer.read(size)
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 463, in read
if not self._read_gzip_header():
File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 411, in _read_gzip_header
raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'<?')
DEBUG:dsdev_utils.paths:Changing to Directory --> C:\Users\Django\AppData\Local\any\main
DEBUG:pyupdater.client:Found version file on file system
DEBUG:pyupdater.client:Loaded version file from file system
DEBUG:dsdev_utils.paths:Moving back to Directory --> C:\Users\Django\privacy 4
DEBUG:pyupdater.client:Data type: <class 'bytes'>
DEBUG:pyupdater.client:App key is None
DEBUG:pyupdater.client:Version Data:
{'latest': {'main': {'stable': {'win': '1.4.0.2.0'}}}, 'updates': {'main': {'1.3.0.2.0': {'win': {'file_hash': '807c743b8c29f0053f4f9d9e6a8895b0e037f77480e7065c1470c2aba1cb08a0', 'file_size': 12194381, 'filename': 'main-win-1.3.zip', 'patch_hash': '29fec1006c2736eb78cc859f89e165af942daae6d9ac994a1a686d9b7b418ef6', 'patch_name': 'main-win-5', 'patch_size': 147}}, '1.4.0.2.0': {'win': {'file_hash': 'd59a22a95229f0a9c64909c646bfba31daf6bf8689dc16c9c93180c1602e9d3c', 'file_size': 12195571, 'filename': 'main-win-1.4.zip', 'patch_hash': 'baf3eba3a4b3184919ed9e57c3e8be9494a50862b40b1590ecb64e39e71a4ce3', 'patch_name': 'main-win-6', 'patch_size': 479625}}}}, 'signature': '<redacted>'}
DEBUG:dsdev_utils.helpers:Version str: 1.3
DEBUG:pyupdater.client:Failed version file verification
For those that want to replicate the error for themselves I’ve written the steps I have taken exactly.

Edit-1:
You need use below code for your main.py without any changes to downloader.py
from __future__ import print_function
import urllib3.poolmanager
orig_urlopen = urllib3.poolmanager.PoolManager.urlopen
def new_urlopen(self, method, url, redirect=True, **kw):
if "s3.amazonaws.com" in url and 'authorization' in self.headers:
self.headers.pop('authorization')
return orig_urlopen(self, method, url, redirect, **kw)
urllib3.poolmanager.PoolManager.urlopen = new_urlopen
import logging
from selenium import webdriver
logging.basicConfig(level=logging.DEBUG)
from client_config import ClientConfig
from pyupdater.client import Client, AppUpdate
import http.client as http_client
http_client.HTTPConnection.debuglevel = 1
def check_for_update():
client = Client(ClientConfig(), refresh=True, headers={'basic_auth': '<username>:<password>'})
app_update = client.update_check(ClientConfig.APP_NAME, ClientConfig.APP_VERSION, channel='stable')
if app_update is not None:
if app_update.download():
if isinstance(app_update, AppUpdate):
app_update.extract_restart()
return True
else:
app_update.extract()
return True
return False
def main():
print('Current version is ', ClientConfig.APP_VERSION)
if check_for_update():
print('there\'s a new update :D')
# driver = webdriver.Firefox()
# driver.get('http://stackoverflow.com')
if __name__ == "__main__":
main()
original answer
You need to use monkey patching for this. Below patch should do the job
import urllib3.poolmanager
orig_urlopen = urllib3.poolmanager.PoolManager.urlopen
def new_urlopen(self, method, url, redirect=True, **kw):
if "s3.amazonaws.com" in url and 'Authorization' in self.headers:
self.headers.pop('Authorization')
return orig_urlopen(self, method, url, redirect, **kw)
urllib3.poolmanager.PoolManager.urlopen = new_urlopen
A sample test worked for me with the above patch
import urllib3
pool = urllib3.PoolManager()
pool.headers.update({'Authorization': 'Basic XYZ=='})
r = pool.urlopen('GET', 'https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz')
print(r.data)
You need to execute the code before import pyupdater

I just checked and I believe it is a problem with pyupdater (I don't know what it is, never used).
It seems to assume that all the response's body will be compressed in GZIP. There is no flag I can find that would prevent this assumption. The actual content is actually not compressed at all.
Here is some relevant code from pyupdater:
pyupdater/client/__init__.py:
def _get_manifest_from_http(self):
log.debug('Downloading online version file')
try:
fd = _FD(self.version_file, self.update_urls, verify=self.verify,
urllb3_headers=self.urllib3_headers)
data = fd.download_verify_return()
try:
import ipdb
ipdb.set_trace()
decompressed_data = _gzip_decompress(data)
except IOError:
log.debug('Failed to decompress gzip file')
# Will be caught down below.
# Just logging the error
raise
log.debug('Version file download successful')
# Writing version file to application data directory
self._write_manifest_2_filesystem(decompressed_data)
return decompressed_data
except Exception as err:
log.debug('Version file download failed')
log.debug(err, exc_info=True)
return None
Here is a sample of data I receive:
ipdb> data
b'{"type": "error", "error": {"message": "keys.gz"}}'
I believe you should open a ticket on https://github.com/JMSwag/PyUpdater and see if they can help you further.

Requests is a pretty fantastic library, don't waste time with anything else, unless there is a really good reason:
import requests
import zlib
def download(url, username, password):
r = requests.get(url, auth=requests.auth.HTTPBasicAuth(username, password))
r.raise_for_status()
return zlib.decompress(r.content, 15 + 32)
download('https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz', 'brofewfefwefewef', your_password)
Also, it's probably worth noting that the credentials here aren't shouldn't be used anymore. Basic Auth can be decoded pretty simply.

Python client for free SOAP service using socket giving error?

This is the code that I'm trying to use:
import os
import socket
import httplib
packet='''
<?xml version="1.0" encoding="utf-8"?>
<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:xsd="http://www.w3.org/2001/XMLSchema"xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
<soap12:Body>
<GetCitiesByCountry xmlns="http://www.webserviceX.NET">
<CountryName>India</CountryName>
</GetCitiesByCountry>
</soap12:Body>
</soap12:Envelope>'''
lent=len(packet)
msg="""
POST "www.webservicex.net/globalweather.asmx" HTTP/1.1
Host: www.webservicex.net
Content-Type: application/soap+xml; charset=utf-8
Content-Length: {length}
SOAPAction:"http://www.webserviceX.NET/GetCitiesByCountry"
Connection: Keep-Alive
{xml}""".format(length=lent,xml=packet)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect( ("www.webservicex.net",80) )
if bytes != str: # Testing if is python2 or python3
msg = bytes(msg, 'utf-8')
client.send(msg)
client.settimeout(10)
print(client.type)
res=client.recv(4096)
print(res)
#res = res.replace("<","<") -- only for stack overflow post
#res = res.replace(">",">") -- only for stack overflow post
The output is:
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 24 Aug 2016 11:40:02 GMT
Connection: close
Content-Length: 324
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">;
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Invalid URL</h2>
<hr><p>HTTP Error 400. The request URL is invalid.</p>
</BODY></HTML>
Any ideas?

You ought to use suds to consume SOAP web service.
EDIT: Example
import suds
import suds.transport
from suds.transport.http import HttpAuthenticated
class MyException(Exception):
pass
service_url = "http://my/service/url"
GET the available services:
try:
transport = HttpAuthenticated(username='elmer', password='fudd')
wsdl = suds.client.Client(service_url, faults=False, transport=transport)
except IOError as exc:
fmt = "Can initialize my service: {reason}"
raise MyException(fmt.format(reason=exc))
except suds.transport.TransportError as exc:
fmt = "HTTP error -- Is it a bad URL: {service_url}? {reason}"
raise MyException(fmt.format(service_url=service_url, raison=exc))
Run a given service (here, its name is RunScript):
# add required options in headers
http_headers = {'sessionID': 1}
wsdl.set_options(soapheaders=http_headers)
params = {"scriptLanguage": "javascript", "scriptFile": "...",
"scriptArgs": [{"name": "...", "value": "..."}]}
try:
exit_code, response = wsdl.service.RunScript([params])
# ...
except suds.MethodNotFound as reason:
raise MyException("No method found: {reason}".format(reason=reason))
except suds.WebFault as reason:
raise MyException("Error running the script: {reason}".format(reason=reason))
except Exception as reason:
err_msg = "{0}".format(reason)
if err_msg == "timed out":
raise MyException("Timeout: {reason}".format(reason=reason))
raise
Of course, it is not required to use an error manager here. But I give an example. The last with the "timed out" is a kind of hack I used to detect a time out in my application.

python parse http response (string)

I'm using python 2.7 and I want to parse string HTTP response fields which I already extracted from a text file. What would be the easiest way? I can parse requests by using the BaseHTTPServer but couldn't manage to find something for the responses.
The responses I have are pretty standard and in the following format
HTTP/1.1 200 OK
Date: Thu, Jul 3 15:27:54 2014
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 626
Thanks in advance,

You might find this useful, keep in mind that HTTPResponse wasn't designed to be "instantiated directly by user."
Also note that the content-length header in your response string may not be valid any more (it depends on how you've aquired these responses) this just means that the call to HTTPResponse.read() needs to have value larger than the content in order to get it all.
In python 2 it can be run this way.
from httplib import HTTPResponse
from StringIO import StringIO
http_response_str = """HTTP/1.1 200 OK
Date: Thu, Jul 3 15:27:54 2014
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 626"""
class FakeSocket():
def __init__(self, response_str):
self._file = StringIO(response_str)
def makefile(self, *args, **kwargs):
return self._file
source = FakeSocket(http_response_str)
response = HTTPResponse(source)
response.begin()
print "status:", response.status
print "single header:", response.getheader('Content-Type')
print "content:", response.read(len(http_response_str)) # the len here will give a 'big enough' value to read the whole content
In python 3, the HTTPResponse is imported from http.client, and the response to be parsed needs to be byte encoded. Depending on where the data is gotten from this may be done already or need to be called explicitly
from http.client import HTTPResponse
from io import BytesIO
http_response_str = """HTTP/1.1 200 OK
Date: Thu, Jul 3 15:27:54 2014
Content-Type: text/xml; charset="utf-8"
Connection: close
Content-Length: 626
teststring"""
http_response_bytes = http_response_str.encode()
class FakeSocket():
def __init__(self, response_bytes):
self._file = BytesIO(response_bytes)
def makefile(self, *args, **kwargs):
return self._file
source = FakeSocket(http_response_bytes)
response = HTTPResponse(source)
response.begin()
print( "status:", response.status)
# status: 200
print( "single header:", response.getheader('Content-Type'))
# single header: text/xml; charset="utf-8"
print( "content:", response.read(len(http_response_str)))
# content: b'teststring'

You might want to consider using python-requests.
Link: http://docs.python-requests.org/en/latest/
Here is an example from http://dancallahan.info/journal/python-requests/
Considering your responses are compliant with HTTP RFC
Does this look like something you want to do?
>>> import requests
>>> url = 'http://example.test/'
>>> response = requests.get(url)
>>> response.status_code
200
>>> response.headers['content-type']
'text/html; charset=utf-8'
>>> response.content
u'Hello, world!'

Python requests - print entire http request (raw)?

While using the requests module, is there any way to print the raw HTTP request?
I don't want just the headers, I want the request line, headers, and content printout. Is it possible to see what ultimately is constructed from HTTP request?

Since v1.2.3 Requests added the PreparedRequest object. As per the documentation "it contains the exact bytes that will be sent to the server".
One can use this to pretty print a request, like so:
import requests
req = requests.Request('POST','http://stackoverflow.com',headers={'X-Custom':'Test'},data='a=1&b=2')
prepared = req.prepare()
def pretty_print_POST(req):
"""
At this point it is completely built and ready
to be fired; it is "prepared".
However pay attention at the formatting used in
this function because it is programmed to be pretty
printed and may differ from the actual request.
"""
print('{}\n{}\r\n{}\r\n\r\n{}'.format(
'-----------START-----------',
req.method + ' ' + req.url,
'\r\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()),
req.body,
))
pretty_print_POST(prepared)
which produces:
-----------START-----------
POST http://stackoverflow.com/
Content-Length: 7
X-Custom: Test
a=1&b=2
Then you can send the actual request with this:
s = requests.Session()
s.send(prepared)
These links are to the latest documentation available, so they might change in content:
Advanced - Prepared requests and API - Lower level classes

import requests
response = requests.post('http://httpbin.org/post', data={'key1': 'value1'})
print(response.request.url)
print(response.request.body)
print(response.request.headers)
Response objects have a .request property which is the PreparedRequest object that was sent.

An even better idea is to use the requests_toolbelt library, which can dump out both requests and responses as strings for you to print to the console. It handles all the tricky cases with files and encodings which the above solution does not handle well.
It's as easy as this:
import requests
from requests_toolbelt.utils import dump
resp = requests.get('https://httpbin.org/redirect/5')
data = dump.dump_all(resp)
print(data.decode('utf-8'))
Source: https://toolbelt.readthedocs.org/en/latest/dumputils.html
You can simply install it by typing:
pip install requests_toolbelt

Note: this answer is outdated. Newer versions of requests support getting the request content directly, as AntonioHerraizS's answer documents.
It's not possible to get the true raw content of the request out of requests, since it only deals with higher level objects, such as headers and method type. requests uses urllib3 to send requests, but urllib3 also doesn't deal with raw data - it uses httplib. Here's a representative stack trace of a request:
-> r= requests.get("http://google.com")
/usr/local/lib/python2.7/dist-packages/requests/api.py(55)get()
-> return request('get', url, **kwargs)
/usr/local/lib/python2.7/dist-packages/requests/api.py(44)request()
-> return session.request(method=method, url=url, **kwargs)
/usr/local/lib/python2.7/dist-packages/requests/sessions.py(382)request()
-> resp = self.send(prep, **send_kwargs)
/usr/local/lib/python2.7/dist-packages/requests/sessions.py(485)send()
-> r = adapter.send(request, **kwargs)
/usr/local/lib/python2.7/dist-packages/requests/adapters.py(324)send()
-> timeout=timeout
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py(478)urlopen()
-> body=body, headers=headers)
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py(285)_make_request()
-> conn.request(method, url, **httplib_request_kw)
/usr/lib/python2.7/httplib.py(958)request()
-> self._send_request(method, url, body, headers)
Inside the httplib machinery, we can see HTTPConnection._send_request indirectly uses HTTPConnection._send_output, which finally creates the raw request and body (if it exists), and uses HTTPConnection.send to send them separately. send finally reaches the socket.
Since there's no hooks for doing what you want, as a last resort you can monkey patch httplib to get the content. It's a fragile solution, and you may need to adapt it if httplib is changed. If you intend to distribute software using this solution, you may want to consider packaging httplib instead of using the system's, which is easy, since it's a pure python module.
Alas, without further ado, the solution:
import requests
import httplib
def patch_send():
old_send= httplib.HTTPConnection.send
def new_send( self, data ):
print data
return old_send(self, data) #return is not necessary, but never hurts, in case the library is changed
httplib.HTTPConnection.send= new_send
patch_send()
requests.get("http://www.python.org")
which yields the output:
GET / HTTP/1.1
Host: www.python.org
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-23-generic-pae

requests supports so called event hooks (as of 2.23 there's actually only response hook). The hook can be used on a request to print full request-response pair's data, including effective URL, headers and bodies, like:
import textwrap
import requests
def print_roundtrip(response, *args, **kwargs):
format_headers = lambda d: '\n'.join(f'{k}: {v}' for k, v in d.items())
print(textwrap.dedent('''
---------------- request ----------------
{req.method} {req.url}
{reqhdrs}
{req.body}
---------------- response ----------------
{res.status_code} {res.reason} {res.url}
{reshdrs}
{res.text}
''').format(
req=response.request,
res=response,
reqhdrs=format_headers(response.request.headers),
reshdrs=format_headers(response.headers),
))
requests.get('https://httpbin.org/', hooks={'response': print_roundtrip})
Running it prints:
---------------- request ----------------
GET https://httpbin.org/
User-Agent: python-requests/2.23.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
None
---------------- response ----------------
200 OK https://httpbin.org/
Date: Thu, 14 May 2020 17:16:13 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 9593
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
<!DOCTYPE html>
<html lang="en">
...
</html>
You may want to change res.text to res.content if the response is binary.

Here is a code, which makes the same, but with response headers:
import socket
def patch_requests():
old_readline = socket._fileobject.readline
if not hasattr(old_readline, 'patched'):
def new_readline(self, size=-1):
res = old_readline(self, size)
print res,
return res
new_readline.patched = True
socket._fileobject.readline = new_readline
patch_requests()
I spent a lot of time searching for this, so I'm leaving it here, if someone needs.

A fork of #AntonioHerraizS answer (HTTP version missing as stated in comments)
Use this code to get a string representing the raw HTTP packet without sending it:
import requests
def get_raw_request(request):
request = request.prepare() if isinstance(request, requests.Request) else request
headers = '\r\n'.join(f'{k}: {v}' for k, v in request.headers.items())
body = '' if request.body is None else request.body.decode() if isinstance(request.body, bytes) else request.body
return f'{request.method} {request.path_url} HTTP/1.1\r\n{headers}\r\n\r\n{body}'
headers = {'User-Agent': 'Test'}
request = requests.Request('POST', 'https://stackoverflow.com', headers=headers, json={"hello": "world"})
raw_request = get_raw_request(request)
print(raw_request)
Result:
POST / HTTP/1.1
User-Agent: Test
Content-Length: 18
Content-Type: application/json
{"hello": "world"}
💡 Can also print the request in the response object
r = requests.get('https://stackoverflow.com')
raw_request = get_raw_request(r.request)
print(raw_request)

I use the following function to format requests. It's like #AntonioHerraizS except it will pretty-print JSON objects in the body as well, and it labels all parts of the request.
format_json = functools.partial(json.dumps, indent=2, sort_keys=True)
indent = functools.partial(textwrap.indent, prefix=' ')
def format_prepared_request(req):
"""Pretty-format 'requests.PreparedRequest'
Example:
res = requests.post(...)
print(format_prepared_request(res.request))
req = requests.Request(...)
req = req.prepare()
print(format_prepared_request(res.request))
"""
headers = '\n'.join(f'{k}: {v}' for k, v in req.headers.items())
content_type = req.headers.get('Content-Type', '')
if 'application/json' in content_type:
try:
body = format_json(json.loads(req.body))
except json.JSONDecodeError:
body = req.body
else:
body = req.body
s = textwrap.dedent("""
REQUEST
=======
endpoint: {method} {url}
headers:
{headers}
body:
{body}
=======
""").strip()
s = s.format(
method=req.method,
url=req.url,
headers=indent(headers),
body=indent(body),
)
return s
And I have a similar function to format the response:
def format_response(resp):
"""Pretty-format 'requests.Response'"""
headers = '\n'.join(f'{k}: {v}' for k, v in resp.headers.items())
content_type = resp.headers.get('Content-Type', '')
if 'application/json' in content_type:
try:
body = format_json(resp.json())
except json.JSONDecodeError:
body = resp.text
else:
body = resp.text
s = textwrap.dedent("""
RESPONSE
========
status_code: {status_code}
headers:
{headers}
body:
{body}
========
""").strip()
s = s.format(
status_code=resp.status_code,
headers=indent(headers),
body=indent(body),
)
return s

test_print.py content:
import logging
import pytest
import requests
from requests_toolbelt.utils import dump
def print_raw_http(response):
data = dump.dump_all(response, request_prefix=b'', response_prefix=b'')
return '\n' * 2 + data.decode('utf-8')
#pytest.fixture
def logger():
log = logging.getLogger()
log.addHandler(logging.StreamHandler())
log.setLevel(logging.DEBUG)
return log
def test_print_response(logger):
session = requests.Session()
response = session.get('http://127.0.0.1:5000/')
assert response.status_code == 300, logger.warning(print_raw_http(response))
hello.py content:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
Run:
$ python -m flask hello.py
$ python -m pytest test_print.py
Stdout:
------------------------------ Captured log call ------------------------------
DEBUG urllib3.connectionpool:connectionpool.py:225 Starting new HTTP connection (1): 127.0.0.1:5000
DEBUG urllib3.connectionpool:connectionpool.py:437 http://127.0.0.1:5000 "GET / HTTP/1.1" 200 13
WARNING root:test_print_raw_response.py:25
GET / HTTP/1.1
Host: 127.0.0.1:5000
User-Agent: python-requests/2.23.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 13
Server: Werkzeug/1.0.1 Python/3.6.8
Date: Thu, 24 Sep 2020 21:00:54 GMT
Hello, World!

Problems to authenticate correctly with pysimplesoap

I'm trying to use pysimplesoap to communicate with the Websitepanel SOAP-API.
The WebsitePanel API Introduction says:
For interacting with WebsitePanel API you should use Basic Authentication. WebsitePanel recognizes “Authorization” header with the user credentials provided in the following format: username:password
My first try was the following:
client = SoapClient(wsdl=endpoint_url, trace=True)
client['Authorization'] = "%s:%s" % (username, password)
which returns a 401 "Unauthorized".
Second try was:
client = SoapClient(wsdl=endpoint_url, trace=True)
client['wsse:Security'] = {
'wsse:UsernameToken': {
'wsse:Username': username,
'wsse:Password': password,
}
}
which works as expected but returns the following:
status: 500
content-length: 924
x-aspnet-version: 4.0.30319
x-powered-by: ASP.NET
server: Microsoft-IIS/7.5
cache-control: private
date: Tue, 12 Feb 2013 14:23:56 GMT
content-type: text/xml; charset=utf-8
And
pysimplesoap.client.SoapFault: q0:Security: SOAP response should be signed.
Why does client['Authorization'] not work and what is meant by the Response should be signed error message?
Thanks in advance.

I figured it out: To authenticate correctly with pysimplesoap you have to call
client = SoapClient(wsdl=u, trace=True,
http_headers={'Authorization': 'Basic %s' % encoded})
with encodedbeeing the base64-encoded string username:password