We Keep Coding

Retry for python requests module hanging

I had to use the request module to fetch a huge number of URLs. Due to network errors, I wanted to implement the Retry mechanism. So my code looks like this:
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import requests
session = requests.Session()
retry = Retry(total=1,backoff_factor=0.5,status_forcelist=(500,502,504,503))
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://',adapter)
head_response = session.get(url,timeout=2)
However, the code keeps hanging when the URL is: tiffins.NET. A normal requests.get with timeout=2 gives 503 status code but doesn't hang.
Am I doing something wrong?
I'm using python 2.7.15rc1.

Just had the same problem and figured out that it's because of the website's header "Retry-After". By default, Retry uses that value to wait until the next request. To ignore it, set the respect_retry_after_header parameter to False when creating a Retry object.

Requests Retry: will same proxy be used

I keep seeing examples similar to this for adding retries when using the Requests library. However, I'm unsure of whether Requests will execute choice(my_proxy_list) at every retry and thus get a new proxy or just keep retrying with the same arguments used for initial request.
import my_proxy_list
from random import choice
import requests
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
session = requests.Session()
retries = Retry(total=5, backoff_factor=0.4, status_forcelist=[400, 429, 500, 502, 503, 504])
session.mount("http://", HTTPAdapter(max_retries=retries))
response = session.get(
url=url,
proxies=choice(my_proxy_list),
timeout=(10, 27),
)
session.close()

Function arguments are evaluated once, regardless of what that function goes on to do later (including any retry logic that function may use internally), so random.choice will only be called once in your example.
The best option if you want a (chance at a) different proxy each time is to do your own retry logic which calls random.choice each time. To guarantee a different proxy each time, you could shuffle the list of possible proxies at the start and then traverse it.
Alternatively, it would be possible to pass in a dictionary-like object for proxies with a __getitem__ designed to return a random proxy each time, but that approach isn't recommended as it'd be very fragile and would rely heavily on the implementation details of session.get.

Reading SOAP with Zeep

I'm trying to read this Soap API 'https://www.shab.ch/soapserver' using Python's Zeep.
from requests import Session
from requests.auth import HTTPBasicAuth
from zeep import Client
from zeep.transports import Transport
session = Session()
session.auth = HTTPBasicAuth('MYUSERNAME', 'MYPASSWORD')
client = Client('https://www.shab.ch/shabforms/acmsservice?wsdl',
transport=Transport(session=session))
But get the following error message:
HTTPError: 403 Client Error: Forbidden for url:
https://www.shab.ch/shabforms/acmsservice?wsdl
Can anybody see what I am doing wrongly here?

It doesn't seem that Python is your problem. Instead you have a rights issue for the WSDL file. A quick Googling turned up:
https://www.shab.ch/soapserver
https://www.shab.ch/shabforms/soapserver?wsdl

You don't have permission to go to that part of the Shab.ch server. You need a username and password. Contact: info#sogc.ch

Python SOAP client with Zeep - import namespace

A little context: I am opening this question arose here, after solving an authentication problem. I prefer to open a new one to avoid polluting the previous with comments not related to the original issue, and to give it the proper visibility.
I am working on a SOAP client running in the same intranet as the server, without internet access.
from requests.auth import HTTPBasicAuth
from zeep import Client
from zeep.transports import Transport
wsdl = 'http://mysite.dom/services/MyWebServices?WSDL'
client = Client(wsdl, transport=HTTPBasicAuth('user','pass'), cache=None)
The problem: WSDL contains an import to an external resource located outside the intranet ('import namespace="schemas.xmlsoap.org/soap/encoding/"') and therefore Zeep Client instantiation fails with:
Exception: HTTPConnectionPool(host='schemas.xmlsoap.org', port=80): Max retries exceeded with url: /soap/encoding/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3dab9d30b8>: Failed to establish a new connection: [Errno 110] Connection timed out',))
Question: is it possible (and does it make sense) to create the Zeep Client without accessing the external resource?
As an additional detail, another client written in Java, based on XML rpc ServiceFactory seems to be more resilient to this kind of problem, the service is created (and works) even if no internet connection is available.
Is it really needed to import the namespace from xmlsoap.org?
Edit, after answer from #mvt:
So, I went for the proposed solution, which allows me at the same time to control the access to external resources (read: forbid access to servers different from the one hosting the endpoint).
class MyTransport(zeep.Transport):
def load(self, url):
if not url:
raise ValueError("No url given to load")
parsed_url = urlparse(url)
if parsed_url.scheme in ('http', 'https'):
if parsed_url.netloc == "myserver.ext":
response = self.session.get(url, timeout=self.load_timeout)
response.raise_for_status()
return response.content
elif url == "http://schemas.xmlsoap.org/soap/encoding/":
url = "/some/path/myfile.xsd"
else:
raise
elif parsed_url.scheme == 'file':
if url.startswith('file://'):
url = url[7:]
with open(os.path.expanduser(url), 'rb') as fh:
return fh.read()

I would suggest doing your custom overriding of the URL and calling load() from the super class. This way if the super class code changes (which it has), you would not need to refactor your CustomTransport class.
from zeep.transports import Transport
class CustomTransport(Transport):
def load(self, url):
# Custom URL overriding to local file storage
if url and url == "http://schemas.xmlsoap.org/soap/encoding/":
url = "/path/to/schemas.xmlsoap.org.xsd"
# Call zeep.transports.Transport's load()
return super(CustomTransport, self).load(url)
The way to use the Transports in zeep is described here, but here is a quick example of using the CustomTransport:
from requests import Session
from requests.auth import HTTPBasicAuth
from zeep import Client
session = Session()
client = Client('http://example.com/production.svc?wsdl', transport=CustomTransport(session=session))
client.service.foo()

You could create your own subclass of the tranport class and add additional logic to the load() method so that specific url's are redirected / loaded from the filesystem.
The code is pretty easy i think: https://github.com/mvantellingen/python-zeep/blob/master/src/zeep/transports.py :-)

Python SOAP client with Zeep - authentication

I am trying to use Zeep to implement a SOAP client, as it seems the only maintained library at the moment:
ZSI looked very good but its latest version on pypi dates 2006
suds seemed to be a popular alternative, but the master is unmaintained since 2011 and there are a lot of forks out there but none seems "official" and "recent" enough to be used in a large project.
So, trying to use Zeep, I am stuck with the authentication required by the server to access the WSDL.
Such operation was quite easy with ZSI:
from ZSI.client import Binding
from ZSI.auth import AUTH
b = Binding(url='http://mysite.dom/services/MyWebServices?WSDL')
b.SetAuth(AUTH.httpbasic, 'userid', 'password')
and I can find something similar in __main__.py of Zeep:
from six.moves.urllib.parse import urlparse
from zeep.cache import InMemoryCache, SqliteCache
from zeep.client import Client
from zeep.transports import Transport
cache = SqliteCache() if args.cache else InMemoryCache()
transport_kwargs = {'cache': cache}
result = urlparse(args.wsdl_file)
if result.username or result.password:
transport_kwargs['http_auth'] = (result.username, result.password)
transport = Transport(**transport_kwargs)
client = Client(args.wsdl_file, transport=transport)
but that does not work in my case, I get an error:
Exception: HTTPConnectionPool(host='schemas.xmlsoap.org', port=80): Max retries exceeded with url: /soap/encoding/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3dab9d30b8>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Probably with the newer Version of zeep the older solution does not work anymore. Here is the new way:
from requests.auth import HTTPBasicAuth # or HTTPDigestAuth, or OAuth1, etc.
from requests import Session
from zeep import Client
from zeep.transports import Transport
session = Session()
session.auth = HTTPBasicAuth(user, password)
client = Client('http://my-endpoint.com/production.svc?wsdl',
transport=Transport(session=session))

For Basic Access Authentication you can use the HTTPBasicAuth class from the requests module, as explained on Zeep documentation http://docs.python-zeep.org/en/master/transport.html:
from requests.auth import HTTPBasicAuth # or HTTPDigestAuth, or OAuth1, etc.
from zeep import Client
from zeep.transports import Transport
client = Client('http://my-endpoint.com/production.svc?wsdl',
transport=Transport(http_auth=HTTPBasicAuth(user, password)))

In my case the API I was working with required WS-Security (WSSE) rather than HTTP.
from zeep import Client
from zeep.wsse.username import UsernameToken
client = Client(<wsdl_url>, wsse=UsernameToken(<username>, <password>)