I'm going through Dive into Python3. When I get to the chapter on http web services section 14.4, I can't seem to duplicate the following output in the python3 shell. Here's what the sample code looks like:
from http.client import HTTPConnection
HTTPConnection.debuglevel = 1
from urllib.request import urlopen
response = urlopen('http://diveintopython3.org/examples/feed.xml')
send: b'GET /examples/feed.xml HTTP/1.1
Host: diveintopython3.org
Accept-Encoding: identity
User-Agent: Python-urllib/3.1'
Connection: close
reply: 'HTTP/1.1 200 OK'
…further debugging information omitted…
When I enter this in ipython3, the final command gives no output. So why am I not getting the debug info in the example? After entering the above code, response.debuglevel == 0. I'm using python3.5.2.
The final command should not give any output, what you probably want is:
print(response.read())
I know this is an old question, bit I thought I would answer to help those who might still be seeing this question.
Ever since Python version 3.5.2 (release ~June 2016) the http.client.HTTPConnection.debuglevel is entirely ignored in favor of the debuglevel constructor argument for urllib.request.HTTPHandler.
This is due to this change that sets the value of http.client.HTTPConnection.debuglevel to whatever is set in urllib.request.HTTPHandler's constructor argument debuglevel, on this line.
A PR has been opened to fix this, but in the mean time you can either do one of two things:
You can use the constructor argument for HTTPHandler and HTTPSHandler (as demonstrated in this SO answer):
import urllib.request
handler = urllib.request.HTTPHandler(debuglevel=10)
opener = urllib.request.build_opener(handler)
content = opener.open('http://stackoverflow.com').read()
print(content[0:120])
You can also monkey patch the __init__ methods of HTTPHandler and HTTPSHandler to respect the global values like so:
https_old_init = urllib.request.HTTPSHandler.__init__
def https_new_init(self, debuglevel=None, context=None, check_hostname=None):
debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
https_old_init(self, debuglevel, context, check_hostname)
urllib.request.HTTPSHandler.__init__ = https_new_init
http_old_init = urllib.request.HTTPHandler.__init__
def http_new_init(self, debuglevel=None):
debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
http_old_init(self, debuglevel)
urllib.request.HTTPHandler.__init__ = http_new_init
Note: I don't recommend setting the debuglevel in HTTPHandler's as a method argument default value because the default values for method arguments get evaluated at function definition evaluation time, which, for HTTPHandler's constructor, is when the module urllib.request is imported.
Related
A simple code as such:
import urllib2
import requests
from PyQt4 import QtCore
import multiprocessing
import time
data = (
['a', '2'],
)
def mp_worker((inputs, the_time)):
r = requests.get('http://www.gpsbasecamp.com/national-parks')
request = urllib2.Request("http://www.gpsbasecamp.com/national-parks")
response = urllib2.urlopen(request)
def mp_handler():
p = multiprocessing.Pool(2)
p.map(mp_worker, data)
if __name__ == '__main__':
mp_handler()
Basically, if i import PyQt4, and i have a urllib request (i believe this is used in almost all web extraction libraries such as BeautifulSoup, Requests or Pyquery. it crashes with a cryptic log on my MAC)
This is exactly True. It always fails on Mac, I have wasted rows of days just to fix this. And honestly there is no fix as of now. The best way is to use Thread instead of Process and it will work like a charm.
By the way -
r = requests.get('http://www.gpsbasecamp.com/national-parks')
and
request = urllib2.Request("http://www.gpsbasecamp.com/national-parks")
response = urllib2.urlopen(request)
do one and the same thing. Why are you doing it twice?
This may be due _scproxy.get_proxies() not being fork-safe on Mac.
This is raised here https://bugs.python.org/issue33725#msg329926
_scproxy has been known to be problematic for some time, see for instance Issue31818. That issue also gives a simple workaround: setting urllib's "no_proxy" environment variable to "*" will prevent the calls to the System Configuration framework.
This is something that urllib may be attempting to do causing failure when multiprocessing.
There is a workaround and that is to set the environmental variable no-proxy to *
Eg. export no_proxy=*
The Python requests library appears to have some rather strange quirks when it comes to its logging behaviour.
Using the latest Python 2.7.8, I have the following code:
import requests
import logging
logging.basicConfig(
filename='mylog.txt',
format='%(asctime)-19.19s|%(task)-36s|%(levelname)s:%(name)s: %(message)s',
level=eval('logging.%s' % 'DEBUG'))
logger = logging.getLogger(__name__)
logger.info('myprogram starting up...', extra={'task': ''}) # so far so good
...
(ommited code)
...
payload = {'id': 'abc', 'status': 'ok'}
# At this point the program continues but throws an exception.
requests.get('http://localhost:9100/notify', params=payload)
print 'Task is complete! NotifyURL was hit! - Exiting'
My program seems to exit normally, however inside the log file it creates (mylog.txt) I always find the following exception:
KeyError: 'task'
Logged from file connectionpool.py, line 362
If I remove this:
requests.get('http://localhost:9100/notify', params=payload)
then the exception is gone.
What exactly am I doing wrong here and how may I fix this?
I am using requests v2.4.3.
The problem is your custom logging format, where you expect a %(task).
Requests (or rather the bundled urllib3) does not include the task parameter when logging, as it has no way of knowing, that you expect this.
As indicated in t-8ch's answer, the logger is being used internally by the requests library and this library doesn't know anything about your parameters. A possible solution is to implant a custom filter in the library's logger (in this case, one of its modules):
class TaskAddingFilter(logging.Filter):
def __init__(self):
logging.Filter.__init__(self)
def filter(self, record):
record.args = record.args + ('task', '')
# ...
requestsLogger = logging.getLogger('requests.packages.urllib3.connectionpool')
requestsLogger.addFilter(TaskAddingFilter())
Potentially, you need to add such filtering to all loggers from requests, which are:
requests.packages.urllib3.util
requests.packages.urllib3.connectionpool
requests.packages
requests.packages.urllib3
requests.packages.urllib3.util.retry
requests
requests.packages.urllib3.poolmanager
in my version, you can find them using the logging.Logger.manager.loggerDict attribute. So, you could do something like this:
for name,logger in logging.Logger.manager.loggerDict.iteritems():
logger = logging.getLogger(name) # because of lazy initialization
if name.startswith('requests.'):
logger.addFilter(TaskAddingFilter())
The TaskAddingFilter can be a bit smarter of course, e.g. adding a particular task entry depending on where you are in your code. I've added only the simplest solution for the code you've provided - the exception doesn't occur anymore - but the wide range of possibilities seems obvious ;)
I would like to be able to enter a server response code and have Requests tell me what the code means. For example, code 200 --> ok
I found a link to the source code which shows the dictionary structure of the codes and descriptions. I see that Requests will return a response code for a given description:
print requests.codes.processing # returns 102
print requests.codes.ok # returns 200
print requests.codes.not_found # returns 404
But not the other way around:
print requests.codes[200] # returns None
print requests.codes.viewkeys() # returns dict_keys([])
print requests.codes.keys() # returns []
I thought this would be a routine task, but cannot seem to find an answer to this in online searching, or in the documentation.
Alternatively, in case of Python 2.x, you can use httplib.responses:
>>> import httplib
>>> httplib.responses[200]
'OK'
>>> httplib.responses[404]
'Not Found'
In Python 3.x, use http module:
In [1]: from http.client import responses
In [2]: responses[200]
Out[2]: 'OK'
In [3]: responses[404]
Out[3]: 'Not Found'
One possibility:
>>> import requests
>>> requests.status_codes._codes[200]
('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\\o/', '\xe2\x9c\x93')
The first value in the tuple is used as the conventional code key.
I had the same problem before and found the
answer in this question
Basically:
responsedata.status_code - gives you the integer status code
responsedata.reason - gives the text/string representation of the status code
requests.status_codes.codes.OK
works nicely and makes it more readable in my application code
Notice that in source code: the requests.status_codes.codes is of type LookupDict which overrides method getitem
You could see all the supported keys with - dir(requests.status_codes.codes)
When using in combination with FLASK:
i like use following enum from flask-api plugin
from flask_api import status where i get more descriptive version of HTTP status codes as in -
status.HTTP_200_OK
With Python 3.x this will work
>>> from http import HTTPStatus
>>> HTTPStatus(200).phrase
'OK'
I run a soap server in django.
Is it possible to create a soap method that returns a soaplib classmodel instance without <{method name}Response><{method name}Result> tags?
For example, here is a part of my soap server code:
# -*- coding: cp1254 -*-
from soaplib.core.service import rpc, DefinitionBase, soap
from soaplib.core.model.primitive import String, Integer, Boolean
from soaplib.core.model.clazz import Array, ClassModel
from soaplib.core import Application
from soaplib.core.server.wsgi import Application as WSGIApplication
from soaplib.core.model.binary import Attachment
class documentResponse(ClassModel):
__namespace__ = ""
msg = String
hash = String
class MyService(DefinitionBase):
__service_interface__ = "MyService"
__port_types__ = ["MyServicePortType"]
#soap(String, Attachment, String ,_returns=documentResponse,_faults=(MyServiceFaultMessage,) , _port_type="MyServicePortType" )
def sendDocument(self, fileName, binaryData, hash ):
binaryData.file_name = fileName
binaryData.save_to_file()
resp = documentResponse()
resp.msg = "Saved"
resp.hash = hash
return resp
and it responses like that:
<senv:Body>
<tns:sendDocumentResponse>
<tns:sendDocumentResult>
<hash>14a95636ddcf022fa2593c69af1a02f6</hash>
<msg>Saved</msg>
</tns:sendDocumentResult>
</tns:sendDocumentResponse>
</senv:Body>
But i need a response like this:
<senv:Body>
<ns3:documentResponse>
<hash>A694EFB083E81568A66B96FC90EEBACE</hash>
<msg>Saved</msg>
</ns3:documentResponse>
</senv:Body>
What kind of configurations should i make in order to get that second response i mentioned above ?
Thanks in advance.
I haven't used Python's SoapLib yet, but had the same problem while using .NET soap libs. Just for reference, in .NET this is done using the following decorator:
[SoapDocumentMethod(ParameterStyle=SoapParameterStyle.Bare)]
I've looked in the soaplib source, but it seems it doesn't have a similar decorator. The closest thing I've found is the _style property. As seen from the code https://github.com/soaplib/soaplib/blob/master/src/soaplib/core/service.py#L124 - when using
#soap(..., _style='document')
it doesn't append the %sResult tag, but I haven't tested this. Just try it and see if this works in the way you want it.
If it doesn't work, but you still want to get this kind of response, look at Spyne:
http://spyne.io/docs/2.10/reference/decorator.html
It is a fork from soaplib(I think) and has the _soap_body_style='bare' decorator, which I believe is what you want.
Using Python 2.7.2 on OSX (darwin), I would like to hide or customize the "Server" response header sent by the wsgiref.simple_server.make_server().
I tried many things without any success and was pretty sure this sample code should work:
from wsgiref import simple_server
class MyWSGIRequestHandler(simple_server.WSGIRequestHandler):
server_version = "X/1"
sys_version = "Y/2"
httpd = simple_server.make_server('', 8082, simple_server.demo_app, handler_class=MyWSGIRequestHandler)
print "version_string: %s %s" % (httpd.RequestHandlerClass.server_version, httpd.RequestHandlerClass.sys_version)
# it prints "X/1 Y/2"
httpd.serve_forever()
But it's always the same and there's no way to get rid of the "Server: WSGIServer/0.1 Python/2.7.2" sent by the server. I've also tried to override the version_string method in my class, for example with something like that:
class MyWSGIRequestHandler(simple_server.WSGIRequestHandler):
def version_string(self):
return "42"
It changes nothing, I really don't understand what's happening here.
Can someone help me please?
I've finally found the solution, no need to override WSGIRequestHandler.
from wsgiref.simple_server import ServerHandler
ServerHandler.server_software = "Fake Server Name Here"
And then you can call make_server().