I am writing a test for a function that downloads the data from an url with Twisted (I know about twisted.web.client.getPage, but this one adds some extra functionality). Either ways, I want to use nosetests since I am using it throughout the project and it doesn't look appropriate to use Twisted Trial only for this particular test.
So what I am trying to do is something like:
from nose.twistedtools import deferred
#deferred()
def test_download(self):
url = 'http://localhost:8000'
d = getPage(url)
def callback(data):
assert len(data) != 0
d.addCallback(callback)
return d
On localhost:8000 listens a test server. The issue is I always get twisted.internet.error.DNSLookupError
DNSLookupError: DNS lookup failed: address 'localhost:8000' not found: [Errno -5] No address associated with hostname.
Is there a way I can fix this? Does anyone actually uses nose.twistedtools?
Update: A more complete traceback
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/nose-0.11.2-py2.6.egg/nose/twistedtools.py", line 138, in errback
failure.raiseException()
File "/usr/local/lib/python2.6/dist-packages/Twisted-9.0.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 326, in raiseException
raise self.type, self.value, self.tb
DNSLookupError: DNS lookup failed: address 'localhost:8000' not found: [Errno -5] No address associated with hostname.
Update 2
My bad, it seems in the implementation of getPage, I was doing something like:
obj = urlparse.urlparse(url)
netloc = obj.netloc
and passing netloc to the the factory when I should've passed netloc.split(':')[0]
Are you sure your getPage function is parsing the URL correctly? The error message seems to suggest that it is using the hostname and port together when doing the dns lookup.
You say your getPage is similar to twisted.web.client.getPage, but that works fine for me when I use it in this complete script:
#!/usr/bin/env python
from nose.twistedtools import deferred
from twisted.web import client
import nose
#deferred()
def test_download():
url = 'http://localhost:8000'
d = client.getPage(url)
def callback(data):
assert len(data) != 0
d.addCallback(callback)
return d
if __name__ == "__main__":
args = ['--verbosity=2', __file__]
nose.run(argv=args)
While running a simple http server in my home directory:
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
The nose test gives the following output:
.
----------------------------------------------------------------------
Ran 1 test in 0.019s
OK
Related
On Google IT automation with python specialization "Using python interact with Operating System"'s week 1 Qwiklabs Assessment: Working with Python 3rd module is not properly work
on ~/scripts directory :
network.py code
#!usr/bin/env python3
import requests
import socket
def check_localhost():
localhost = socket.gethostbyname('localhost')
print(localhost)
if localhost == '127.0.0.1':
return True
return False
def check_connectivity():
request = requests.get("http://www.google.com")
responses = request.status_code
print(responses)
if responses == 200:
return True
return False
By using this code I make "Create a new Python module
"
but Qwiklab show me that I cannot properly write code.!!!
now what is the problem?
I am responding with this piece here because I noticed a lot of folks taking this course "Using Python to interact with the Operating System" on Coursera do have similar issues with writing the Python function check_localhost and check_connectivity. Please, copy these functions to your VM and try again.
To ping the web and also check whether the local host is correctly configured, we will import requests module and socket module.
Next, write a function check_localhost, which checks whether the local host is correctly configured. And we do this by calling the gethostbyname within the function.
localhost = socket.gethostbyname('localhost')
The above function translates a hostname to IPv4 address format. Pass the parameter localhost to the function gethostbyname. The result for this function should be 127.0.0.1.
Edit the function check_localhost so that it returns true if the function returns 127.0.0.1.
import requests
import socket
#Function to check localhost
def check_localhost():
localhost = socket.gethostbyname('localhost')
if localhost == "127.0.0.1":
return True
else:
return False
Now, we will write another function called check_connectivity. This checks whether the computer can make successful calls to the internet.
A request is when you ping a website for information. The Requests library is designed for this task. You will use the request module for this, and call the GET method by passing a http://www.google.com as the parameter.
request = requests.get("http://www.google.com")
This returns the website's status code. This status code is an integer value. Now, assign the result to a response variable and check the status_code attribute of that variable. It should return 200.
Edit the function check_connectivity so that it returns true if the function returns 200 status_code.
#Function to check connectivity
def check_connectivity():
request = requests.get("http://www.google.com")
if request.status_code == 200:
return True
else:
return False
Once you have finished editing the file, press Ctrl-o, Enter, and Ctrl-x to exit.
When you're done, Click Check my progress to verify the objective.
I was also using a similar code, although It was getting executed fine in the lab terminal, however, it was not getting successfully verified. I contacted the support team using support chat and they provided a similar but relatively efficient code that worked ;
#!/usr/bin/env python3
import requests
import socket
localhost = socket.gethostbyname('localhost')
request = requests.get("http://www.google.com")
def check_localhost():
if localhost == "127.0.0.1":
return True
def check_connectivity():
if request.status_code == 200:
return True
I use the same code of yours to check what's the problem in the code, but your code successfully passed the qwiklabs check.
I think there is something wrong, Did you retry again by end this lab session and create another session just to check if this is something wrong with their end.
It's so easy, in this case, the shebang line would be /usr/bin/env python3.
you type a wrong shebang line:
#!usr/bin/env python3
But you should type:
#!/usr/bin/env python3
Add a shebang line to define where the interpreter is located.
It can be written like this
#!/usr/bin/env python3
import requests
import socket
def check_localhost():
localhost = socket.gethostbyname('localhost')
return localhost == "127.0.0.1"
def check_connectivity():
request = requests.get("http://www.google.com")
return request.status_code() == 200
This script does work even if you are facing the issue in the post:
import requests
import socket
def check_localhost():
localhost = socket.gethostbyname('localhost')
return True # or return localhost == "127.0.0.1"
def check_connectivity():
request = requests.get("http://www.google.com")
return True #or return request==200
What could be wrong with your code? The verification system is faulty and doesn't accept:
-tab instead of 4 spaces as right identation
-spaces between lines
#!usr/bin/env python3
import requests
import socket
def check_localhost():
localhost = socket.gethostbyname('localhost')
print(localhost)
if localhost == '127.0.0.1':
return True
def check_connectivity():
request = requests.get("http://www.google.com")
responses = request.status_code()
print(responses)
if responses == '200':
return True
#!/usr/bin/env python3
import requests
import socket
def check_localhost():
localhost = socket.gethostbyname('localhost')
if localhost == "127.0.0.1":
return True
def check_connectivity():
request = requests.get("http://www.google.com")
if request.status_code() == 200:
return True
This question already has an answer here:
How to reliably check if a domain has been registered or is available?
(1 answer)
Closed 2 years ago.
I have a large list of domains and I need to check if domains are available now. I do it like this:
import requests
list_domain = ['google.com', 'facebook.com']
for domain in list_domain:
result = requests.get(f'http://{domain}', timeout=10)
if result.status_code == 200:
print(f'Domain {domain} [+++]')
else:
print(f'Domain {domain} [---]')
But the check is too slow. Is there a way to make it faster? Maybe someone knows an alternative method for checking domains for existence?
You can use the socket library to determine if a domain has a DNS entry:
>>> import socket
>>>
>>> addr = socket.gethostbyname('google.com')
>>> addr
'74.125.193.100'
>>> socket.gethostbyname('googl42652267e.com')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known
>>>
If you want to check which domains are available, the more correct approach would be to catch the ConnectionError from the requests module, because even if you get a response code that is not 200, the fact that there is a response means that there is a server associated with that domain. Hence, the domain is taken.
This is not full proof in terms of checking for domain availability, because a domain might be taken, but may not have appropriate A record associated with it, or the server may just be down for the time being.
The code below is asynchronous as well.
from concurrent.futures import ThreadPoolExecutor
import requests
from requests.exceptions import ConnectionError
def validate_existence(domain):
try:
response = requests.get(f'http://{domain}', timeout=10)
except ConnectionError:
print(f'Domain {domain} [---]')
else:
print(f'Domain {domain} [+++]')
list_domain = ['google.com', 'facebook.com', 'nonexistent_domain.test']
with ThreadPoolExecutor() as executor:
executor.map(validate_existence, list_domain)
You can do that via "requests-futures" module.
requests-futures runs Asynchronously, If you have a average internet connection it can check 8-10 url per second (Based on my experience).
What you can do is run the script multiple times but add only a limited amount of domains to each to make it speedy.
Use scrapy it is way faster and by default it yields only 200 response until you over ride it so in your case follow me
pip install scrapy
After installing in your project folder user terminal to creat project
Scrapy startproject projectname projectdir
It will create folder name projectdir
Now
cd projectdir
Inside projectdir enter
scrapy genspider mydomain mydomain.com
Now navigate to spiders folder open mydomain.py
Now add few lines of code
import scrapy
class MydomainSpider(scrapy.Spider):
name = "mydomain"
def start_requests(self):
urls = [
'facebook.com',
'Google.com',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
yield { ‘Available_Domains’ : response.url}
Now back to projectdir and run
scrapy crawl mydomain -o output.csv
You will have all the working domains having status code 200 in output.csv file
For more see
I'm rather new to Python and Pyro4. So I try to follow the second example of this page Pyro - Python Remote Objects - 4.41, but when I run the server throw this exception:
Traceback (most recent call last):
File "greeting-server.py", line 10, in <module>
ns = Pyro4.locateNS() # find the name server
File "/usr/lib/python2.7/dist-packages/Pyro4/naming.py", line 344, in locateNS
raise e
Pyro4.errors.NamingError: Failed to locate the nameserver
Code Server:
# saved as greeting-server.py
import Pyro4
class GreetingMaker(object):
def get_fortune(self, name):
return "Hello, {0}. Here is your fortune message:\n" \
"Tomorrow's lucky number is 12345678.".format(name)
daemon = Pyro4.Daemon() # make a Pyro daemon
ns = Pyro4.locateNS() # find the name server
uri = daemon.register(GreetingMaker) # register the greeting maker as a Pyro object
ns.register("example.greeting", uri) # register the object with a name in the name server
print("Ready.")
daemon.requestLoop() # start the event loop of the server to wait for calls
Run pyro-ns in another terminial first:
$pyro-ns
*** Pyro Name Server ***
Name server listening on: ('0.0.0.0', 9090)
WARNING: daemon bound on hostname that resolves to loopback address 127.0.x.x
URI is: PYRO://127.0.1.1:9090/7f0001011d2a21ca9fb63702dd216e1143
URI written to: /home/guille/Documents/pyro examples/Pyro4-master/examples/banks/Pyro_NS_URI
Name Server started.
Remark: I work on Debian 8 and I've installed:
sudo apt-get install pyro4
sudo apt-get install python2-pyro4
pip install https://pypi.python.org/packages/2.7/s/serpent/serpent-
1.7-py2.py3-none-any.whl
to run this example
Maybe I missed something. Any ideas why this is not working, or things that I'm doing wrong?
thanks in advance.
This work for me:
Run python -m Pyro4.naming in another terminial first:
Not starting broadcast server for localhost.
NS running on localhost:9090 (127.0.0.1)
URI = PYRO:Pyro.NameServer#localhost:9090
and not pyro-ns I've done before for pyro4 as you see this procedure change
While the URI method in the docs is great, another way to connect is register the domain / IP using Pyro4 SimpleServe
Edit:
This was written for use with Python 3, thanks to #Cyberguille for pointing out that raw_input should be used instead of input on the client code when using Python 2.x
Server
Note that 0.0.0.0 exposes it to the world
# saved as greeting-server.py
import Pyro4
#Pyro4.expose
class GreetingMaker(object):
def get_fortune(self, name):
return "Hello, {0}. Here is your fortune message:\n" \
"Behold the warranty -- the bold print giveth and the fine print taketh away.".format(name)
Pyro4.Daemon.serveSimple({
GreetingMaker: 'Greeting',
}, host="0.0.0.0", port=9090, ns=False, verbose=True)
Then running python greeting-server.py to start the script
Client
# saved as greeting-client.py
import Pyro4
ipAddressServer = "" # TODO add your server remote IP here
# Works for Python3, see edit above for notes on Python 2.x
name = input("What is your name? ").strip()
greetingMaker = Pyro4.core.Proxy('PYRO:Greeting#' + ipAddressServer + ':9090')
print(greetingMaker.get_fortune(name)) # call method normally
I think you are mixing python 3 and python 2 versions here, because you wrote you had to install both 'pyro4' and 'python2-pyro4' packages.
I suspect the former is for python 3 and the latter is the legacy python 2 version.
The 'pyro-ns' shell command seems to launch an older, incompatible version of the name server.
I'm new to python and have been struggling with this for quite a while. Okay so I have a program that logs into a server and then constantly pings the server every 10 seconds to see if the status of the server has changed. In the same script I have a function which sends a message to the server.
a simple example of the send method:
def send(self, message):
url = ("https://testserver.com/socket?message=%s") % (message)
req = urllib2.Request(url, None, None)
response = urllib2.urlopen(req).read()
print response
would it be possible for me to call this method from another script while this one was running? as in using the same session. It seems as though when I run my script which calls this function it creates a new instance of it instead of using the current instance of that script causing it to throw my exception saying I am not connected to the server.
Sorry for the noob question. I have tried googling for a while but I cant seem to find the answer. I have read the following but these didn't solve the problem:
Python call function within class
Python code to get current function into a variable?
Hi #nFreeze thanks for the reply I have tried to use ZeroRPC but every time I run the script/example you gave (obviously edited) I run into this error:
Traceback (most recent call last):
File "C:\Users\dwake\Desktop\Web Projects\test.py", line 1, in <module>
import zerorpc
File "C:\Python27\lib\site-packages\zerorpc\__init__.py", line 27, in <module>
from .context import *
File "C:\Python27\lib\site-packages\zerorpc\context.py", line 29, in <module>
import gevent_zmq as zmq
File "C:\Python27\lib\site-packages\zerorpc\gevent_zmq.py", line 33, in <module>
import gevent.event
File "C:\Python27\lib\site-packages\gevent\__init__.py", line 48, in <module>
from gevent.greenlet import Greenlet, joinall, killall
File "C:\Python27\lib\site-packages\gevent\greenlet.py", line 6, in <module>
from gevent.hub import greenlet, getcurrent, get_hub, GreenletExit, Waiter
File "C:\Python27\lib\site-packages\gevent\hub.py", line 30, in <module>
greenlet = __import__('greenlet').greenlet
ImportError: No module named greenlet
Even though I have installed gevent. I'm not sure how to fix this. Have been googling for a good hour now.
What you're looking for is called an RPC server. It allows external clients to execute exposed functions in your app. Luckily python has many RPC options. ZeroRPC is probably my favorite as it is easy to use and supports node.js. Here is an example of how to expose your send method using ZeroRPC:
In your app (server)
import zerorpc
class HelloRPC(object):
def send(self, message):
url = ("https://testserver.com/socket?message=%s") % (message)
req = urllib2.Request(url, None, None)
response = urllib2.urlopen(req).read()
return response
s = zerorpc.Server(HelloRPC())
s.bind("tcp://0.0.0.0:4242")
s.run()
In the other app (client)
import zerorpc
c = zerorpc.Client()
c.connect("tcp://127.0.0.1:4242")
print c.send("RPC TEST!")
Simpliest way is to use UNIX signals. You'll need no third-party libraries.
# your-daemon.py
import signal
from time import sleep
def main():
while True:
print "Do some job..."
sleep(5)
def send():
print "Send your data"
def onusr1(*args):
send()
if __name__ == '__main__':
signal.signal(signal.SIGUSR1, onusr1)
main()
Run in terminal:
$ pgrep -f your-daemon.py | xargs kill -SIGUSR1
Of course, this works only in local machine. Also you can't specify any argument to send function, and if you want to have many handlers, then use RPC as adviced below.
So i've looked around at a few things involving writting an HTTP Proxy using python and the Twisted framework.
Essentially, like some other questions, I'd like to be able to modify the data that will be sent back to the browser. That is, the browser requests a resource and the proxy will fetch it. Before the resource is returned to the browser, i'd like to be able to modify ANY (HTTP headers AND content) content.
This ( Need help writing a twisted proxy ) was what I initially found. I tried it out, but it didn't work for me. I also found this ( Python Twisted proxy - how to intercept packets ) which i thought would work, however I can only see the HTTP requests from the browser.
I am looking for any advice. Some thoughts I have are to use the ProxyClient and ProxyRequest classes and override the functions, but I read that the Proxy class itself is a combination of the both.
For those who may ask to see some code, it should be noted that I have worked with only the above two examples. Any help is great.
Thanks.
To create ProxyFactory that can modify server response headers, content you could override ProxyClient.handle*() methods:
from twisted.python import log
from twisted.web import http, proxy
class ProxyClient(proxy.ProxyClient):
"""Mangle returned header, content here.
Use `self.father` methods to modify request directly.
"""
def handleHeader(self, key, value):
# change response header here
log.msg("Header: %s: %s" % (key, value))
proxy.ProxyClient.handleHeader(self, key, value)
def handleResponsePart(self, buffer):
# change response part here
log.msg("Content: %s" % (buffer[:50],))
# make all content upper case
proxy.ProxyClient.handleResponsePart(self, buffer.upper())
class ProxyClientFactory(proxy.ProxyClientFactory):
protocol = ProxyClient
class ProxyRequest(proxy.ProxyRequest):
protocols = dict(http=ProxyClientFactory)
class Proxy(proxy.Proxy):
requestFactory = ProxyRequest
class ProxyFactory(http.HTTPFactory):
protocol = Proxy
I've got this solution by looking at the source of twisted.web.proxy. I don't know how idiomatic it is.
To run it as a script or via twistd, add at the end:
portstr = "tcp:8080:interface=localhost" # serve on localhost:8080
if __name__ == '__main__': # $ python proxy_modify_request.py
import sys
from twisted.internet import endpoints, reactor
def shutdown(reason, reactor, stopping=[]):
"""Stop the reactor."""
if stopping: return
stopping.append(True)
if reason:
log.msg(reason.value)
reactor.callWhenRunning(reactor.stop)
log.startLogging(sys.stdout)
endpoint = endpoints.serverFromString(reactor, portstr)
d = endpoint.listen(ProxyFactory())
d.addErrback(shutdown, reactor)
reactor.run()
else: # $ twistd -ny proxy_modify_request.py
from twisted.application import service, strports
application = service.Application("proxy_modify_request")
strports.service(portstr, ProxyFactory()).setServiceParent(application)
Usage
$ twistd -ny proxy_modify_request.py
In another terminal:
$ curl -x localhost:8080 http://example.com
For two-way proxy using twisted see the article:
http://sujitpal.blogspot.com/2010/03/http-debug-proxy-with-twisted.html