I'm trying to use the undocumented library uwhoisd and I cannot figure out how to successfully call the whois() function on the UWhois object.
I start the deamon by calling a file UwhoisDeamon.py with following content:
import uwhoisd
uwhoisd.main()
The argument I give to python is the *.ini file that was included in the extra directory when I downloaded the source code
Starting the deamon is successful with following output:
2016-04-21 19:00:19,830 INFO:uwhoisd Reading config file at
'/path/to/my/ini/uwhoisd.ini' 2016-04-21
19:00:19,838 INFO:uwhoisd Listen on 0.0.0.0:4243 2016-04-21
19:00:19,842 INFO:uwhoisd Caching activated [2016/04/21 17:00:19]
{diesel} WARNING|Starting diesel
Now I'm trying perform a whois on google.com by starting a separate python instance and I get slapped in the face (I reused this code):
from os import path
import uwhoisd
from uwhoisd.utils import make_config_parser
HERE = path.dirname(__file__)
def create_uwhois():
"""Prepare a UWhois object for testing."""
config = path.join(HERE, 'uwhoisd.ini')
parser = make_config_parser(uwhoisd.CONFIG, config)
uwhois = uwhoisd.UWhois()
uwhois.read_config(parser)
return uwhois
uwhois = create_uwhois()
# uwhois = uwhoisd.UWhois()
result = uwhois.whois("google.com")
print result
And here the result:
Traceback (most recent call last):
File "/home/user/.eclipse/org.eclipse.platform_3.8_155965261/plugins/org.python.pydev_3.9.0.201411111611/pysrc/pydevd.py", line 2183, in <module>
globals = debugger.run(setup['file'], None, None)
File "/home/user/.eclipse/org.eclipse.platform_3.8_155965261/plugins/org.python.pydev_3.9.0.201411111611/pysrc/pydevd.py", line 1622, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/path/to/my/project/Whois/WhoisChecker.py", line 17, in <module>
result = uwhois.whois("google.com")
File "/usr/local/lib/python2.7/dist-packages/uwhoisd/__init__.py", line 137, in whois
with net.WhoisClient(server, port) as client:
File "/usr/local/lib/python2.7/dist-packages/diesel/client.py", line 16, in __init__
ip = self._resolve(self.addr)
File "/usr/local/lib/python2.7/dist-packages/diesel/client.py", line 21, in _resolve
return resolve_dns_name(addr)
File "/usr/local/lib/python2.7/dist-packages/diesel/resolver.py", line 67, in resolve_dns_name
except (NotFound, Timeout):
File "/usr/local/lib/python2.7/dist-packages/diesel/util/lock.py", line 24, in __exit__
self.release()
File "/usr/local/lib/python2.7/dist-packages/diesel/util/lock.py", line 18, in release
fire(self)
File "/usr/local/lib/python2.7/dist-packages/diesel/core.py", line 92, in fire
return current_loop.fire(*args, **kw)
AttributeError: 'NoneType' object has no attribute 'fire'
It seems to me that the diesel library which is used by uwhoisd to communicate between client and deamon application requires some special conditions to make it work. From what I can see the mysterious current_loop is not running. However I never used diesel before and I have no idea what to do. It seems to me that I have to somehow start that diesel loop before I can perform those whois queries.
A working example would be great. The main reason why I'm choosing uwhoisd is that I can perform whois on exotic websites such as "win.win". It also caches responses which means more speed and less network.
Author of uwhoisd here.
First up, get rid of UwhoisDeamon.py: it's a daemon, not a library. You're meant to run the daemon and then query it with a whois client. You don't have to write any additional code to get it up and running. It already installs a working console script that does the right thing.
When you install it, say, into a virtual environment, it will create a script called uwhoisd that you use to run the daemon. Here's an example:
$ virtualenv env
$ env/bin/pip install uwhoisd
Downloading/unpacking uwhoisd
Downloading uwhoisd-0.0.7-py2-none-any.whl
... installation log ...
Successfully installed uwhoisd diesel greenlet twiggy pyopenssl flask http-parser dnspython six cryptography Werkzeug Jinja2 itsdangerous idna pyasn1 setuptools enum34 ipaddress cffi MarkupSafe pycparser
Cleaning up...
If you run the daemon without, here's what you should get, assuming you had all the necessary system packages installed:
$ env/bin/uwhoisd
Usage: uwhoisd <config>
uwhoisd requires a configuration file that tells it how to talk to the various WHOIS servers it needs to be able to query. You can download the file like so:
$ wget https://raw.githubusercontent.com/kgaughan/uwhoisd/master/extra/uwhoisd.ini
Running the daemon with this config file should give you the following:
$ env/bin/uwhoisd uwhoisd.ini
2016-05-03 17:41:25,028 INFO:uwhoisd Reading config file at 'uwhoisd.ini'
2016-05-03 17:41:25,038 INFO:uwhoisd Listen on 0.0.0.0:4243
2016-05-03 17:41:25,058 INFO:uwhoisd Caching activated
The second line indicates what address and port the daemon is running on. In this case, it's running on all interfaces on port 4243, so you can now issue requests to it on that port (in a different terminal window):
$ whois -h localhost -p 4243 stackoverflow.com
Domain Name: STACKOVERFLOW.COM
Registry Domain ID: 108907621_DOMAIN_COM-VRSN
...
And you should see the following additional log lines in the terminal the daemon is running in:
2016-05-03 17:43:36,841 INFO:uwhoisd Querying whois.verisign-grs.com about stackoverflow.com
2016-05-03 17:43:37,283 INFO:uwhoisd Recursive query to whois.name.com about stackoverflow.com
You shouldn't be using uwhois's internals directly: it's a daemon and isn't designed to be used as a library, certainly not outside of diesel's event loop like that, which is why you got that traceback. Run the daemon with the config file, however, and you'll be golden.
Unfortunately, I haven't had time to write all the documentation for the daemon it needs, but I'm hoping to resolve this soon.
Related
https://raw.githubusercontent.com/paramiko/paramiko/master/demos/demo_server.py
I see the above demo_server of paramiko. But I don't see the instructions on how to run it. I run the following ./demo_server.py command. But once I run ssh robey#127.0.0.1 -p 2200, the server fails. Could anybody let me know the complete steps on how to run this example? Thanks.
$ python3 ./demo_server.py
Read key: 60733844cb5186657fdedaa22b5a57d5
Listening for connection ...
Got a connection!
*** Caught exception: <class 'ImportError'>: Unable to import a GSS-API / SSPI module!
Traceback (most recent call last):
File "./demo_server.py", line 140, in <module>
t = paramiko.Transport(client, gss_kex=DoGSSAPIKeyExchange)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/paramiko/transport.py", line 445, in __init__
self.kexgss_ctxt = GSSAuth("gssapi-keyex", gss_deleg_creds)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/paramiko/ssh_gss.py", line 107, in GSSAuth
raise ImportError("Unable to import a GSS-API / SSPI module!")
ImportError: Unable to import a GSS-API / SSPI module!
$ ssh robey#127.0.0.1 -p 2200
kex_exchange_identification: read: Connection reset by peer
I managed to get it to work, make sure to go through these steps (Thank you Sandeep for the pip insight), chances are you may be missing the Kerberos dependencies:
You may need to perform pip install gssapi in the CLI if that has not already been done (I'm using the Windows Command Prompt, Linux/WSL might need pip3 instead depending on your version of python)
From there you will need to import the gssapi library into the code at the top with the other imported libraries, so just call
import gssapi in demo_server.py
After running demo_server.py again, the CLI should eventually say something like it is missing files located in Program Files\ MIT\ Kerberos\ bin, as Kerberos is a dependency for gssapi, you can install it from here:
https://web.mit.edu/KERBEROS/dist/
Make sure you do custom install if you're not sure where it will be downloaded, so that you can set up the file location where the CLI says is missing in step 3 (Should automatically say Program Files\ MIT). I unchecked the boxes for auto-start and tickets, but not sure as to what your preferences may be. After all that, a computer restart is required.
Good day,
I'm a student and I would just like to ask for a minute of your time.
I'm working on a barcode reader connected via USB port to a board name Arduino Yun. This board runs a version of embedded linux derived from OpenWrt using a microprocessor named Atheros AR9331
I would like to ask you, what's necessary to make the Python Evdev binding (python-evdev.readthedocs.org/en/latest/), to be able to run in this type of MIPS microarchitecture? At the momento, it's only for Ubuntu and ArchLinux.
I'm kind of guessing that cross compilation would be needed, or the indication of the usage of a specific C compiler inside this linux.
The current python version supported for OpenWrt is 2.7.3
I already know , if you compile C code in your PC, the resulting executable will only run in this type of architecture. If you use that compiled program inside the microprocessor, it wont work.
I've used this binding without trouble within ubuntu in my PC. I followed the instructions, python setup.py install, with a previous installation of setuptools, and it worked just fine.
But regarding OpenWrt, this was not the case.
The python script I'm using requires this library within the first line of code in order to reach the data from the device (it works like a keyboard /dev/input/event0):
#!/usr/bin/env python
from evdev import InputDevice, ecodes, list_devices
from select import select
I've seen suggestions of copying the entire library inside the arduino, and run the script inside the same folder. But it doesn't work, since the evdev module has files created with the architecture of the PC and not the MIPS.
So, what are the messages displayed for the error?
If you run python setup.py install in Openwrt to try to install the evdev binding, this appears on screen:
File "setup.py", line 10, in <module>
from setuptools.command.develop import develop
ImportError: No module named setuptools.command.develop
It's obvious from here that you need the module aforementioned. So, I tried to install it with this script (pypi.python.org/pypi/setuptools):
python ez_setup.py
And the output shows this:
Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-11.3.1.zip
Traceback (most recent call last):
File "ez_setup.py", line 332, in <module>
sys.exit(main())
File "ez_setup.py", line 327, in main
downloader_factory=options.downloader_factory,
File "ez_setup.py", line 287, in download_setuptools
downloader(url, saveto)
File "ez_setup.py", line 209, in download_file_curl
_clean_check(cmd, target)
File "ez_setup.py", line 169, in _clean_check
subprocess.check_call(cmd)
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['curl','https://pypi.python.org/packages/source/s/setuptools/setuptools-11.3.1.zip', '--silent', '--output', '/mnt/sda1/evdev-0.4.6/setuptools-11.3.1.zip']' returned non-zero exit status 60
I pressume this output is due to the fact that pypi doesn't exist for the python 2.7.3 in OpenWrt , only for newer versions and other architectures. Evedv binding is requiring the setuptools module in order to make things easier and standard, but if the binding is not supported for the target architecture, what's needed to be able to use it anyways?
Thanks for your time,
Good day everyone,
The solution was provided by Georgi Valkov. He is the creator of the python-evdev binding. I contacted him directly, and he was so kind that he cross compiled a version for the OpenWrt / Yun .
You can install the package using the openwrt package manager - opkg. The installation process is along the lines of:
$ opkg update
$ opkg install /path/to/python-evdev_0.4.7-1_ar71xx.ipk
To verify that the install was successful:
$ opkg files python-evdev
/usr/lib/python2.7/site-packages/evdev-0.4.7-py2.7.egg-info
/usr/lib/python2.7/site-packages/evdev/genecodes.py
/usr/lib/python2.7/site-packages/evdev/ff.py
/usr/lib/python2.7/site-packages/evdev/_input.so
/usr/lib/python2.7/site-packages/evdev/device.py
/usr/lib/python2.7/site-packages/evdev/events.py
/usr/lib/python2.7/site-packages/evdev/__init__.py
/usr/lib/python2.7/site-packages/evdev/ecodes.py
/usr/lib/python2.7/site-packages/evdev/_ecodes.so
/usr/lib/python2.7/site-packages/evdev/util.py
/usr/lib/python2.7/site-packages/evdev/uinput.py
/usr/lib/python2.7/site-packages/evdev/_uinput.so
This works just fine. Thanks.
PS. If someone needs the file, please contact me. Georgi sent me this address, but I didn't download the file from there because he sent it to me over email.
https://github.com/gvalkov/openwrt-packages-yun/blob/master/lang/python-evdev/Makefile
In the output, you can see that curl returned the status code 60. According to man curl
60 Peer certificate cannot be authenticated with known CA certifi‐
cates.
According to the setuptools page, you can instead use python ez_setup.py --insecure but obviously do that at your own risk. Alternatively you could do what the advanced instructions say and manually download the setuptools tarball, verify its md5 hash yourself, and install it using its setup.py .
I have scrapy crawler scraping thru sites. On some occasions scrapy kills itself due to RAM issues. I rewrote the spider such that it can be split and run for a site.
After the initial run, I use subprocess.Popen to submit the scrapy crawler again with new start item.
But I am getting error
ImportError: No module named shop.settingsTraceback (most recent call last):
File "/home/kumar/envs/ishop/bin/scrapy", line 4, in <module> execute()
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/cmdline.py", line 109, in execute settings = get_project_settings()
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/utils/project.py", line 60, in get_project_settings settings.setmodule(settings_module_path, priority='project')
File "/home/kumar/envs/ishop/lib/python2.7/site-packages/scrapy/settings/__init__.py", line 109, in setmodule module = import_module(module)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module __import__(name)ImportError: No module named shop.settings
The subprocess cmd is
newp = Popen(comm, stderr=filename, stdout=filename, cwd=fp, shell=True)
comm -
source /home/kumar/envs/ishop/bin/activate && cd /home/kumar/projects/usg/shop/spiders/../.. && /home/kumar/envs/ishop/bin/scrapy crawl -a category=laptop -a site=newsite -a start=2 -a numpages=10 -a split=1 'allsitespider'
cwd - /home/kumar/projects/usg
I checked sys.path and it is correct ['/home/kumar/envs/ishop/bin', '/home/kumar/envs/ishop/lib64/python27.zip', '/home/kumar/envs/ishop/lib64/python2.7', '/home/kumar/envs/ishop/lib64/python2.7/plat-linux2', '/home/kumar/envs/ishop/lib64/python2.7/lib-tk', '/home/kumar/envs/ishop/lib64/python2.7/lib-old', '/home/kumar/envs/ishop/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/home/kumar/envs/ishop/lib/python2.7/site-packages']
But looks like the import statement is using "/usr/lib64/python2.7/importlib/__init__.py" instead of my virtual env.
Where am I wrong? Help please?
Looks like the settings in not being loaded properly. One solution would be to build an egg and deploy it in the env before starting the crawler.
Official docs, Eggify scrapy project
I suggest let python focus on the scrape task and use something else for process control. If it were me, I'd write a small bash script to run your program.
Test that the launcher script works by running it with env -i yourscript.sh because that will make sure it runs without any inherited environment settings.
Once the bash script works correctly, including setting up virtualenv etc, you could have python run that bash script, not python. You've sidestepped any strange environment issues at that point and got yourself a pretty solid launcher script.
Even better, given you have the bash script at that point, use a "proper" process controller (daemontools, supervisor...) spin up the process, restart on crash, etc.
A little background: I've been trying to make a restful server that can query and insert via the management API. After banging my head against the wall using node.js and javascript I switched over to python knowing it has more support.
Currently I am trying to follow the GA Tutorial: Hello Analytics API with a slight twist. I trigger the what was the main method in hello_analytics_api_v3.py when trying to access a particular end point on the server. This is only for testing. The method is called insertExperiment and attempts to insert an experiment into a private Google Analytics account.
However I'm always receiving the exception:
NotImplementedError('The gflags library must be installed to use tools.run(). Please install gflags or preferrably switch to using tools.run_flow().',)
Here is the full trace stack:
File "/0zzz/bottle.py", line 764, in _handle
return route.call(**args)
File "/0zzz/bottle.py", line 1575, in wrapper
rv = callback(*a, **ka)
File "server.py", line 39, in server_static
hello_analytics_api_v3.insertExperiment("xxxxxxx", "xxxxxxxxx", "xxxxxx", experiment_body)
File "/0zzz/hello_analytics_api_v3.py", line 11, in insertExperiment
service = hello_analytics_api_v3_auth.initialize_service()
File "/0zzz/hello_analytics_api_v3_auth.py", line 32, in
initialize_service
credentials = prepare_credentials()
File "/0zzz/hello_analytics_api_v3_auth.py", line 25, in
prepare_credentials
credentials = run(FLOW, storage)
File "/0zzz/oauth2client/tools.py", line 241, in run
'The gflags library must be installed to use tools.run(). '
NotImplementedError: The gflags library must be installed to use
tools.run(). Please install gflags or preferrably switch to using
tools.run_flow().
A few extra notes:
I have tried this exact same thing (invoking main on pinging the correct address) when copying the exact code from the Tutorial.
I have write access with the management API -- so that's not a problem here at all.
I thought there might be a namespace issue with one of my libraries (which also imports a run) but resolved this by renaming the other library.
GA Client Library is installed! as per the instructions on Hello Analytics API.
Thank you for any help you can provide!
I received the same error while running through a tutorial for google APIs called 'Google APIs Console Help'. The fix was simple in my case, just update the gflags library:
easy_install --upgrade python-gflags
You need to install gflags library.
$ wget https://python-gflags.googlecode.com/files/python-gflags-2.0.tar.gz
$ tar xfvz python-gflags-2.0.tar.gz ; cd python-gflags-2.0
$ python setup.py install
Im running a solaris server which uses supervisor to monitor some Python applications.
Previously, I could run the command:
paster serve /opt/pyapps/menuadmin/prod.ini
from any directory on the server. There were some recent issues and the /opt folder was restored from a previous backup. This folder contained all of the applications including supervisor.
Now we are facing issues where supervisor will not start the applications because of "version conflicts" in Pylons.
This is where it gets weird and it makes no sense why these errors would occur.
If I run the paster command from outside of the program directory, it will throw the version conflict error. eg:
cd /
paster serve /opt/pyapps/menuadmin/prod.ini
Traceback (most recent call last):
File "/opt/csw/bin/paster", line 8, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/opt/csw/lib/python2.6/site-packages/PasteScript-1.7.5-py2.6.egg/paste/script/command.py", line 93, in run
commands = get_commands()
File "/opt/csw/lib/python2.6/site-packages/PasteScript-1.7.5-py2.6.egg/paste/script/command.py", line 135, in get_commands
plugins = pluginlib.resolve_plugins(plugins)
File "/opt/csw/lib/python2.6/site-packages/PasteScript-1.7.5-py2.6.egg/paste/script/pluginlib.py", line 82, in resolve_plugins
pkg_resources.require(plugin)
File "/opt/csw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py", line 626, in require
File "/opt/csw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py", line 528, in resolve
pkg_resources.VersionConflict: (Pylons 0.9.7 (/opt/csw/lib/python2.6/site-packages/Pylons-0.9.7-py2.6.egg), Requirement.parse('Pylons>=0.10'))
But if I run the command from inside the program directory, it will run fine. eg:
cd /opt/pyapps/menuadmin/
paster serve /opt/pyapps/menuadmin/prod.ini
Starting server in PID 29902.
serving on http://127.0.0.1:3002
I absolutely cannot get my head around why this would happen!
Any thoughts or comments at all are appreciated!!!!
Based upon what you have said it seems you are running two different version of paster. The first version is running the older Pylons package 0.9.7, whilst the second has the more up to date version that meets or exceeds your app's requirements.
What I would do is first check which version of paster you are running. From outside of the project just run:
which paster
Then run the same command again within the project directory and compare the results. I suspect that you will find that the paths differ. If that is the case then all you need to do is update the version of pylons for the first version, which I'm guessing is the global install.
However, as others have commented it would be better to run apps within virtualenv, especially if as you seem to indicate you have multiple virtualenv and thus multiple projects. Trust me when I say it will save you from loads of headaches later on, from someone that didn't do this originally.