We Keep Coding

How do i get Gremlin-Python with gremlin-server 3.4.3 to work?

My environment is:
java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
python3 --version
Python 3.7.4
Which should be the only relevant environment parts. Unfortunately it might make a difference that I am on MacOS and using macports which should not be an issue.
I just tested things on Ubuntu 18.04 LTS see http://wiki.bitplan.com/index.php/Gremlin_python#Installlation_automation and the behavior is the same.
So i wanted to tryout gremlin-python.
I found https://pypi.org/project/gremlinpython/
and happily added
gremlinpython
to my requirements.txt
sudo -H pip install -r requirements.txt
ran successfully- great.
Let's find some python code to play with .
https://gist.githubusercontent.com/okram/f193d5616563a69ad5714a42c504276f/raw/b8075410e400e18f18360015945f3760d99d044a/gremlin-python-play.py
has it.
Oh - I need a server ...
Let's download it
http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/tinkerpop/3.4.3/apache-tinkerpop-gremlin-server-3.4.3-bin.zip
unzip it and create a starter file for it as outlined in the example above:
gspath=$HOME/Downloads/apache-tinkerpop-gremlin-server-3.4.3
#$gspath/bin/gremlin-server.sh install org.apache.tinkerpop gremlin-python 3.4.3
$gspath/bin/gremlin-server.sh $gspath/conf/gremlin-server-modern-py.yaml
It doesn't work - get a 599 timeout from tornado.
Look at:
Why can't I connect to Gremlin-Server?
Try hints e.g. using 0.0.0.0 as host
Check the startup.
Oops - there is an error:
Could not initialize gremlin-jython GremlinScriptEngine as init script could not be evaluated
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: gremlin-jython is not an available GremlinScriptEngine
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.lambda$new$4(ServerGremlinExecutor.java:141)
at java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:559)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:136)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.lang.IllegalArgumentException: gremlin-jython is not an available GremlinScriptEngine
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.registerLookUpInfo(CachedGremlinScriptEngineManager.java:95)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.getEngineByName(CachedGremlinScriptEngineManager.java:58)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[WARN] ServerGremlinExecutor - Could not initialize gremlin-python GremlinScriptEngine as init script could not be evaluated
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: gremlin-python is not an available GremlinScriptEngine
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.lambda$new$4(ServerGremlinExecutor.java:141)
at java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:559)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:136)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.lang.IllegalArgumentException: gremlin-python is not an available GremlinScriptEngine
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.registerLookUpInfo(CachedGremlinScriptEngineManager.java:95)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.getEngineByName(CachedGremlinScriptEngineManager.java:58)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
gremlin-python is not an available GremlinScriptEngine
mentions the plugin for the server should be installed. I had assumed that
$gspath/bin/gremlin-server.sh install org.apache.tinkerpop gremlin-python 3.4.3
would do that and when i run it again I get:
Installing dependency org.apache.tinkerpop gremlin-python 3.4.3
Could not install the dependency: a module with the name gremlin-python is already installed
How to get this working from here?
I cut down the python test code to:
# https://pypi.org/project/gremlinpython/
# start GremlinServer
# bin/gremlin-server.sh -i org.apache.tinkerpop gremlin-python 3.2.2-SNAPSHOT
# bin/gremlin-server.sh conf/gremlin-server-modern-py.yaml
# in practice, you really only need the 3 imports below
from gremlin_python import statics
from gremlin_python.structure.graph import Graph
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
# this allows us to do g.V().repeat(out()) instead of g.V().repeat(__.out())-type traversals
#statics.load_statics(globals())
# create a remote connection using RemoteStrategy
graph = Graph()
g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182','g'))
print (g.V().count().value())

I suspect you have a workaround for this already - but the 599 is caused I think because you can't get to the ws port on anything other than a local machine setup. I faced a similar problem with a segregated install.
If you set the host property in your gremlin yaml to 0.0.0.0 - it will listen on all interfaces. Testing everything on localhost gave me a false sense of security - worked when I listened on all network interfaces - which in turn allowed me to have separate clients to the gremlin server.

I think you've just grabbed bits and pieces of code, scripts and answers and they are mismatching in terms of the various versions over multiple years of evolution of TinkerPop.
I think you just need to start over and simplify.
Settle on the must recently released version at 3.4.3 and download Gremlin Server.
Unzip and start the server with: bin/gremlin-server.sh conf/gremlin-server-modern.yaml and initially try to connect from as "localhost" so that no configuration changes are required and to keep things simple.
Validate that you can connect to it with Gremlin Console.
Now install gremlinpython with pip being sure to use 3.4.3.
At which point I would expect that the following code would work out of the box:
g=traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))
print (g.V().count().toList())
Note that I terminated the traversal with toList() (I noticed that your code had no such terminating step) - without that step the traversal will not be executed. I'm not sure if that is part of the problem you're facing.
As for this command:
bin/gremlin-server.sh install org.apache.tinkerpop gremlin-python 3.4.3
you only need to run that if you intend to send native python lambdas to the server. While that may sound neat, keep in mind that you should avoid lambdas if at all possible and that the python ScriptEngine isn't nearly as robust as the Groovy one in terms of caching and execution. If you absolutely had to use a lambda I'd prefer Groovy even if you were coding in Python. That said, I'm not sure why you are getting an error in server startup (fwiw, i actually tested the simple startup with the conf/gremlin-server-modern-py.yaml and it started without error). Perhaps if you retry with a fresh setup you will get a clean start.
Once you have the basic configuration establish, then you can make adjustments, like altering the host configuration in the Gremlin Server yaml file so that you can connect to it from an external IP address.

Based on #stephen mallette's efforts I have created:
https://github.com/WolfgangFahl/gremlin-python-tutorial
which is explained in detail at
http://wiki.bitplan.com/index.php/Gremlin_python
There is a script based installation available:
./run -h
usage: ./run [-h|-i|-s|-c]
-h|--help: show this usage
-i|--install: install prerequisites
-s|--server: start server
-c|--console: start console
-p|--python: start python trial code
It tries to automate the necessary steps
Installation
Gremlin-Server start
Gremlin-Console start (for debugging)
Python script start
1. Installation
run -i
installs
gremlin server
gremlin console
gremlin python module
2. Gremlin-Server start
./run -s
starts the gremlin server with a default yaml-file in foreground
3. Gremlin-Console start (for debugging)
./run -c
starts the gremlin console
4. Python script start
./run -p
starts the python test script.
# minimal imports
from gremlin_python.process.anonymous_traversal import traversal
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
# get the remote graph traversal
g = traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))
vCount=g.V().count().next()
print("The modern graph has %d vertices" % (vCount))
with the expected result:
starting python test code
The modern graph has 6 vertices
The above script has been tested with:
Ubuntu 18.04 LTS
Travis
MacOS and MacPorts

Building VRPN server with Python 3.4 64-bit on Windows

I'm trying to build a VRPN server with Python3 flag using Python 3.4 64-bit on Windows 7 64-bit but there seems to be a problem. I need this for BlenderVR software.
This is my procedure:
1) I use CMake to create makefiles (I'm using 3.4.0 version but I've also tried different ones). I do it with this command (those flags should be there but the result seems to be the same without them anyway):
cmake -G"MinGW Makefiles" -HD:\My\BlenderVR\plugins\vrpn
-BD:\My\BlenderVR\plugins\cmake -DVRPN_BUILD_PYTHON=OFF -DVRPN_BUILD_PYTHON_HANDCODED_2X=OFF -DVRPN_BUILD_PYTHON_HANDCODED_3X=ON
I used to add those flags as well but it seems that it can find Python without them
-DPYTHON_INCLUDE_DIR=D:\My\BlenderVR\Required\Python3\include
-DPYTHON_LIBRARY=D:\My\BlenderVR\Required\Python3\libs\python34.lib
Python is correctly found and this operation doesn't throw any error.
2) Then I use mingw32-make.exe to build it and I get this error:
[ 90%] Linking CXX shared module vrpn.pyd D:/My/BlenderVR/Required/Python3/libs/python34.lib: error adding
symbols: File f ormat not recognized collect2.exe: error: ld
returned 1 exit status
python\CMakeFiles\vrpn-python.dir\build.make:505: recipe for
target 'python/vrpn .pyd' failed mingw32-make[2]: * * *
[python/vrpn.pyd] Error 1 CMakeFiles\Makefile2:3247: recipe for
target 'python/CMakeFiles/vrpn-python.dir/ all' failed
mingw32-make[1]: * * * [python/CMakeFiles/vrpn-python.dir/all]
Error 2 Makefile:159: recipe for target 'all' failed
mingw32-make: [all] Error 2
vprn.pyd is the crucial thing for my future work.
I figured out that it needs libpython34.a file (probably). When I created it and copied to Python3/libs folder it worked and finished without errors but the crated vprn.pyd didn't worked as it should.
What I need is to get import vrpn to work with this simple test in python (appending path where vrpn.pyd was build):
import sys
sys.path.append('D:/My/BlenderVR/plugins/cmake/python')
import vrpn
It lags my whole computer for a while and then pops out that Python has stop working.
I suspect that problem is in the libpython34.a file that I created doing this:
gendef python34.dll (in Windows/System32)
dlltool -D python34.dll -d python34.def -l libpython34.a
I don't how else should I get the libpython file. I've tried various versions of CMake and MinGW (like MinGWPy, TDM, w64) with many CMake flags. I was able to make it work using 32-bit Python but I need 64-bit version otherwise it is not working with BlenderVR enviroment.
I know this is very specific problem and probably kind of confusing at first but I didn't know how else to put it. I'll be glad for anything that could help. Thank you.

mingwpy should be installed with pip (until it is officially released at PYPI):
pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
all necessary import files are atomatically copied into the python\libs folder.
If python\Scripts is in the PATH it should work out of the box.
You have to make sure, that Blender Python is equiped with two import files
D:\My\BlenderVR\Required\Python3\libs\libpython\libpython34.dll.a
D:\My\BlenderVR\Required\Python3\libs\libpython\libmsvcr100.a

ImportError: No module named MySQLdb - Debian 6.0

I got a problem in Python while try to import mysql. The beginning of my code is (I'm using monkeyrunner to run script) :
#!/usr/bin/env python
import sys
import MySQLdb
.
.
.
etc
Each time I try to run the code, I got the current error:
dani#debian:~/public/Yandex Maps Test$ monkeyrunner test.py
130413 00:49:22.066:S [main] [com.android.monkeyrunner.MonkeyRunnerOptions] Script terminated due to an exception
130413 00:49:22.066:S [main] [com.android.monkeyrunner.MonkeyRunnerOptions]Traceback (most recent call last):
File "/home/dani/public/Yandex Maps Test/test.py", line 4, in <module>
import MySQLdb
ImportError: No module named MySQLdb
130413 00:49:22.066:S [main] [com.android.monkeyrunner.MonkeyRunnerOptions] at org.python.core.PyException.fillInStackTrace(PyException.java:70)
.
.
.
etc
I tried to re-install mysql for python:
sudo apt-get reinstall python-mysqldb mysql-server
But nothing helped. Any ideas or tips?
Regards,,,

The problem is that monkeyrunner isn't the same interpreter as your default system Python, and doesn't share the same site-packages, etc.
That #!/usr/bin/env python at the top of your script is misleading. If you actually run your script with /usr/bin/env python test.py, or just ./test.py (assuming chmod +x), it will work. But you're not, you're running it with monkeyrunner.
So, nothing you install for your system Python is going to be usable within monkeyrunner. You have to install it for monkeyrunner instead.
Moreover, as the docs say:
The monkeyrunner tool uses Jython, a implementation of Python that uses the Java programming language.
While Jython is Python, and implements the same Python language and (almost) standard library as CPython, it doesn't implement the CPython C API, and therefore can't use C-API extensions like MySQLdb. (They're actually working on a clever wrapper that will change this in the future, but that doesn't help you today.)
There are other MySQL drivers that work with both CPython and Jython (because they use pure Python), such as PyMySQL. But the simplest solution is to just use Java's JDBC drivers. As the Jython FAQ suggests, the easiest way to do that is:
Use zxJDBC which gives data database connectivity from Jython using the Python DB API 2.0 interface…
Unlike using JDBC directly, using zxJDBC often means you only need to change one line in your MySQLdb code to make it work.

i have the same question with u.
and i tried to use zxJDBC to fix it.
but when i run the script, it cannot find "org.gjt.mm.mysql.Driver" either "com.mysql.jdbc.Driver". Even i put mysql-connector-java-5.1.34-bin.jar and zxJDBC.jar under system CLASSPATH.
i tried run this script under jython. it worked.
so i think this question is related with monkeyrunner.

How to bundle Python dependancies in IronWorker?

I'm writing a simple IronWorker in Python to do some work with the AWS API.
To do so I want to use the boto library which is distributed via PyPi repository. The boto library is not installed by default in the IronWorker runtime environment.
How can I bundle the boto library dependancy with my IronWorker code?
Ideally I'm hoping I can use something like the gem dependancy bundling available for Ruby IronWorkers - i.e in myRuby.worker specify
gemfile '../Gemfile', 'common', 'worker' # merges gems from common and worker groups
In the Python Loggly sample, I see that the hoover library is used:
#here we have to include hoover library with worker.
hoover_dir = os.path.dirname(hoover.__file__)
shutil.copytree(hoover_dir, worker_dir + '/loggly') #copy it to worker directory
However, I can't see where/how you specify which hoover library version you want, or where to download it from.
What is the official/correct way to use 3rd party libraries in Python IronWorkers?

Newer iron_worker version has native support of pip command.
So, you need:
runtime "python"
exec "something.py"
pip "boto"
pip "someotherpip"
full_remote_build true

[edit]We've worked on our toolset a bit since this answer was written and accepted. The answer from my colleague below is the recommended course moving forward.[/edit]
I wrote the Python client library for IronWorker. I'm also employed by Iron.io.
If you're using the Python client library, the easiest (and recommended) way to do this is to just copy over the library's installed folder, and include it when uploading the package. That's what the Python Loggly sample is doing above. As you said, that doesn't specify a version or where to download the library from, because it doesn't care. It just takes the one installed on your system and uses it. Whatever you get when you enter "import boto" on your local machine is what would be uploaded.
The other option is using our CLI to upload your worker, with a .worker file.
To do this, here's what you'd need to do:
Create a botoworker.worker file:
runtime "binary"
build 'pip install --install-option="--prefix=`pwd`/pips" boto'
file 'botoworker.py'
exec "botoworker.sh"
That second line is the pip command that will be run to install the dependency. You can modify it like you would any pip command run from the command line. It's going to execute that command on the worker during the "build" phase, so it's only executed once instead of every time you run a task.
The third line should be changed to the Python file you want to run--it's your Python worker file. Here's the one we used to test this:
import boto
If you save that as botoworker.py, the above should work without any modification. :)
The fourth line is a shell script that's going to actually run your worker. I've included the one we used below. Just save it as botoworker.sh, and you won't have to worry about modifying the .worker file above.
PYTHONPATH="$HOME/pips/lib/python2.7/site-packages:$PYTHONPATH" python botoworker.py "$#"
You'll notice it refers to your Python file--if you don't name your Python file botoworker.py, remember to change it here, too. All this does is set your PYTHONPATH to include the installed library, and then runs your Python file.
To upload this, just make sure you have the CLI installed (gem install iron_worker_ng, making sure your Ruby version is 1.9.3 or higher) and then run "iron_worker upload botoworker" in your shell, from the same directory your botoworker.worker file is in.
Hope this helps!

How can I import hbase in python?

I'm trying to play around with hbase in python and I am using the cloudera repository to install the hadoop/hbase packages. It seems to work as I can access and work on the database using the shell but its not fully working within python.
I know to communicate with hbase I need thrift so I downloaded and complied it from source, I can import thrift into python but when I do from hbase import Hbase, I get module not found errors.
Does anyone know what package/module I would need to get it to work? I tried to look around easy_install and yum(I'm using centos6) and no luck. I did find an article where a person using debain installed it by doing sudo aptiutde install python-hbase I don't have that command/package, so I'm not sure how to get it(or if I have to compile from source to get it).
Also if it helps, I installed most of the base from cloudera and followed some instructions(the ones didn't require install) from http://yannramin.com/2008/07/19/using-facebook-thrift-with-python-and-hbase/
Any help/tips/suggestions would be great.
Thanks!

Have a look at HappyBase (see https://github.com/wbolster/happybase for info). It is the modern way to interact with HBase from Python. It covers the complete Thrift API but wraps it in a much better interface.

Okay, I figured it out. If anyone else is having problems with this in the future its actually pretty easy. In the step where you run thrift --gen py Hbase.thrift, it creates a hbase folder in the location you ran that command. Simply take that command and copy it to your default module folder(or in the folder where you run your program and it should work).

search for /src/contrib/thriftfs/gen-py under hadoop installation folder
Copy the output of thrift --gen py Hbase.thrif onto the location below (part till /home/hadoop/data/ will differ in your case) /home/hadoop/data/hadoop-1.0.4/src/contrib/thriftfs/gen-py
then
$ python
import sys
sys.path.append("/home/hadoop/data/hadoop-1.0.4/src/contrib/thriftfs/gen-py")
import hbase
It should work now