Suppose Hive is installed in say, "g" cluster. I do not have access to the Gold Cluster. I'm doing my python development work in "s" cluster. I can access Hive from the "s" cluster and run queries.
I have the below code to connect to Hive from a Python script running in "s" cluster.
some_table is a table that already exists in Hive. I would like to execute a simple select * from some_table command to get some results.
import sys
sys.path.append("/usr/lib/hive/lib/py")
from hive_service import ThriftHive
from hive_service.ttypes import HiveServerException
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
transport = TSocket.TSocket('what-ever-server', what-ever-port)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ThriftHive.Client(protocol)
transport.open()
print "connect success"
client.execute("SELECT * FROM some_table")
print client.fetchAll()
print "executed"
But I get the below error after "connect success" is printed. I am assuming that the connection was successful.
Traceback (most recent call last):
File "hiveConnect.py", line 30, in <module>
row = client.execute("SELECT * FROM some_table")
File "/usr/lib/hive/lib/py/hive_service/ThriftHive.py", line 68, in execute
self.recv_execute()
File "/usr/lib/hive/lib/py/hive_service/ThriftHive.py", line 79, in recv_execute
(fname, mtype, rseqid) = self._iprot.readMessageBegin()
File "/usr/lib/hive/lib/py/thrift/protocol/TBinaryProtocol.py", line 137, in readMessageBegin
name = self.trans.readAll(sz)
File "/usr/lib/hive/lib/py/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz-have)
File "/usr/lib/hive/lib/py/thrift/transport/TTransport.py", line 155, in read
self.__rbuf = StringIO(self.__trans.read(max(sz, self.DEFAULT_BUFFER)))
File "/usr/lib/hive/lib/py/thrift/transport/TSocket.py", line 94, in read
raise TTransportException('TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException: None
What am I doing wrong in this code? I am not experiencing any error while connecting to Hive using the server-name and port, so I'm assuming everything is fine there and that the connection to Hive is not the issue.
Well after a bit more research I found that it was actually HiveServer2 that was being used with a port number of 10,000. After that I had to install pyhs2 for it to work properly.
Related
I installed SQL Anywhere 17 client. I ran /opt/sqlanywhere17/bin64/sa_config.sh And still have problem with sqlanydb.InterfaceError: ('Could not load dbcapi. Tried:None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
#!/usr/bin/python3.6
# -*- coding: utf-8 -*-
import ctypes
import sqlanydb
conn = sqlanydb.connect(UserID='XXX', Password='XXX', Host='10.10.10.10', DatabaseName='XXX', ServerName='XXX')
curs = conn.cursor()
curs.close()
conn.close()
Log:
Traceback (most recent call last):
File "./systest.py", line 6, in <module>
conn = sqlanydb.connect(UserID='XXX', Password='XXX', Host='10.10.10.10', DatabaseName='XXX', ServerName='XXX')
File "/usr/local/lib/python3.6/site-packages/sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "/usr/local/lib/python3.6/site-packages/sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "/usr/local/lib/python3.6/site-packages/sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "/usr/local/lib/python3.6/site-packages/sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
sqlanydb.InterfaceError: ('Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
Thanks for your help in advance
I faced the same issue and took me 2 days to solve it.
Here is the solution, your solution is in point 2.
If running on Linux or one of the Unix platforms, you'll need to set
up your environment before launching Python. So, download the sql
anywhere12 / 17 and install. ( You have done this)
open the terminal and run sa_config{.sh|.csh} script so your Python application can locate the required libraries (including libdbcapi12.dll).
Make sure that you run sa_config.sh file and app.py in same terminal otherwise you will face the same issue.
Example, I have created the bash file which run first.
. /opt/sqlanywhere17/bin32/sa_config.sh
python app.py
source : Full answer can be found here on sqlanywhere forum
I am trying to connect my raspberry pi to a Bluetooth OBD adapter. I have verified that the OBD device is working by connecting to it from the android phone using Torque lite app.
I am following this tutorial to connect the raspberry pi to the OBD adapter. I am able to pair with the OBD adapter and then trust it. Then I manually create the "/dev/rfcomm0" abstraction. As per the tutorial "screen /dev/rfcomm0" is supposed to print some output. But for me it just says "Screen is terminating" and it exits.
I tried running the "obd_reader.py" script, but it is throwing the following error:
python obd_reader.py
Ports:
['/dev/rfcomm0']
[obd.obd] ======================= python-OBD (v0.6.1) =======================
[obd.obd] Explicit port defined
[obd.elm327] Initializing ELM327: PORT=/dev/rfcomm0 BAUD=auto PROTOCOL=auto
Traceback (most recent call last):
File "obd_reader.py", line 11, in <module>
connection = obd.OBD(ports[0])
File "/usr/local/lib/python2.7/dist-packages/obd/obd.py", line 58, in __init__
self.__connect(portstr, baudrate, protocol) # initialize by connecting and loading sensors
File "/usr/local/lib/python2.7/dist-packages/obd/obd.py", line 85, in __connect
self.interface = ELM327(portstr, baudrate, protocol)
File "/usr/local/lib/python2.7/dist-packages/obd/elm327.py", line 137, in __init__
if not self.set_baudrate(baudrate):
File "/usr/local/lib/python2.7/dist-packages/obd/elm327.py", line 265, in set_baudrate
return self.auto_baudrate()
File "/usr/local/lib/python2.7/dist-packages/obd/elm327.py", line 293, in auto_baudrate
response = self.__port.read(1024)
File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 460, in read
raise SerialException('device reports readiness to read but returned no data (device disconnected?)')
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected?)
I tried a suggestion provided here, but there was no change in output. This was the change suggested:
You need to add the SP profile editing this:
sudo nano /etc/systemd/system/dbus-org.bluez.service
Then add the compatibility flag -C for adding the SP to sdptool. Modify the lines from the file above to:
ExecStart=/usr/lib/bluetooth/bluetoothd -C
ExecStartPost=/usr/bin/sdptool add SP
I feel that I am creating or configuring the rfcomm device in the wrong way. But not sure how to fix this. Can someone please suggest a solution?
It seems that line 27 of the script you are running you are missing a dot in the query:
res = connection.query(obd.commands[command])
Tt should be like this as far as I know:
res= connection.query(obd.commands.[command])
I'm getting the error:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at
(http://www.w3.org/2001/XMLSchema.x sd), failed
when passing a Doctor (constructed with ImportDoctor) to the suds Client constructor.
I'm working on two Windows machines, both of them got the same version of suds installed, but only one of them rises the error above.
Could someone guide me here to know why this error rises?, so I can figure out what's missing on the machine where it happens?.
Thanks in advance!!!.
UPDATE: I don't really know if this is important, but it's worth noting that my Windows machine that is rising the error is an Amazon Web Services instance. At my local machine everything's working well!.
UPDATE: Here's some code I ran at the python interpreter of the machine I mentioned. Here you can detail how the error is rising...
>>> from suds.client import Client
>>> from suds.xsd.doctor import ImportDoctor, Import
>>> missing_import = Import("http://www.w3.org/2001/XMLSchema")
>>> missing_import.filter.add("http://tempuri.org/")
>>> doctor = ImportDoctor(missing_import)
>>> client = Client("http://etcfulfill.ebooks.com/Fulfillment.asmx?wsdl")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "suds\client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "suds\reader.py", line 152, in open
d = self.fn(url, self.options)
File "suds\wsdl.py", line 159, in __init__
self.build_schema()
File "suds\wsdl.py", line 220, in build_schema
self.schema = container.load(self.options)
File "suds\xsd\schema.py", line 95, in load
child.dereference()
File "suds\xsd\schema.py", line 323, in dereference
midx, deps = x.dependencies()
File "suds\xsd\sxbasic.py", line 422, in dependencies
raise TypeNotFound(self.ref)
suds.TypeNotFound: Type not found: '(schema, http://www.w3.org/2001/XMLSchema, )'
>>> client = Client("http://etcfulfill.ebooks.com/Fulfillment.asmx?wsdl", doctor=doctor)
No handlers could be found for logger "suds.xsd.sxbasic"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "suds\client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "suds\reader.py", line 152, in open
d = self.fn(url, self.options)
File "suds\wsdl.py", line 159, in __init__
self.build_schema()
File "suds\wsdl.py", line 220, in build_schema
self.schema = container.load(self.options)
File "suds\xsd\schema.py", line 93, in load
child.open_imports(options)
File "suds\xsd\schema.py", line 305, in open_imports
imported = imp.open(options)
File "suds\xsd\sxbasic.py", line 542, in open
result = self.download(options)
File "suds\xsd\sxbasic.py", line 567, in download
raise Exception(msg)
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
UPDATE:
I realized that suds connections always open in TCP increasing ports, and if it reaches the maximum TCP port (65535) then it starts opening again from the minimum TCP port available, so there's no problem with this.
The problem shows up when using suds ImportDoctor, because it has to open a previous connection to the location where the import should be retrieved, and for some reason, if the system reaches the maximum TCP port count, then suds somehow assumes that there's no TCP port available to open the connection for obtaining the import, and in consecuence it throws the exception:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
I repeat, this only happens if suds has to open this previous connection for obtaining the import. If ImportDoctor is not used, then suds has no problem if the TCP port count reaches its maximum, it just restarts at the minimum port available.
Does anyone has any clue on how to resolve this issue???. I'd really appreciate the help!!!.
I've figured out what the problem was. The schema that was missing from the WSDL I was trying to use with suds was:
http://www.w3.org/2001/XMLSchema
And the XSD file for this schema is at:
http://www.w3.org/2001/XMLSchema.xsd
So when I used suds ImportDoctor to add this schema import, sometimes the w3.org domain was denying my access (don't know why really) and that's why this error was rising:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
What did I do to solve this problem?. I just downloaded this schema to my machine and used suds ImportDoctor to retrieve this import locally.
And that was it!!!. Confusing bug!!!. But SOLVED.
This is in my application file head:
import os
import sys
from cgi import parse_qs, escape
import pymongo
from pymongo import MongoClient
I have the mongoDB 2.4 gear installed, and am trying to connect via
client = MongoClient('mongodb:$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/')
I get the errors:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/var/lib/openshift/531b77fd500446980900010d/python/virtenv/lib/python2.7/site-packages/pymongo/mongo_client.py", line 239, in __init__
res = uri_parser.parse_uri(entity, port)
File "/var/lib/openshift/531b77fd500446980900010d/python/virtenv/lib/python2.7/site-packages/pymongo/uri_parser.py", line 269, in parse_uri
nodes = split_hosts(hosts, default_port=default_port)
File "/var/lib/openshift/531b77fd500446980900010d/python/virtenv/lib/python2.7/site-packages/pymongo/uri_parser.py", line 209, in split_hosts
nodes.append(parse_host(entity, port))
File "/var/lib/openshift/531b77fd500446980900010d/python/virtenv/lib/python2.7/site-packages/pymongo/uri_parser.py", line 137, in parse_host
raise ConfigurationError("Port number must be an integer.")
pymongo.errors.ConfigurationError: Port number must be an integer.
looks like OPENSHIFT_MONGODB_DB_PORT isn't set
print OPENSHIFT_MONGODB_DB_PORT --> NameError: name 'OPENSHIFT_MONGODB_DB_PORT' is not defined
Same with OPENSHIFT_MONGODB_DB_HOST
What would I need to do, to set up a connection?
Update:
I was able to connect directly via client by hardcoding info from rockmongo
client = MongoClient('mongodb://admin:password#[ip addr]:[port]/')
but when I do
client = MongoClient('mongodb:admin:password#%s:%s/' % os.environ['OPENSHIFT_MONGODB_DB_HOST'], os.environ['OPENSHIFT_MONGODB_DB_PORT']))
I get
[error] (<type 'exceptions.KeyError'>, KeyError('OPENSHIFT_MONGODB_DB_HOST',), <traceback object at 0x7f7bc8367248>)
The OpenShift connection variables are defined as environment variables, they cannot be accessed as normal Python variables. So the print statement you gave does not work, the following should;
import os
print os.environ['OPENSHIFT_MONGODB_DB_PORT']
You should change your code to;
client = MongoClient('mongodb:%s:%s/' % (os.environ['OPENSHIFT_MONGODB_DB_HOST'], os.environ['OPENSHIFT_MONGODB_DB_PORT))
You can refer to an example here.
I'm using ubuntu 10.10, and after making the mistake of installing LAMP, (was unable to connect to a database with it), I came here and read how important it is to use apt-get install python-mysqldb instead. Deleted LAMP, re-installed using apt-get, and am now getting the same error when trying to run a basic server_version.py script.
The script is the server_version.py found here: http://www.kitebird.com/articles/pydbapi.html
My server_version.py script:
# server_version.py - retrieve and display database server version
import MySQLdb
# I have also tried setting host = 'localhost'
conn = MySQLdb.connect (host = "/opt/lampp/var/mysql/mysql.sock",
user = "root",
passwd = "myrealpass",
db = "testdb1")
cursor = conn.cursor ()
cursor.execute ("SELECT VERSION()")
row = cursor.fetchone ()
print "server version:", row[0]
cursor.close ()
conn.close ()
The error is:
Traceback (most recent call last):
File "server_version.py", line 10, in
db = "testdb1")
File "/usr/lib/pymodules/python2.6/MySQLdb/init.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 129, in init
from converters import conversions
File "/usr/lib/pymodules/python2.6/MySQLdb/converters.py", line 165, in
from decimal import Decimal
File "/usr/lib/python2.6/decimal.py", line 137, in
import copy as _copy
File "/home/browning/copy.py", line 4, in
ValueError: need more than 1 value to unpack
Just trying to get some basic experience using databases with python here, so I'm not set on MySQL if there is a better option. I have attempted to re-install mysqldb multiple times using apt-get and pip.
It looks like you have a file named copy.py that is being picked up instead of the Python standard library module copy. Rename or delete your file (and copy.pyc if it was created). Or run from a different directory.