We Keep Coding

Cant import mysql.connector in python

I've installed several types of this mysql-connector such as mysql-connector and mysql-connector-python and mysql-connector-rf.
But it keeps saying its not imported. And when im trying to install the mysql-connector-rf it comes with a big red error message.
This is the error code when im trying to run my test.py:
PS C:\Users\Oscar GP\OneDrive - Aalborg Universitet\GPRO\aau\gpro\mysql\opgaveark1> & "C:/Users/Oscar GP/AppData/Local/Programs/Python/Python38-32/python.exe" "c:/Users/Oscar GP/OneDrive -
Aalborg Universitet/GPRO/aau/gpro/mysql/opgaveark1/opgave1/test.py"
Traceback (most recent call last):
File "c:/Users/Oscar GP/OneDrive - Aalborg Universitet/GPRO/aau/gpro/mysql/opgaveark1/opgave1/test.py", line 1, in <module>
import mysql.connector
ModuleNotFoundError: No module named 'mysql'
PS C:\Users\Oscar GP\OneDrive - Aalborg Universitet\GPRO\aau\gpro\mysql\opgaveark1>
and the python file:
import mysql.connector
db = mysql.connector.connect (
host='localhost',
user='oscar',
pwd='password',
database='gpro1'
)
cursor = db.cursor (dictionary = True)
cursor.execute ("CREATE TABLE IF NOT EXISTS TestTable (id INTEGER)")
for i in range (0,100):
cursor.execute (f"INSERT INTO TestTable VALUE({i})")
db.commit ()

PyHive Connection Error - Could not start SASL

I'm trying to connect to Hive server with PyHive.
So far, I have this:
from pyhive import hive
import pandas as pd
# Create Hive connection
conn = hive.Connection(host="*********", port=10000, auth='NONE')
df = pd.read_sql("select max_temperature_f from `201402_weather_data` LIMIT 10", conn)
print(df.head())
In my hive-site.xml configuration I have this:
<property>
<name>hive.server2.authentication</name>
<value>NONE</value>
<final>false</final>
<source>programmatically</source>
<source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream#56f71edb</source>
</property>
and
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
<final>false</final>
<source>programmatically</source>
<source>org.apache.hadoop.hive.conf.LoopingByteArrayInputStream#56f71edb</source>
</property>
From what I read, these are the correct settings with which I could connect to the hive server (assuming that I want NONE for authentication).
Executing the script I get an error:
Traceback (most recent call last):
File "D:/****/hive_connector.py", line 8, in <module>
auth='NONE')
File "C:\Users\***\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyhive\hive.py", line 192, in __init__
self._transport.open()
File "C:\Users\***\AppData\Local\Programs\Python\Python37-32\lib\site-packages\thrift_sasl\__init__.py", line 85, in open
message=("Could not start SASL: %s" % self.sasl.getError()))
thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2'
I'm using windows, so I had to manually download and install SASL - sasl-0.2.1-cp37-cp37m-win32.whl
I'm using:
PyHive - 0.6.3,
sasl - 0.2.1,
thrift - 0.13.0,
thrift-sasl - 0.4.2,
thriftpy2 - 0.4.11 (not sure where that came from)
I've seen a lot of questions and I've tried several things but I'm not able to run the script successfully. Can you point me to the correct solution? Is it the sasl package that is causing the problems?

AWS Glue Python Shell package import

We create a python shell job which is connecting Redshift and fetching data, below program is working fine in my local system.
Below are the steps and programs.
Program:-
import sqlalchemy as sa
from sqlalchemy.orm import sessionmaker
#>>>>>>>> MAKE CHANGES HERE <<<<<<<<<<<<<
DATABASE = "#####"
USER = "#####"
PASSWORD = "#####"
HOST = "#####.redshift.amazonaws.com"
PORT = "5439"
SCHEMA = "test" #default is "public"
####### connection and session creation ##############
connection_string = "redshift+psycopg2://%s:%s#%s:%s/%s" % (USER,PASSWORD,HOST,str(PORT),DATABASE)
engine = sa.create_engine(connection_string)
session = sessionmaker()
session.configure(bind=engine)
s = session()
SetPath = "SET search_path TO %s" % SCHEMA
s.execute(SetPath)
###### All Set Session created using provided schema #######
################ write queries from here ######################
query = "SELECT * FROM test1 limit 2;"
rr = s.execute(query)
all_results = rr.fetchall()
def pretty(all_results):
for row in all_results :
print("row start >>>>>>>>>>>>>>>>>>>>")
for r in row :
print(" ----" , r)
print("row end >>>>>>>>>>>>>>>>>>>>>>")
pretty(all_results)
########## close session in the end ###############
s.close()
Steps:-
sudo pip install psycopg2
sudo pip install sqlalchemy
sudo pip install sqlalchemy-redshift
I have uploaded the files psycopg2-2.8.4-cp27-cp27m-win32.whl, Flask_SQLAlchemy-2.4.1-py2.py3-none-any.whl and sqlalchemy_redshift-0.7.5-py2.py3-none-any.whl in S3 (s3://####/lib/), and map the folder in Python library path in AWS Glue Job.
When I run the program below error is occurring.
Traceback (most recent call last):
File "/tmp/runscript.py", line 113, in <module>
download_and_install(args.extra_py_files)
File "/tmp/runscript.py", line 56, in download_and_install
download_from_s3(s3_file_path, local_file_path)
File "/tmp/runscript.py", line 81, in download_from_s3
s3.download_file(bucket_name, s3_key, new_file_path)
File "/usr/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 172, in download_file
extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 307, in download_file
future.result()
File "/usr/local/lib/python2.7/site-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/usr/local/lib/python2.7/site-packages/s3transfer/futures.py", line 265, in result
raise self._exception
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
PS:- The Glue Job Role has full access to S3.
Please suggest how to map those libraries with the program.

You can specify your own Python libraries packaged as an .egg or a .whl file under the "—extra-py-files" flag as shown in below example.
Command line example :
aws glue create-job --name python-redshift-test-cli --role role --command '{"Name" : "pythonshell", "ScriptLocation" : "s3://MyBucket/python/library/redshift_test.py"}'
--connections Connections=connection-name --default-arguments '{"--extra-py-files" : ["s3://MyBucket/python/library/redshift_module-0.1-py2.7.egg", "s3://MyBucket/python/library/redshift_module-0.1-py2.7-none-any.whl"]}'
Refernece : Create a glue job with extra python library

There is a simple way to import python dependencies using whl files, that can be find on Python site for particular module.
You can also add multiple wheel files from S3 using comma.
For eg
"s3://xxxxxxxxx/common/glue/glue_whl/fastparquet-0.4.1-cp37-cp37m-macosx_10_9_x86_64.whl,s3://xxxxxx/common/glue/glue_whl/packaging-20.4-py2.py3-none-any.whl,s3://xxxxxx/common/glue/glue_whl/s3fs-0.5.0-py3-none-any.whl"
enter image description here

mysql malformed packet error after running makemigrations django

The issue that we are getting is : we have run manage.py makemigrations and manage.py migrate successfully, after which we start getting the below error:
File "/usr/local/goibibo/python/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 110, in execute
return self.cursor.execute(query, args)
File "/usr/local/goibibo/python/lib/python3.5/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/usr/local/goibibo/python/lib/python3.5/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/usr/local/goibibo/python/lib/python3.5/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/usr/local/goibibo/python/lib/python3.5/site-packages/MySQLdb/cursors.py", line 411, in _query
rowcount = self._do_query(q)
File "/usr/local/goibibo/python/lib/python3.5/site-packages/MySQLdb/cursors.py", line 374, in _do_query
db.query(q)
File "/usr/local/goibibo/python/lib/python3.5/site-packages/MySQLdb/connections.py", line 292, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (2027, 'Malformed packet')
This issue does not come every time we run makemigrations.
It comes randomly, but once it comes it just sticks and we have no clue on why this is happening.
Can somebody assist us in fixing this and explain why this might be happening.
Packages Used in App:
adium-theme-ubuntu==0.3.4
appdirs==1.4.3
asn1crypto==0.22.0
backports-abc==0.5
blinker==1.4
boto3==1.4.4
botocore==1.5.35
certifi==2017.4.17
cffi==1.10.0
chardet==2.3.0
click==6.7
ConcurrentLogHandler==0.9.1
configobj==5.0.6
decorator==4.0.11
Django==1.10.4
djangorestframework==3.5.3
docutils==0.13.1
Flask==0.11.1
futures==3.1.1
gevent==1.2a1
greenlet==0.4.10
gyp==0.1
idna==2.5
itsdangerous==0.24
Jinja2==2.8
jmespath==0.9.2
jsonpatch==1.10
jsonpath==0.75
jsonpath-rw==1.4.0
jsonpointer==1.10
MarkupSafe==1.0
newrelic==2.82.0.62
oauthlib==1.0.3
packaging==16.8
Pillow==3.1.2
ply==3.10
prettytable==0.7.2
pyasn1==0.1.9
pycparser==2.17
pycurl==7.43.0
PyJWT==1.3.0
PyMySQL==0.7.9
PyOpenGL==3.0.2
pyparsing==2.2.0
Pyrex==0.9.8.5
pyserial==3.0.1
PySocks==1.6.5
python-dateutil==2.6.0
PyYAML==3.11
requests==2.14.2
s3transfer==0.1.10
singledispatch==3.4.0.3
six==1.10.0
SQLAlchemy==1.1.4
ssh-import-id==5.5
tornado==4.4.2
unity-lens-photos==1.0
urllib3==1.19.1
virtualenv==15.1.0
Werkzeug==0.12.1

Resolved in mysql 5.6.17 by SET read_rnd_buffer_size=256000 for the session. Did not have to change my.ini (cfg).

Same issue haunted my team for a while, and there's very little useful information about it on the web. We spent a lot of time troubleshooting the issue - and FINALLY found solution that solved it (at least - for our team)!
We found that Django sets "charset" option to "utf8" for database connections by default. During troubleshooting we used two separate database connection objects: one created for us by Django, and another - created manually using direct _mysql.connect() command. When we executed same exact query using both connection objects - the one created by Django resulted in "django.db.utils.OperationalError: (2027, 'Malformed packet')" (which is exactly what we were getting in our API), but the second connection (manual) - worked without any issues. Further comparison of the two connection objects (we actually had to use Python debugger "pdb" and set breakpoint within django.db.backends.mysql.base.py for that) - revealed that Django creates connection by passing "charset":"utf8", while manual _mysql connection - uses "latin1". As soon as we added "charset":"latin1" to DATABASES["default"]["OPTIONS"] within our settings.py - this error disappeared.
To summarize, solution (for us) was to explicitly set "charset":"latin1" within "OPTIONS" config section of DATABASES configuration - for every database alias. I can not say for sure that this will work for everyone who experiences this error - but it certainly works for us.

Accessing OrientDB from Python

I want to convert a >1mn record MySQL database into a graph database, because it is heavily linked network-type data. The free version of Neo4J had some restrictions I thought I might bump up against, so I've installed OrientDB (Community 2.2.0) (on Ubuntu Server 16.04) and got it working. Now I need to access it from Python (3.5.1+), so I'm trying pyorient (1.5.2). (I tried TinkerPop since I eventually want to use Gremlin, and couldn't get the gremlin console to talk to the OrientDB.)
The following simple Python code, to connect to one of the test graphs in OrientDB:
import pyorient
username="user"
password="password"
client = pyorient.OrientDB("localhost", 2424)
session_id = client.connect( username, password )
print("SessionID=",session_id)
db_name="GratefulDeadConcerts"
if client.db_exists( db_name, pyorient.STORAGE_TYPE_MEMORY ):
print("Database",db_name,"exists")
client.db_open( db_name, username, password )
else:
print("Database",db_name,"doesn't exist")
gives a weird error:
SessionID= 27
Database GratefulDeadConcerts exists
Traceback (most recent call last):
File "FirstTest.py", line 18, in <module>
client.db_open( db_name, username, password )
File "/home/tom/MyProgs/TestingPyOrient/env/lib/python3.5/site-packages/pyorient/orient.py", line 379, in db_open
.prepare((db_name, user, password, db_type, client_id)).send().fetch_response()
File "/home/tom/MyProgs/TestingPyOrient/env/lib/python3.5/site-packages/pyorient/messages/database.py", line 141, in fetch_response
info = OrientVersion(release)
File "/home/tom/MyProgs/TestingPyOrient/env/lib/python3.5/site-packages/pyorient/otypes.py", line 202, in __init__
self._parse_version(release)
File "/home/tom/MyProgs/TestingPyOrient/env/lib/python3.5/site-packages/pyorient/otypes.py", line 235, in _parse_version
self.build = int( self.build )
ValueError: invalid literal for int() with base 10: '0 (build develop#r79d281140b01c0bc3b566a46a64f1573cb359783; 2016'
Does anyone know what that is or how I can fix it? Should I really be using TinkerPop instead? If so I'll post a seperate question about my struggles with that.

I firstly got the error, but after upgrading Pyorient to last version 1.5.4 I get no errors.
$ python test.py
('SessionID=', 6)
('Database', 'GratefulDeadConcerts', 'exists')
$ python --version
Python 2.7.11