Python.Dataframe to MySql: MySQL server has gone away - python

I am trying to write several dataframes in mysql. I use mysql.connector for the connection and sqlalchemy to create an engine.
Most dataframes are written correctly to the database. Unfortunately the application stops with the following error:
sqlalchemy.exc.DatabaseError: (mysql.connector.errors.DatabaseError) 2006 (HY000): MySQL server has gone away
...
During handling of the above exception, another exception occurred:
...
_mysql_connector.MySQLInterfaceError: MySQL server has gone away
Since the dataframe that breaks the connection is also the largest one (pickle file: 198MB) I assumed that it is due to the MySql-Server setting max_allowed_packet or timeout. (As described here)
So I have extended the config file within my SQLConnector (see below) by 'connect_timeout': 900.
Unfortunately without result.
I have also read that you should adjust the my.cnf of the MySql server. Unfortunately I did not know exactly where to find them. After a long search I found the following order. C:\Program Files\MySQL\MySQL Server 8.0\etc with the file mysqlrouter.conf.sample.
Here I have downloaded a my.cnf file and put it into the folder. Unfortunately without result, because I did not know how to set up MySql so that the server uses this file.
Does anyone know how I can fix this error? Or how I can configure the MySql Server preference settings max_allowed_packet or timeout?
Main:
mysqlConnector = MySQLConnector()
dbConnection = mysqlConnector.init_engine()
for df_tuple in df_all_tuple:
df_name = df_tuple[0]
df = pd.DataFrame(df_tuple[1])
df.to_sql(con=mydb, name=df_name, if_exists='fail', chunksize=20000)
SQLConnector:
import mysql.connector
from mysql.connector import errorcode
import sqlalchemy as db
config = {
'user': 'root',
'password': '',
'host': '127.0.0.1',
'database': 'test',
'raise_on_warnings': True,
'use_pure': False,
'autocommit': True,
'connect_timeout': 900
}
class MySQLConnector:
def __init__(self):
connectTest = self.connect()
print("Connection to database: " + str(connectTest))
def init_engine(self):
try:
connect_string = 'mysql+mysqlconnector://{}:{}#{}/{}?charset=utf8mb4'.format(config.get("user"),
config.get("password"),
config.get("host"),
config.get("database"),
pool_pre_ping=True)
print("Engine: " + connect_string)
sqlengine = db.create_engine(connect_string)
except:
print("SQLConnector: Sqlalchemy error. Can't create engine....")
else:
dbConnection = sqlengine.connect()
return dbConnection
def connect(self):
try:
con = mysql.connector.connect(**config)
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with your user name or password")
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exist")
if self.createDB() is True:
return self.connect()
elif err.errno == errorcode.CR_SERVER_GONE_ERROR:
print("The client couldn't send a question to the server.")
print("err: " + err)
elif err.errno == errorcode.CR_SERVER_LOST:
print(
"The client didn't get an error when writing to the server, but it didn't get a full answer (or any answer) to the question.")
print("err: " + err)
else:
print(err)
return None
else:
if con.is_connected():
db_Info = con.get_server_info()
print("Connected to MySQL Server version ", db_Info)
sqlcursor = con.cursor()
return con
def createDB(self):
try:
mydb = mysql.connector.connect(
host=config.get("host"),
user=config.get("user"),
passwd=config.get("password")
)
except mysql.connector.Error as err:
print(err)
else:
sqlcursor = mydb.cursor()
try:
sqlcursor.execute('CREATE DATABASE {}'.format(config.get("database")))
except mysql.connector.Error as err:
print(err)
return False
else:
print("Database created!")
return True

Related

SQL Database file not showing up in folder?

I am trying to create a database file using the following code:
def dbconnect():
try:
sqliteConnection = sqlite3.connect('SQLite_Python.db')
cursor = sqliteConnection.cursor()
print("Database created and Successfully Connected to SQLite")
sqlite_select_Query = "select sqlite_version();"
cursor.execute(sqlite_select_Query)
record = cursor.fetchall()
print("SQLite Database Version is: ", record)
cursor.close()
except sqlite3.Error as error:
print("Error while connecting to sqlite", error)
finally:
if sqliteConnection:
sqliteConnection.close()
print("The SQLite connection is closed")
conn = dbconnect()
conn.close()
yet when I run the code, although there are no file errors, it doesnt print anything or create a SQL file in the folder of the python code.
I can't seem to figure out what is going wrong.

Connect to MongoDB Database using Python

I am new in using python and I want to try first connecting to mongodb database using python
This is my code but is has error
import pymongo
DATABASE_NAME = "testdb"
DATABASE_HOST = "mongodb://10.10.100.100:22222"
DATABASE_USERNAME = "testuserid"
DATABASE_PASSWORD = "testpass"
try:
myclient = pymongo.MongoClient(DATABASE_HOST)
myclient.test.authenticate( DATABASE_USERNAME , DATABASE_PASSWORD )
mydb = myclient[DATABASE_NAME]
print("[+] Database connected!")
except Exception as e:
print("[+] Database connection error!")
raise e
10.10.100.100 is just my sample address and also 22222 is sample port
DATABASE_URI = f"mongodb://{DATABASE_USERNAME}:{DATABASE_PASSWORD}#{DATABASE_HOST}"
myclient = pymongo.MongoClient(DATABASE_URI)

MySQL database connection through Pycharm on MacBook

Im currently attempting to follow a youtube tutorial on how to connect a database through Pycharm however it's not connecting and I'm not sure where the problem is or how I am able to solve it. the code is:
import mysql.connector
from mysql.connector import errorcode
try:
cnn = mysql.connector.connect(
user='root',
password='root',
host='localhost',
database='name')
print("it works")
except mysql.connector.Error as e:
if e.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("something is wrong with username and password")
elif e.errno == errorcode.ER_BAD_DB_ERROR:
print("database does not exist")
else:
print("E")
cursor = cnn.cursor()
addName = "INSERT INTO name (fName, lName) values (%s, %s)"
fName = "Rae"
lName = "Smith"
empName = (fName, lName)
cursor.execute(addName, empName)
cnn.commit()
cursor.close()
cnn.close()
Once I run it I get the error:
Traceback (most recent call last):
something is wrong with username and password
line 19, in <module>
cursor = cnn.cursor()
NameError: name 'cnn' is not defined
I've seen the user on the tutorial receiving the same warning however when he runs his code there is no error like the one I receive. How I may be able to solve this?

How to trigger 'return' when specific output given

I'm developing a authentication script in Python. I need PyMySQL to check if user exists, how can I do that?
I read the documention but didn't find answer to my question.
def check():
connection = pymysql.connect(host='localhost',
user='user',
password='passwd',
db='db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
userexist = # PyMySQL check code here
if userexist=='True':
return 'good'
It's not clear whether you want to check for the pymysql user or a user from the database. This code will address both instances.
pymysql will raise an exception: pymysql.err.OperationalError if the user does not exist. You therefore need a try, except clause like this:
def check():
try:
connection = pymysql.connect(host='localhost',
user='user',
password='passwd',
db='db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
except pymysql.err.OperationalError:
print('User with entered connection credentials does not exists')
return False
try:
with connection.cursor() as cursor:
cursor.execute("select...") # Fill with your select statement
result = cursor.fetchone()
if result is None:
connection.close()
return False
finally:
connection.close()
return 'good'

Python to MySQL Database: What exception handling

for a project at uni i need to insert different kinds of variables into a MySql Database. Connecting and Inserting the data so far works fine. I don't know how to handle potential errors though. Which potential mistakes and exceptions do i need to catch and take care of ?
In my code i used a main method to just test the program. In the final version just the connection and the SQL queries are copied into the main script. I am open to use either the MySQL or the mysql.connector. Also: Do i need to put the query into a try block aswell ? Here is my code so far:
import mysql.connector
import time
from mysql.connector import errorcode
try:
con = mysql.connector.connect(
user= 'root',
password= '',
host='localhost',
database= 'testdb')
print("Connected.")
cursor = con.cursor()
except mysql.connector.Error as e:
if e.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Passwort // Username")
elif e.errno == errorcode.ER_BAD_DB_ERROR:
print("DataBase does not exist")
else:
print(e)
def insert_temp(Temperatur_ID, Zeitpunkt, Wert, Thermometer_ID):
query = "INSERT INTO Temperatur (Temperatur_ID, Zeitpunkt, Wert, Thermometer_ID)" \
"VALUES(%s, %s, %s, %s)"
args= (Temperatur_ID, Zeitpunkt, Wert, Thermometer_ID)
cursor.execute(query, args)
con.commit()
def main():
# just test values so far
value = 18.5;
insert_temp(' ', time.strftime('%Y-%m-%d %H:%M:%S'), value, '1');
cursor.close()
con.close()
if __name__ == '__main__':
main()
please note that i have very little experience in python programming

Categories