We Keep Coding

Why pymysql not insert record into table?

I am pretty new in python developing. I have a long python script what "clone" a database and add additional stored functions and procedures. Clone means copy only the schema of DB.These steps work fine.
My question is about pymysql insert exection:
I have to copy some table contents into the new DB. I don't get any sql error. If I debug or print the created INSERT INTO command is correct (I've tested it in an sql editor/handler). The insert execution is correct becuse the result contain the exact row number...but all rows are missing from destination table in dest.DB...
(Ofcourse DB_* variables have been definied!)
import pymysql
liveDbConn = pymysql.connect(DB_HOST, DB_USER, DB_PWD, LIVE_DB_NAME)
testDbConn = pymysql.connect(DB_HOST, DB_USER, DB_PWD, TEST_DB_NAME)
tablesForCopy = ['role', 'permission']
for table in tablesForCopy:
with liveDbConn.cursor() as liveCursor:
# Get name of columns
liveCursor.execute("DESCRIBE `%s`;" % (table))
columns = '';
for column in liveCursor.fetchall():
columns += '`' + column[0] + '`,'
columns = columns.strip(',')
# Get and convert values
values = ''
liveCursor.execute("SELECT * FROM `%s`;" % (table))
for result in liveCursor.fetchall():
data = []
for item in result:
if type(item)==type(None):
data.append('NULL')
elif type(item)==type('str'):
data.append("'"+item+"'")
elif type(item)==type(datetime.datetime.now()):
data.append("'"+str(item)+"'")
else: # for numeric values
data.append(str(item))
v = '(' + ', '.join(data) + ')'
values += v + ', '
values = values.strip(', ')
print("### table: %s" % (table))
testDbCursor = testDbConn.cursor()
testDbCursor.execute("INSERT INTO `" + TEST_DB_NAME + "`.`" + table + "` (" + columns + ") VALUES " + values + ";")
print("Result: {}".format(testDbCursor._result.message))
liveDbConn.close()
testDbConn.close()
Result is:
### table: role
Result: b"'Records: 16 Duplicates: 0 Warnings: 0"
### table: permission
Result: b'(Records: 222 Duplicates: 0 Warnings: 0'
What am I doing wrong? Thanks!

You have 2 main issues here:
You don't use conn.commit() (which would be either be liveDbConn.commit() or testDbConn.commit() here). Changes to the database will not be reflected without committing those changes. Note that all changes need committing but SELECT, for example, does not.
Your query is open to SQL Injection. This is a serious problem.
Table names cannot be parameterized, so there's not much we can do about that, but you'll want to parameterize your values. I've made multiple corrections to the code in relation to type checking as well as parameterization.
for table in tablesForCopy:
with liveDbConn.cursor() as liveCursor:
liveCursor.execute("SELECT * FROM `%s`;" % (table))
name_of_columns = [item[0] for item in liveCursor.description]
insert_list = []
for result in liveCursor.fetchall():
data = []
for item in result:
if item is None: # test identity against the None singleton
data.append('NULL')
elif isinstance(item, str): # Use isinstance to check type
data.append(item)
elif isinstance(item, datetime.datetime):
data.append(item.strftime('%Y-%m-%d %H:%M:%S'))
else: # for numeric values
data.append(str(item))
insert_list.append(data)
testDbCursor = testDbConn.cursor()
placeholders = ', '.join(['`%s`' for item in insert_list[0]])
testDbCursor.executemany("INSERT INTO `{}.{}` ({}) VALUES ({})".format(
TEST_DB_NAME,
table,
name_of_columns,
placeholders),
insert_list)
testDbConn.commit()
From this github thread, I notice that executemany does not work as expected in psycopg2; it instead sends each entry as a single query. You'll need to use execute_batch:
from psycopg2.extras import execute_batch
execute_batch(testDbCursor,
"INSERT INTO `{}.{}` ({}) VALUES ({})".format(TEST_DB_NAME,
table,
name_of_columns,
placeholders),
insert_list)
testDbConn.commit()

How to insert data into table using python pymsql
Find my solution below
import pymysql
import datetime
# Create a connection object
dbServerName = "127.0.0.1"
port = 8889
dbUser = "root"
dbPassword = ""
dbName = "blog_flask"
# charSet = "utf8mb4"
conn = pymysql.connect(host=dbServerName, user=dbUser, password=dbPassword,db=dbName, port= port)
try:
# Create a cursor object
cursor = conn.cursor()
# Insert rows into the MySQL Table
now = datetime.datetime.utcnow()
my_datetime = now.strftime('%Y-%m-%d %H:%M:%S')
cursor.execute('INSERT INTO posts (post_id, post_title, post_content, \
filename,post_time) VALUES (%s,%s,%s,%s,%s)',(5,'title2','description2','filename2',my_datetime))
conn.commit()
except Exception as e:
print("Exeception occured:{}".format(e))
finally:
conn.close()

How to store results from SQL query and use it?

I am trying to see whether the type is either a the letter "T" or between number 1-6 for the specific data entry found with name and password.
sql = 'SELECT type FROM table name WHERE name = "{}" AND password = "{}"'.format(username, password)
and then in psedocode i need something like:
if type =< 5:
int()
elif type = "T"
string()
I am using python 2.7

Here is a full script that will query the mysql DB, and use your above-mentioned logic to print the values. I've included the python code as well as the sample database code for this test case. Let me know if you have any questions.
Python
import pymysql
connection = pymysql.connect(user='username', passwd='password',
host='localhost',
database='database')
cursor = connection.cursor()
NAME = 'Person_A'
PASSWORD = 'Password_A'
query = ("SELECT * FROM My_TABLE WHERE NAME = '%(1)s' AND PASSWORD = '%(2)s';" % {"1" : NAME, "2" : PASSWORD})
cursor.execute(query)
for item in cursor:
type = item[0]
if type.isdigit():
if int(type) <6:
print('Type is a number less than 6')
else:
print('Type is a number but not less than 6')
else:
if type == 'T':
print('Type is a string == T')
else:
print('Type is a string but not the letter T')
MYSQL
CREATE TABLE MY_TABLE (TYPE VARCHAR(255), NAME VARCHAR(255), PASSWORD VARCHAR(255));
INSERT INTO MY_TABLE VALUES ('T','Person_A','Password_A'),('4','Person_A','Password_A'),('7','Person_B','Password_B'),('t','Person_C','Password_C');

python loop over sqlite select with .fetchmany() until no entries left

I need to retrieve results from my sqlite3 database 160 rows at a time, and repeat that until there are no rows left for my query, this is what I have:
conn = sqlite3.connect("C:\\Users\\%s\\AppData\\Roaming\\GridcoinResearch\\reports\\Rain.db" % user_account)
c = conn.cursor()
conn.text_factory = str
address = c.execute('select Address from NNDATA where NeuralMagnitude != 0 and NeuralMagnitude is not null and CPID in (select cpids from GRIDCOINTEAM)').fetchmany(160)
conn.text_factory = float
nn_mag = c.execute('select NeuralMagnitude from NNDATA where NeuralMagnitude != 0 and NeuralMagnitude is not null and CPID in (select cpids from GRIDCOINTEAM)').fetchmany(160)
conn.close()
while True:
if nn_mag == ():
sys.exit("Complete")
The reason for sys.exit is I have a bunch of other code to go between conn.close() and while True:, so when the last loop is done I can exit the program. Right now its doing the first pass then the cmd.exe is hanging.
EDIT: Just relaised I dont tell the loop to select the NEXT 160, oh dear!

The fetchmany attribute returns an empty list if there is no item lefts so you can just check the validation of its result. Also note that you should remove the limit from your query and the fetchall. Because the whole essence of using fetchmany is to fetch limited results from your cursor object.
chunk_size = 160
while True:
result = nn_mag.fetchmany(chunk_size)
if not result:
sys.exit("Complete")
else:
# do something with result

OK the full answer is:
conn = sqlite3.connect("C:\\Users\\%s\\AppData\\Roaming\\GridcoinResearch\\reports\\Rain.db" % user_account)
c = conn.cursor()
position = 00
while True:
conn.text_factory = str
address_db = c.execute('select Address from NNDATA where NeuralMagnitude != 0 and NeuralMagnitude is not null and CPID in (select cpids from GRIDCOINTEAM) limit {}, 160'.format(position)).fetchall()
conn.text_factory = float
rac_db = c.execute('select rac from GRIDCOINTEAM where rac != 0 and rac is not null limit {}, 160'.format(position)).fetchall()
if not address_db:
conn.close()
sys.exit("Complete")
else:
position += 160

ValueError: invalid literal for int() with base 10: '' date field

I searched for this error and was able to find some information. I am importing Excel into postgreSQL and below is my script. I know the error is regarding how Python is handling a blank field. The script None if not has functioned fine for other tables I have imported and it also works fine for the Traffic field for this table but for some odd reason throws out an error for Week_Ending date field. I am new to Python and StackOverflow so if you decide to down vote my question please tell me why.
import psycopg2
import xlrd
book = xlrd.open_workbook("consolidateddata.xlsx")
sheet = book.sheet_by_name("Builder_Traffic")
database = psycopg2.connect (database = "", user="", password="", host="", port="")
cursor = database.cursor()
delete = """Drop table if exists "Python".buildertraffic"""
print (delete)
mydata = cursor.execute(delete)
cursor.execute('''CREATE TABLE "Python".buildertraffic
(Builder_Name varchar(55),
Traffic integer,
Week_Ending date,
Project_ID integer
);''')
print "Table created successfully"
query = """INSERT INTO "Python".buildertraffic (Builder_Name, Traffic, Week_Ending, Project_ID)
VALUES (%s, %s, %s, %s)"""
for r in range(1, sheet.nrows):
Builder_Name = sheet.cell(r,0).value
Traffic = None if not sheet.cell(r,1).value else sheet.cell(r,1).value
Week_Ending = None if not sheet.cell(r,2).value else xlrd.xldate.xldate_as_datetime(sheet.cell(r,2).value,book.datemode)
Project_ID = sheet.cell(r,3).value
# Assign values from each row
values = (Builder_Name, Traffic, Week_Ending, Project_ID)
# Execute sql query
cursor.execute(query, values)
cursor.close()
database.commit()
database.close()
print ""
print "All Done! Bye, for now."
print ""
columns = str(sheet.ncols)
rows = str(sheet.nrows)
print "I just imported Excel into postgreSQL"
When run, this fails with the stack frame
Week_Ending = None if not sheet.cell(r,2).value else xlrd.xldate.xldate_as_datetime(sheet.cell(r,2).value,book.datemode)
File "C:\Python27\lib\site-packages\xlrd\xldate.py", line 123, in xldate_as_datetime
days = int(xldate)
ValueError: invalid literal for int() with base 10: ''

Python Code reading from port and storing in db gives error: #1062, Duplicate entry 1 for key PRIMARY

Using pyserial, I am getting data from a sensor plugged into the USB port. I am trying to store that data, using MYSQLdb, into a database.
cur = db.cursor()
cur.execute("TRUNCATE TABLE randomdata;")
if ser.isOpen():
try:
i = 1
while 1:
num = str(1)
readserial = ser.readline()
print readserial
query="INSERT INTO randomdata(id,randomString)VALUES("+num+",'"+readserial+"');"
cur.execute(query)
db.commit()
i+=1
time.sleep(2)
if (i >= 50):
break
ser.close()
except Exception, e1:
print "error communicating...: " + str(e1)
else:
print "cannot open serial port "
It will store one value in the database before showing the error message: "error communicating...: (1062, "Duplicate entry '1' for key 'PRIMARY'")"
Any help would be greatly appreciated.

You cannot insert twice a value with the same id.
Here your id is "1" each time.
Depending on what you want to do you will need to use UPDATE instead of INSERT if you want to keep always the same id (and thus keep only one value). Something like that (untested) :
"UPDATE randomdata SET randomString = " + readserial +" WHERE id = " + num
Or you want to store all your value and insert new entry each time, then you need to generate a new id each time (by using your i instead of 1 in the while loop).

num = str(1)
which means num is always 1.
query="INSERT INTO randomdata(id,randomString)VALUES("+num+",'"+readserial+"');"
You insert 1 which is num as the ID at each time which is your primary key. Then, you got the error.
Why don't you use i instead like this :
num = str(i)
Since you increase i, the ID will not be duplicated.