We Keep Coding

I can't get Python to work with mySQL

Okay, so I'll preface with, 'I am new to Python'. I am using PyCharm to run this programme I am building to dump to a Database (locally hosted mySQL).
Unfortunately I am running into problems getting data into the database with Python, but I am able to insert data in SQL running through the phpMyAdmin web gui. I must be missing something obvious. I am using 'mysql.connector' addon for PyCharm. Here is the code relevant.
So, at the start I import the module? like so;
import mysql.connector
And this code is referenced inside a couple while loops;
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
c = cnx.cursor()
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
cnx.commit()
Please dont eat me alive either for the way I write, I am new :)
Other important factors;
I was able to get it to enter data a couple times (but it came through to SQL as a NULL value. My guess is I was passing the wrong format of number? or something to do with the db Collation?)
I have also tried this with a number of different addons for PyCharm (All of them seem to be implemented in similar ways).
I have tried it with and without quote marks on the variable names being inserted, I have tried so much, I am at my wits end.
As for the error codes, this is what it looks like currently;
C:\Users\logge\PycharmProjects\untitled3\venv\Scripts\python.exe C:/Users/logge/PycharmProjects/untitled3/main.py
Traceback (most recent call last):
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 111, in <module>
weather(1)
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 101, in weather
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\cursor.py", line 561, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 525, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 427, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir' at line 1
Process finished with exit code 1
The 'line 101' refered to in the error codes is talking about the line starting with 'c.execute... INSERT' in the second section of code I posted.
Here is the code I have also tried, just to see if I could get any results (I still get the same errors though):
from __future__ import print_function
from datetime import date, datetime, timedelta
import mysql.connector
dtime1 = 1
tmp1 = 5
apptmp1 = 2
dewpoint1 = 64
relhum1 = 3
delta_t1 = 4
wind_dir1 = 5
wind_spd_kmh1 = 6
wind_gust_kmh1 = 7
wind_spd_kts1 = 8
wind_gust_kts1 = 9
press_qnh1 = 10
press_msl1 = 11
rainsince9am1 =12
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
cursor = cnx.cursor()
tomorrow = datetime.now().date() + timedelta(days=1)
add_employee = ("INSERT INTO stations "
"(dtime, tmp, apptmp, dewpoint, relhum, delta_t, wind_dir, wind_spd_kmh, wind_gust_kmh, wind_spd_kts, wind_gust_kts, press_qnh, press_msl, rainsince9am) "
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,)")
data_employee = (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)
# Insert new employee
cursor.execute(add_employee, data_employee)
emp_no = cursor.lastrowid
# Make sure data is committed to the database
cnx.commit()
cursor.close()
cnx.close()
Any help or suggestions would be greatly appreciated. I am happy to post the entire code if required also. I just didn't want to clutter this whole thing up.

I found your error. You have referenced the input data to be strings (%s). But you are putting in integers (numbers).
If you want to put the numbers as strings, in the first chunk of code (yours), you need to put in str(variable) instead of just variable. In the second chunk of code (copied from MySQL), you can declare the variables as variable = "0000" with the quotes.
If you want to work with numbers in your code, change the table structure to be INTEGER or DOUBLE instead of VARCHAR or CHAR.
I would assume that the table is present and all fields are present as there is no error connecting to the database and finding the table.

SQLAlchemy - simple select statement

Background:
I am very new to SQLAlchemy and it seems to be fairly confusing as to how I should be selecting things.
I have a table in my mysql database which is called genes, where I have gene_id, gene_description, and gene_symbol
What I want to do:
All I want to do is a simple select query:
Select * from Genes
But I seem to be confused as to how we shall achieve this
Here is what I have done:
import sqlalchemy
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.automap import automap_base
import csv
import pandas as pd
engine = sqlalchemy.create_engine('mysql://root:toor#localhost') # connect to server
metadata = sqlalchemy.MetaData(bind=engine)
engine.execute("USE TestDB")
genes = sqlalchemy.table('Genes')
s = sqlalchemy.select([genes])
engine.execute(s)
The problem:
ProgrammingError: (_mysql_exceptions.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'FROM `Genes`' at line 2") [SQL: u'SELECT \nFROM `Genes`']
Also is there some type of "intellisense" where I can just do something like gene_table = engine.Gene. If I am not mistake there is a way to do this with mapping but it didn't work for me
EDIT:
This may help:
How to automatically reflect database to sqlalchemy declarative?

So we can use reflection and do not have to create classes explicitly, but if we want speed we can create them using something like sqlautocode as stated here:
Reverse engineer SQLAlchemy declarative class definition from existing MySQL database?
Also there is an issue with mysql databases where it will give an error that looks like the following: (taken from bitbucket: https://bitbucket.org/zzzeek/sqlalchemy/issues/1909/reflection-issue-with-mysql-url-with-no)
SNIP...
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/schema.py", line 1927, in __init__
self.reflect()
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/schema.py", line 2037, in reflect
connection=conn))
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/engine/base.py", line 1852, in table_names
return self.dialect.get_table_names(conn, schema)
File "<string>", line 1, in <lambda>
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/engine/reflection.py", line 32, in cache
return fn(self, con, *args, **kw)
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/dialects/mysql/base.py", line 1791, in get_table_names
self.identifier_preparer.quote_identifier(current_schema))
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/sql/compiler.py", line 1517, in quote_identifier
return self.initial_quote + self._escape_identifier(value) + self.final_quote
File "/opt/buildout-eggs/SQLAlchemy-0.6.4-py2.5.egg/sqlalchemy/dialects/mysql/mysqldb.py", line 77, in _escape_identifier
value = value.replace(self.escape_quote, self.escape_to_quote)
AttributeError: 'NoneType' object has no attribute 'replace'
This is resolved by adding a database name (the one you are using) as follows:
engine = create_engine('mysql+mysqldb://USER_NAME:PASSWORD#127.0.0.1/DATABASE_NAME', pool_recycle=3600) # connect to server
I used this to connect correctly:
http://docs.sqlalchemy.org/en/latest/orm/extensions/automap.html
and this:
http://docs.sqlalchemy.org/en/latest/core/engines.html
This also may help:
How to automatically reflect database to sqlalchemy declarative?
My code finally looks like this:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
Base = automap_base()
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine('mysql+mysqldb://root:toor#127.0.0.1/TestDB', pool_recycle=3600) # connect to server
# reflect the tables
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by default
# matching that of the table name.
Genes = Base.classes.Genes
Address = Base.classes.address
#Start Session
session = Session(engine)
#add a row:
session.add(Genes(Gene_Id=1,Gene_Symbol = "GENE_SYMBOL", Gene_Description="GENE_DESCRIPTION"))
session.commit()
q = session.query(Genes).all()
for gene in q:
print "This is the Gene ID {},\n This is the Gene Desc {},\n this is the Gene symbol {}.".format(gene.Gene_Id,gene.Gene_Description, gene.Gene_Symbol )

Is SQLAlchemy still recommended if only used for raw sql query?

Using Flask, I'm curious to know if SQLAlchemy is still the best way to go for querying my database with raw SQL (direct SELECT x FROM table WHERE ...) instead of using the ORM or if there is an simpler yet powerful alternative ?
Thank for your reply.

I use SQLAlchemy for direct queries all the time.
Primary advantage: it gives you the best protection against SQL injection attacks. SQLAlchemy does the Right Thing whatever parameters you throw at it.
I find it works wonders for adjusting the generated SQL based on conditions as well. Displaying a result set with multiple filter controls above it? Just build your query in a set of if/elif/else constructs and you know your SQL will be golden still.
Here is an excerpt from some live code (older SA version, so syntax could differ a little):
# Pull start and end dates from form
# ...
# Build a constraint if `start` and / or `end` have been set.
created = None
if start and end:
created = sa.sql.between(msg.c.create_time_stamp,
start.replace(hour=0, minute=0, second=0),
end.replace(hour=23, minute=59, second=59))
elif start:
created = (msg.c.create_time_stamp >=
start.replace(hour=0, minute=0, second=0))
elif end:
created = (msg.c.create_time_stamp <=
end.replace(hour=23, minute=59, second=59))
# More complex `from_` object built here, elided for example
# [...]
# Final query build
query = sa.select([unit.c.eli_uid], from_obj=[from_])
query = query.column(count(msg.c.id).label('sent'))
query = query.where(current_store)
if created:
query = query.where(created)
The code where this comes from is a lot more complex, but I wanted to highlight the date range code here. If I had to build the SQL using string formatting, I'd probably have introduced a SQL injection hole somewhere as it is much easier to forget to quote values.

After I worked on a small project of mine, I decided to try to just use MySQLDB, without SQL Alchemy.
It works fine and it's quite easy to use, here's an example (I created a small class that handles all the work to the database)
import MySQLdb
from MySQLdb.cursors import DictCursor
class DatabaseBridge():
def __init__(self, *args, **kwargs):
kwargs['cursorclass'] = DictCursor
self.cnx = MySQLdb.connect (**kwargs)
self.cnx.autocommit(True)
self.cursor = self.cnx.cursor()
def query_all(self, query, *args):
self.cursor.execute(query, *args)
return self.cursor.fetchall()
def find_unique(self, query, *args):
rows = self.query_all(query, *args);
if len(rows) == 1:
return rows[0]
return None
def execute(self, query, params):
self.cursor.execute(query, params)
return self.cursor.rowcount
def get_last_id(self):
return self.cnx.insert_id()
def close(self):
self.cursor.close()
self.cnx.close()
database = DatabaseBridge(**{
'user': 'user',
'passwd': 'password',
'db': 'my_db'
})
rows = database.query_all("SELECT id, name, email FROM users WHERE is_active = %s AND project = %s", (1, "My First Project"))
(It's a dumb example).
It works like a charm BUT you have to take these into consideration :
Multithreading is not supported ! It's ok if you don't work with multiprocessing from Python.
You won't have all the advantages of SQLAlchemy (Database to Class (model) wrapper, Query generation (select, where, order_by, etc)). This is the key point on how you want to work with your database.
But on the other hand, and like SQLAlchemy, there is protections agains't SQL injection attacks :
A basic query would be like this :
cursor.execute("SELECT * FROM users WHERE data = %s" % "Some value") # THIS IS DANGEROUS
But you should do :
cursor.execute("SELECT * FROM users WHERE data = %s", ("Some value",)) # This is secure!
Saw the difference ? Read again ;)
The difference is that I replaced %, by , : We pass the arguments as ... arguments to the execute, and these are escaped. When using %, arguments aren't escaped, enabling SQL Injection attacks!
The final word here is that it depends on your usage and what you plan to do with your project. For me, SQLAlchemy was on overkill (it's a basic shell script !), so MysqlDB was perfect.

Best way to iterate through all rows in a DB-table

I often write little Python scripts to iterate through all rows of a DB-table.
For example sending all to all subscribers a email.
I do it like this
conn = MySQLdb.connect(host = hst, user = usr, passwd = pw, db = db)
cursor = conn.cursor()
subscribers = cursor.execute("SELECT * FROM tbl_subscriber;")
for subscriber in subscribers:
...
conn.close()
I wonder if there is a better way to do this cause it is possible that my code loads thousands of rows into the memory.
I thought about that it could be done better with LIMIT.
Maybe something like that:
"SELECT * FROM tbl_subscriber LIMIT %d,%d;" % (actualLimit,steps)
Whats the best way to do it?
How would you do it?

unless you have BLOBs in there, thousands of rows shouldn't be a problem. Do you know that it is?
Also, why bring shame on yourself and your entire family by doing something like
"SELECT * FROM tbl_subscriber LIMIT %d,%d;" % (actualLimit,steps)
when the cursor will make the substitution for you in a manner that avoids SQL injection?
c.execute("SELECT * FROM tbl_subscriber LIMIT %i,%i;", (actualLimit,steps))

You don't have to modify the query, you can use the fetchmany method of cursors. Here is how I do it :
def fetchsome(cursor, some=1000):
fetch = cursor.fetchmany
while True:
rows = fetch(some)
if not rows: break
for row in rows:
yield row
This way you can "SELECT * FROM tbl_subscriber;" but you will only fetch some at a time.

Most MySQL connectors based on libmysqlclient will buffer all the results in client memory by default for performance reasons (with the assumption you won't be reading large resultsets).
When you do need to read a large result in MySQLdb you can use a SSCursor to avoid buffering entire large resultsets.
http://mysql-python.sourceforge.net/MySQLdb.html#using-and-extending
SSCursor -
A "server-side" cursor. Like Cursor
but uses CursorUseResultMixIn. Use
only if you are dealing with
potentially large result sets.
This does introduce complications that you must be careful of. If you don't read all the results from the cursor, a second query will raise an ProgrammingError:
>>> import MySQLdb
>>> import MySQLdb.cursors
>>> conn = MySQLdb.connect(read_default_file='~/.my.cnf')
>>> curs = conn.cursor(MySQLdb.cursors.SSCursor)
>>> curs.execute('SELECT * FROM big_table')
18446744073709551615L
>>> curs.fetchone()
(1L, '2c57b425f0de896fcf5b2e2f28c93f66')
>>> curs.execute('SELECT NOW()')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
This means you have to always read everything from the cursor (and potentially multiple resultsets) before issuing another - MySQLdb won't do this for you.

First of all maybe you don't need Select * from...
maybe it's enough for you just to get some stuff like: "SELECT email from..."
that would decrease the amount of memory usage anyway:)

Do you have actual memory problems? When iterating over a cursor, results are fetched one at a time (your DB-API implementation might decide to prefetch results, but then it might offer a function to set the number of prefetched results).

RS_1st = win32com.client.Dispatch(r’ADODB.Recordset’)
RS_1st.Open(“SELECT colA_, colB_, colC_ FROM tbl_subscriber “, Conn, CursorType = 3, LockType = 1)
for colA_, colB_, colC_ in numpy.transpose(RS_1st.GetRows()).tolist():
OR
for arrCol_ in numpy.transpose(RS_1st.GetRows()).tolist():
OR
cursor.execute("SELECT * FROM tbl_subscriber;")
for subscriber in cursor.fetchall():

django + south + python: strange behavior when using a text string received as a parameter in a function

this is my first question.
I'm trying to execute a SQL query in django (south migration):
from django.db import connection
# ...
class Migration(SchemaMigration):
# ...
def transform_id_to_pk(self, table):
try:
db.delete_primary_key(table)
except:
pass
finally:
cursor = connection.cursor()
# This does not work
cursor.execute('SELECT MAX("id") FROM "%s"', [table])
# I don't know if this works.
try:
minvalue = cursor.fetchone()[0]
except:
minvalue = 1
seq_name = table + '_id_seq'
db.execute('CREATE SEQUENCE "%s" START WITH %s OWNED BY "%s"."id"', [seq_name, minvalue, table])
db.execute('ALTER TABLE "%s" ALTER COLUMN id SET DEFAULT nextval("%s")', [table, seq_name + '::regclass'])
db.create_primary_key(table, ['id'])
# ...
I use this function like this:
self.transform_id_to_pk('my_table_name')
So it should:
Find the biggest existent ID or 0 (it crashes)
Create a sequence name
Create the sequence
Update the ID field to use sequence
Update the ID as PK
But it crashes and the error says:
File "../apps/accounting/migrations/0003_setup_tables.py", line 45, in forwards
self.delegation_table_setup(orm)
File "../apps/accounting/migrations/0003_setup_tables.py", line 478, in delegation_table_setup
self.transform_id_to_pk('accounting_delegation')
File "../apps/accounting/migrations/0003_setup_tables.py", line 20, in transform_id_to_pk
cursor.execute(u'SELECT MAX("id") FROM "%s"', [table.encode('utf-8')])
File "/Library/Python/2.6/site-packages/django/db/backends/util.py", line 19, in execute
return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "E'accounting_delegation'" does not exist
LINE 1: SELECT MAX("id") FROM "E'accounting_delegation'"
^
I have shortened the file paths for convenience.
What does that "E'accounting_delegation'" mean? How could I get rid of it?
Thank you!
Carlos.

The problem is that you're using DB-API parameterization for things that are not SQL data. When you do something like:
cursor.execute('INSERT INTO table_foo VALUES (%s, %s)', (col1, col2))
the DB-API module (django's frontend for whatever database you are using, in this case) will know to escape the contents of 'col1' and 'col2' appropriately, and replace the %s's with them. Note that there are no quotes around the %s's. But that only works for SQL data, not for SQL metadata, such as table names and sequence names, because they need to be quoted differently (or not at all.) When you do
cursor.execute('INSERT INTO "%s" VALUES (%s, %s)', (tablename, col1, col2))
the tablename gets quoted as if you mean it to be string data to insert, and you end up with, for example, "'table_foo'". You need to separate your SQL metadata, which is part of the query, and your SQL data, which is not, like so:
sql = 'INSERT INTO TABLE "%s" VALUES (%%s, %%s)' % (tablename,)
cursor.execute(sql, (col1, col2))
Note that because the django DB-API frontend's paramstyle is 'pyformat' (it uses %s for placeholders) you need to escape those when you do the string formatting to create the SQL you want to execute. And note that this isn't secure against SQL injection attacks when you take the tablename from an insecure source and don't validate it.