We Keep Coding

SELECT in a while loop in python with mysql

I am trying to find the latest entry in a MySQL database by using a query with SELECT MAX(id). I already get the latest id, so I know the query works, but now I want to use it in a while loop so I keep getting the latest entry with each iteration.
This is what I have so far:
import pymysql
con = pymysql.connect(host='.....', user='.....',
password='.....', database='.....')
cur = con.cursor()
while True:
query = "SELECT MAX(id) FROM reports"
cur.execute(query)
data = cur.fetchall()
last = (data[0][0])
print(last)
The problem is that I keep getting the same result after updating the database. For instance, right now I have 45 entries, so my script prints '45' in a while loop. But after I add another row to the table it keeps printing '45' instead of the '46' I would expect. When I stop the script and run it again, it will print '46' and keep printing this even after I add another row.
I have only started working with MySQL about two weeks ago, so I don't have all that much knowledge about it. I feel like I'm missing something really small here. What should I do? Any help would be greatly appreciated.

I had this same problem, and just wanted to make it clear for anyone else searching for the solution.
Setting autocommit to True solved my issue and didn't require calling a commit after each query.
import pymysql
con = pymysql.connect(host='.....', user='.....',
password='.....', database='.....')
con.autocommit = True
cur = con.cursor()
while True:
query = "SELECT MAX(id) FROM reports"
cur.execute(query)
data = cur.fetchall()
last = (data[0][0])
print(last)
Here is a link to the documentation

Postgresql error statement is too large

I developed a script on python and sqlalchemy to get and update the last activity of my active users.
But the users are increasing a lot, now i´m getting the following error
psycopg2.ProgrammingError: Statement is too large. Statement Size: 16840277 bytes. Maximum Allowed: 16777216 bytes
I was thinking if I update the file postgres.conf it will work, so with the help of pgtune I updated the file, but it does not work, so I updated my kernel on /etc/syslog.conf, with the following parameters
kern.sysv.shmmax=4194304
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=1024
and again it does not work.
After that I divide my query into slices to reduce the size but I got the same error.
How can know what parameter I need to update, to increase the size of my statement?
Workflow
query = "SELECT id FROM {}.{} WHERE status=TRUE".format(schema, customer_table)
ids = ["{}".format(i)for i in pd.read_sql(query, insert_uri).id.tolist()]
read_query = """
SELECT id,
MAX(CONVERT_TIMEZONE('America/Mexico_City', last_activity)) lastactivity
FROM activity WHERE
DATE_TRUNC('d', CONVERT_TIMEZONE('America/Mexico_City', last_activity)) =
DATE_TRUNC('d', CONVERT_TIMEZONE('America/Mexico_City', CURRENT_DATE))-{} and
id in ({})
GROUP BY id
""".format(day, ",".join(ids))
last_activity = pd.read_sql(read_query, read_engine, parse_dates=True)

If you are only fetching the IDs from the database and not filtering them by any other way, there is no need to fetch them at all, you can just insert the SQL statement as a subquery into the second:
SELECT id,
MAX(CONVERT_TIMEZONE('America/Mexico_City', last_activity)) lastactivity
FROM activity WHERE
DATE_TRUNC('d', CONVERT_TIMEZONE('America/Mexico_City', last_activity)) =
DATE_TRUNC('d', CONVERT_TIMEZONE('America/Mexico_City', CURRENT_DATE))-%s and
id in (
SELECT id FROM customerschema.customer WHERE status=TRUE
)
GROUP BY id
Also, as Antti Haapala said, don't use string formatting for SQL parameters, because it is insecure and if any parameter contains appropriate quotes, postgres will interpret them as commands instead of data.

How to load data into existing database tables, using sqlalchemy?

I have my data loaded from excel files and organized as python dict where each key is database table name and its value is defined as list of dictionaries (the rows)
system_data = {table_name1:[{'col_1':val1, 'col2': val1...},
{'col_1':val2, 'col2': val2..}..],
table_name2:[{},{}..],[{},{}..]..}
This data needs to be loaded into existing database while picking table_names keys and values from system_data.
Additionally I use ordered_table list which I've created in specific order to avoid FK problems while data is being loaded.
Here is the code (one of the 1000 versions I've tried):
from sqlalchemy import create_engine
from sqlalchemy.sql import insert
def alchemy_load():
system_data = load_current_system_data()
engine = create_engine('mysql+pymysql://username:password#localhost/my_db')
conn = engine.connect()
for table_name in ordered_tables:
conn.execute(insert(table_name, system_data[table_name]))
print("System's Data successfully loaded into Database!")
This function yield a following error:
"TypeError: 'method' object is not iterable"
I've wasted almost all day on this stuff (((
All the online examples describe the situation when a user uses MetaData and creates its own tables... There is nothing about how to actually add data into existing tables.
There is a solution to my problem using "dataset" library.
The code:
import dataset
def current_data():
db = dataset.connect(url='mysql+pymysql://user:pass#localhost/my_db')
system_data = load_current_system_data()
for table_name in ordered_tables:
db[table_name].insert_many(system_data[table_name])
print("System's Data successfully loaded into Database!")
BUT, I have no idea how to implement this code using sqlalchemy...
Any help will be appreciated.

One possible solution using SQLAlchemy metadata would go like this:
In [7]:
from sqlalchemy.schema import MetaData
meta = MetaData()
meta.reflect(bind=engine)
In [20]:
for t in meta.tables:
for x in engine.execute(meta.tables[t].select()):
print x
(1, u'one')
(2, u'two')
(1, 1, 1)
(2, 1, 2)
(3, 2, 0)
(4, 2, 4)
(I use select instead of insert and apply it to a silly database I've got for trials.)
Hope it helps.
EDIT: After comments, I add some clarification.
In MetaData(), tables is a dictionary of the tables in the schema. The dictionary goes by table name, and it is actually very similar to the dictionary in your code. You could iterate the metadata like this,
for table_name, table in meta.tables.items():
for x in engine.execute(table.select()):
print x
or, trying to adapt it into your code, you could just do something like,
for table_name in ordered_tables:
conn.execute(meta.tables[table_name].insert_many(system_data[table_name]))

This is the solution I used:
from sqlalchemy import create_engine, MetaData
# from myplace import load_current_system_data() and other relevant functions
def alchemy_current():
system_data = load_current_system_data()
engine = create_engine('mysql+pymysql://user:password#localhost/your_db_name')
meta = MetaData()
meta.reflect(bind=engine)
conn = engine.connect()
for table_name in ordered_tables:
conn.execute(meta.tables[table_name].insert().values(system_data[table_name]))
conn.close()
# engine.dispose()
print("System's Data successfully loaded into Database!")
All this should work, assuming that one have:
Existing mysql database.
An organized data as I've described prior in this question.
An ordered table_name_list in order to keep referential integrity and avoid FK problems.

you could import text:
from sqlalchemy.sql import text
and then execute the following
conn.execute(text("mysql commands goes in here"))
example for insert:
conn.execute(text("INSERT INTO `table_name`(column_1,column_2,...) VALUES (value_1,value_2,...);"))

Python MySQLdb doesn't wait for the result

I am trying to run some querys that needs to create some temporary tables and then returns a result set, but i am unable to do that with MySQLdb api.
I already dig something about this issue like here but without success.
My query is like this:
create temporary table tmp1
select * from table1;
alter tmp1 add index(somefield);
create temporary table tmp2
select * from table2;
select * from tmp1 inner join tmp2 using(somefield);
This returns immediatly an empty result set. If i go to the mysql client and do a show full processlist i can see my queries executing. They take some minutes to complete.
Why cursor returns immediatly and don't wait to query to run.
If i try to run another query i have a "Commands out of sync; you can't run this command now"
I already tried to put my connection with autocommit to True
db = MySQLdb.connect(host='ip',
user='root',
passwd='pass',
db='mydb',
use_unicode=True
)
db.autocommit(True)
Or put every statement in is own cursor.execute() and between them db.commit() but without success too.
Can you help me to figure what is the problem? I know mysql don't support transactions for some operations like alter table, but why the api don't wait until everything is finished like it does with a select?
By the way i'm trying to do this on a ipython notebook.

I suspect that you're passing your multi-statement SQL string directly to the cursor.execute function. The thing is, each of the statements is a query in its own right so it's unclear what the result set should contain.
Here's an example to show what I mean. The first case is passing a semicolon set of statements to execute which is what I presume you have currently.
def query_single_sql(cursor):
print 'query_single_sql'
sql = []
sql.append("""CREATE TEMPORARY TABLE tmp1 (id int)""")
sql.append("""INSERT INTO tmp1 VALUES (1)""")
sql.append("""SELECT * from tmp1""")
cursor.execute(';'.join(sql))
print list(cursor.fetchall())
Output:
query_single_sql
[]
You can see that nothing is returned, even though there is clearly data in the table and a SELECT is used.
The second case is where each statement is executed as an independent query, and the results printed for each query.
def query_separate_sql(cursor):
print 'query_separate_sql'
sql = []
sql.append("""CREATE TEMPORARY TABLE tmp3 (id int)""")
sql.append("""INSERT INTO tmp3 VALUES (1)""")
sql.append("""SELECT * from tmp3""")
for query in sql:
cursor.execute(query)
print list(cursor.fetchall())
Output:
query_separate_sql
[]
[]
[(1L,)]
As you can see, we consumed the results of the cursor for each query and the final query has the results we expect.
I suspect that even though you've issued multiple queries, the API only has a handle to the first query executed and so immediately returns when the CREATE TABLE is done. I'd suggest serializing your queries as described in the second example above.

sqlalchemy database table is locked

I am trying to select all the records from a sqlite db I have with sqlalchemy, loop over each one and do an update on it. I am doing this because I need to reformat ever record in my name column.
Here is the code I am using to do a simple test:
def loadDb(name):
sqlite3.connect(name)
engine = create_engine('sqlite:///'+dbPath(), echo=False)
metadata = MetaData(bind=engine)
return metadata
db = database("dealers.db")
metadata = db.loadDb()
dealers = Table('dealers', metadata, autoload=True)
dealer = dealers.select().order_by(asc(dealers.c.id)).execute()
for d in dealer:
u = dealers.update(dealers.c.id==d.id)
u.execute(name="hi")
break
I'm getting the error:
sqlalchemy.exc.OperationalError: (OperationalError) database table is locked u'UPDATE dealers SET name=? WHERE dealers.id = ?' ('hi', 1)
I'm very new to sqlalchemy and I'm not sure what this error means or how to fix it. This seems like it should be a really simple task, so I know I am doing something wrong.

With SQLite, you can't update the database while you are still performing the select. You need to force the select query to finish and store all of the data, then perform your loop. I think this would do the job (untested):
dealer = list(dealers.select().order_by(asc(dealers.c.id)).execute())
Another option would be to make a slightly more complicated SQL statement so that the loop executes inside the database instead of in Python. That will certainly give you a big performance boost.