We Keep Coding

getting only updated data from database

I have to get the recently updated data from database. For the purpose of solving it, I have saved the last read row number into shelve of python. The following code works for a simple query like select * from rows. My code is:
from pyodbc import connect
from peewee import *
import random
import shelve
import connection
d = shelve.open("data.shelve")
db = SqliteDatabase("data.db")
class Rows(Model):
valueone = IntegerField()
valuetwo = IntegerField()
class Meta:
database = db
def CreateAndPopulate():
db.connect()
db.create_tables([Rows],safe=True)
with db.atomic():
for i in range(100):
row = Rows(valueone=random.randrange(0,100),valuetwo=random.randrange(0,100))
row.save()
db.close()
def get_last_primay_key():
return d.get('max_row',0)
def doWork():
query = "select * from rows" #could be anything
conn = connection.Connection("localhost","","SQLite3 ODBC Driver","data.db","","")
max_key_query = "SELECT MAX(%s) from %s" % ("id", "rows")
max_primary_key = conn.fetch_one(max_key_query)[0]
print "max_primary_key " + str(max_primary_key)
last_primary_key = get_last_primay_key()
print "last_primary_key " + str(last_primary_key)
if max_primary_key == last_primary_key:
print "no new records"
elif max_primary_key > last_primary_key:
print "There are some datas"
optimizedQuery = query + " where id>" + str(last_primary_key)
print query
for data in conn.fetch_all(optimizedQuery):
print data
d['max_row'] = max_primary_key
# print d['max_row']
# CreateAndPopulate() # to populate data
doWork()
While the code will work for a simple query without where clause, but the query can be anything from simple to complex, having joins and multiple where clauses. If so, then the portion where I'm adding where will fail. How can I get only last updated data from database whatever be the query?
PS: I cannot modify database. I just have to fetch from it.

Use an OFFSET clause. For example:
SELECT * FROM [....] WHERE [....] LIMIT -1 OFFSET 1000
In your query, replace 1000 with a parameter bound to your shelve variable. That will skip the top "shelve" number of rows and only grab newer ones. You may want to consider a more robust refactor eventually, but good luck.

Python not able to modify MySQL but user can

Sorry if this question is stupid, I am 2 days into learning python
I have been beating my head against a wall trying to understand why my python script can run SELECT statements but not UPDATE or DELETE statements.
I believe this would be a MySQL issue and not a Python issue but I am no longer able to troubleshoot
pcheck.py
import re
import time
import json
import MySQLdb
import requests
from array import *
conn = MySQLdb.connect([redacted])
cur = conn.cursor()
sql1 = "SELECT pkey,pmeta FROM table1 WHERE proced = 0 LIMIT 1"
cur.execute(sql1)
row = cur.fetchone()
while row is not None:
print "row is: ",row[0]
rchk = [
r"(SHA256|MD5)",
r"(abc|def)"
]
for trigger in rchk:
regexp = re.compile(trigger)
pval = row[1]
if regexp.search(pval) is not None:
print "matched on: ",row[0]
sql2 = """INSERT INTO table2 (drule,dval,dmeta) VALUES('%s', '%s', '%s')"""
try:
args2 = (trigger, pval, row[1])
cur.execute(sql2, args2)
print(cur._last_executed)
except UnicodeError:
print "pass-uni"
break
else:
pass
sql3 = """UPDATE table1 SET proced=1 WHERE pkey=%s"""
args3 = row[0]
cur.execute(sql3, args3)
print(cur._last_executed)
row = cur.fetchone()
sql3 = """DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"""
cur.execute(sql3)
print(cur._last_executed)
cur.close()
conn.close()
print "Finished"
And the actual (and suprisingly expected) output:
OUTPUT
scrape#:~/python$ python pcheck.py
row is: 0GqQ0d6B
UPDATE table1 SET proced=1 WHERE pkey='0GqQ0d6B'
DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)
Finished
However, the database is not being UPDATED. I checked that the query was making it to MySQL:
MySQL Log
"2015-12-14 22:53:56","localhost []","110","0","Query","SELECT `pkey`,`pmeta` FROM `table1` WHERE `proced`=0 LIMIT 200"
"2015-12-14 22:53:57","localhost []","110","0","Query","UPDATE `table1` SET `proced`=1 WHERE `pkey`='0GqQ0d6B'"
"2015-12-14 22:53:57","localhost []","110","0","Query","DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"
However proced value for row 0GqQ0d6B is still NOT 1
If I make the same queries via Sqlyog (logged in as user) the queries work as expected.

These kind of issues can be very frustrating. You sure there's no extra spaces here?
print "row is:*"+row[0]+"*"
Perhaps comment out the
for trigger in rchk:
section, and sprinkle some print statements around?

As the commenter Bob Dylan was able to deduce the cursor needed to be committed after the change.

In Django cursor.execute("select ..") randomly returns None

I am running Django + MySql on OpenShift server and have this problem.
Code:
from django.db import connection
...
cursor = connection.cursor()
somedate = calculateSomeDate()
query = "SELECT id FROM levelbuffer WHERE START > '%s' ORDER BY START ASC LIMIT 1" % somedate
print "First select: " + query
cursor.execute(query)
sql_result = cursor.fetchone()
if not sql_result == None:
gameId = int(sql_result[0])
cursor.fetchall() #To clear anything left behind (this wasn't here before, but no changes appeared after adding)
query = "SELECT gamescores.skore, users.name FROM gamescores JOIN users ON gamescores.userId = users.id where gamescores.gameId = " + str(gameId) + " ORDER BY skore ASC"
cursor.execute(query);
#cursor.execute("SELECT gamescores.skore, users.name FROM gamescores JOIN users ON gamescores.userId = users.id where gamescores.gameId = %s ORDER BY skore ASC", (gameId))
print "Second select " + query
row = cursor.fetchone()
The weird thing is, that the row is sometimes None and sometimes it contains wanted values. I cant figure out, what i am missing. The first query works fine every time.
When I try to perform the second query printed in log on phpmyadmin - works always fine.

Read things out of database WHERE 'id'=

I am trying to build an examine (Dutch = overhoor) program in Python. Here's my code:
cursor = cnx.cursor()
willekeurig = random.randint(0,9)
query = ('SELECT FRwoord, NLwoord FROM unite8app1 WHERE "id"=willekeurig')
cursor.execute(query)
for (NLwoord, FRwoord) in cursor:
print("Wat is de vertaling van: " + FRwoord + "?")
vertaling = input()
if vertaling == NLwoord:
print("Well done")
else:
print("Helaas")
for (FRwoord, NLwoord) in cursor:
print(FRwoord,NLwoord)
print(query)
cnx.close
I am trying to get a random question based upon the id of that question in my database. I tried some alternatives like:
WHERE 'id'=1
WHERE 'ID'=1
etc
But it doesn't want to work if I run my program (after pressing enter) it says that NLwoord and FRwoord aren't defined, but when I remove that bit of code it works just fine. (if I have just 1 item in my database)
My question is how to grab a random id from my database? I have included some pictures of the database.
http://updo.nl/file/37e39c15.JPG
http://updo.nl/file/1de64d64.JPG

Right now you are not passing the value of willekeurig, but the text willekeurig. Try these two lines for your query & execute insted:
query = "SELECT FRwoord, NLwoord FROM unite8app1 WHERE id=%s"
cursor.execute(query, (willekeurig,))

Have you tried this without any quotes around id?
You should also be able to use:
SELECT FRwoord, NLwoord FROM unite8app1 ORDER BY RAND() LIMIT 1

using results from a sql query in a python program in another sql query

sorry for my previous question which was very ambiguous, but i think if i get the answer to this question I can work it out.
In the program below i have selected the barcodes of products where the amount is less than the quantity. I want to say, that if the barcodes(in the fridge table) match barcodes in another table(products), set the stock field equal to 0. The problem Im getting is that the program is trying to match all the barcodes that it found in the query against single barcodes in the products table(thats what I think). does anyone know what to do. thanks a million. lincoln.
import MySQLdb
def order():
db = MySQLdb.connect(host='localhost', user='root', passwd='$$', db='fillmyfridge')
cursor = db.cursor()
cursor.execute('select barcode from fridge where amount < quantity')
db.commit()
row = cursor.fetchall()
cursor.execute('update products set stock = 0 where barcode = %s', row)

UPDATE products SET stock = 0 WHERE barcode IN (
SELECT fridge.barcode FROM fridge WHERE fridge.amount < fridge.quantity );
I know this doesn't answer the question exactly but two SQL statements are not required.
To do it in python:
import MySQLdb
def order():
db = MySQLdb.connect(host='localhost', user='root', passwd='$$', db='fillmyfridge')
cursor = db.cursor()
cursor.execute('select barcode from fridge where amount < quantity')
db.commit()
rows = cursor.fetchall()
for row in rows
cursor.execute('update products set stock = 0 where barcode = %s', row[0])

This is more of SQL query than Python, but still I will try to answer that:
(I haven't worked with MySQL but PostgreSQL, so there might slight variation in interpretation of things here).
when you did
cursor.execute('select barcode from fridge where amount < quantity')
db.commit()
row = cursor.fetchall()
the variable 'row' now is a resultset (to understand: a list of rows from the database)
something like [(barcode1), (barcode2), (barcode3)..]
when you do the update statement
cursor.execute('update products set stock = 0 where barcode = %s', row)
this turns into something like:
update products set stock = 0 where barcode = [(barcode1), (barcode2), (barcode3)..]
which is not a correct SQL statement.
you should do something like this:
cursor.execute('update products set stock = 0 where barcode in (%s)', ','.join([each[0] for each in row]))
or better, the optimized thing:
import MySQLdb
def order():
db = MySQLdb.connect(host='localhost', user='root', passwd='$$', db='fillmyfridge')
cursor = db.cursor()
cursor.execute('update products set stock = 0 where barcode in (select barcode from fridge where amount < quantity)')
db.commit()
Well, to add more you have a db.commit() after a select query and not after an update query, thats a basic fault. Select is idempotent, doesn't need commit, whereas Update does. I will strongly recommend you to go through some SQL before continuing.