i just started out with programmming and wrote a few lines of code in pyscripter using sqlite3.
The table "gather" is created beforehand. i then select certain rows from "gather" to put them into another table. i try to sort this table by a specific column 'date'. But it doesn't seem to work. it doesn't give me an error message or something like that. It's just not sorted. If i try the same command (SELECT * FROM matches ORDER BY date) in sqlitemanager, it works fine on the exact same table! what is the problem here? i googled quite some time, but i don't find a solution. it's proobably something stupid i'm missing..
as i said i'm a total newbie. i guess you all break out in tears looking at the code. so if you have any tips how i can shorten the code or make it faster or whatever, you're very welcome :) (but everything works fine except the above mentioned part.)
import sqlite3
connection = sqlite3.connect("gather.sqlite")
cursor1 = connection.cursor()
cursor1.execute('Drop table IF EXISTS matches')
cursor1.execute('CREATE TABLE matches(date TEXT, team1 TEXT, team2 TEXT)')
cursor1.execute('INSERT INTO matches (date, team1, team2) SELECT * FROM gather WHERE team1=? or team2=?, (a,a,))
cursor1.execute("SELECT * FROM matches ORDER BY date")
connection.commit()
OK, I think I understand your problem. First of all: I'm not sure if that commit call is necessary at all. However, if it is, you'll definitely want that to be before your select statement. 'connection.commit()' is essentially saying, commit the changes I just made to the database.
Your second issue is that you are executing the select query but never actually doing anything with the results of the query.
try this:
import sqlite3
connection = sqlite3.connect("gather.sqlite")
cursor1 = connection.cursor()
cursor1.execute('Drop table IF EXISTS matches')
cursor1.execute('CREATE TABLE matches(date TEXT, team1 TEXT, team2 TEXT)')
cursor1.execute('INSERT INTO matches (date, team1, team2) SELECT * FROM gather WHERE team1=? or team2=?, (a,a,))
connection.commit()
# directly iterate over the results of the query:
for row in cursor1.execute("SELECT * FROM matches ORDER BY date"):
print row
you are executing the query, but never actually retrieving the results. There are two ways to do this with sqlite3: One way is the way I showed you above, where you can just use the execute statement directly as an iteratable object.
The other way is as follows:
import sqlite3
connection = sqlite3.connect("gather.sqlite")
cursor1 = connection.cursor()
cursor1.execute('Drop table IF EXISTS matches')
cursor1.execute('CREATE TABLE matches(date TEXT, team1 TEXT, team2 TEXT)')
cursor1.execute('INSERT INTO matches (date, team1, team2) SELECT * FROM gather WHERE team1=? or team2=?, (a,a,))
connection.commit()
cursor1.execute("SELECT * FROM matches ORDER BY date")
# fetch all means fetch all rows from the last query. here you put the rows
# into their own result object instead of directly iterating over them.
db_result = cursor1.fetchall()
for row in db_result:
print row
Try moving the commit before the SELECT * (I'm not sure 100% that this is an issue) You then just need to fetch the results of the query :-) Add a line like res = cursor1.fetchall() after you've executed the SELECT. If you want to display them like in sqlitemanager, add
for hit in res:
print '|'.join(hit)
at the bottom.
Edit: To address your issue of storing the sort order to the table:
I think what you're looking for is something like a clustered index. (Which doesn't actually sort the values in th table, but comes close; see here).
SQLIte doesn't have such such indexes, but you can simulate them by actually ordering the table. You can only do this once, as you're inserting the data. You would need an SQL command like the following:
INSERT INTO matches (date, team1, team2)
SELECT * FROM gather
WHERE team1=? or team2=?
ORDER BY date;
instead of the one you currently use.
See point 4 here, which is where I got the idea.
Related
Having a strange issue with pymysql and python. I have a table where date_rec is one of 3 columns composing a primary key. If I do this select, it takes forever to get the result
query = f"SELECT * FROM string WHERE date_rec BETWEEN {date_before} AND {date_after} ORDER BY date_rec"
with connection.cursor() as cursor:
cursor.execute(query)
result = cursor.fetchone()
for row in result:
print(row)
However if I add a limit of 5000, it works superfast, even though there are only 1290 records to be found. The 5000 number doesn't matter... 50,000 fixes the problem exactly the same way (just as fast). As long as it's more than 1290, I get all the records.
query = f"SELECT * FROM string WHERE date_rec BETWEEN {date_before} AND {date_after} ORDER BY date_rec LIMIT 5000"
with connection.cursor() as cursor:
cursor.execute(query)
result = cursor.fetchone()
for row in result:
print(row)
Can someone explain what's happening here and how to make the first case work as fast as the second? Thanks.
EDIT:
3 columns compose primary key:
date_rec
customer_number
order_number
So I did explain on SQL workbench and got this:
Limit-less query
Query with 5000 limit
So Mysql wasn't using the index for whatever reason. Putting in "USE INDEX(PRIMARY)" inside the query fixed the problem.
I have an explanation as to why adding a LIMIT clause speeds up the query, but if you want to tune the query, then consider adding the following index:
CREATE INDEX idx ON string (date_rec);
This index will let MySQL quickly filter off records not inside the date range, and it also provides the ordering needed in the ORDER BY clause.
I'm currently writing a program for a parents evening system. I have two tables, a bookings table and a teacher table - set up with the following column headings: TeacherSubject | 15:30 | 15:35 | 15:40 etc... When people make a booking, they select a teacher from a drop-down menu and also a time. Therefore, I need the bookingID added into the booking table where the teacher selected = to the same teacher in the table and where time selected = time in the database.
At the moment, my code only attempts to match the teacher, but this doesn't work as I'm getting the error of: (line 5)
TypeError: 'str' object is not callable
Am I doing the whole thing wrong and is this actually possible with the way I have set the table up?
def insert(parent_name, parent_email, student_name,student_form,teacher,app_time,comments):
conn=sqlite3.connect("parentsevening.db")
cur=conn.cursor()
cur.execute("INSERT INTO bookings VALUES (NULL,?,?,?,?,?,?,?)",(parent_name,parent_email,student_name,student_form,teacher,app_time,comments))
cur.execute("INSERT INTO teachers VALUES (?) WHERE teachers = (?)" (id,teacherName,))
conn.commit()
conn.close()
This SQL Query is invalid.
INSERT INTO teachers VALUES (?) WHERE teachers = (?)
It should be
INSERT INTO teachers (id, name) VALUES(?, ?)
Note that I'm guessing the teachers columns (id, name) WHERE on the insert isn't valid because it's used to find data (SELECT, UPDATE, DELETE)
OK, let's take out the comments and make this into an answer.
Python error
I think your error comes from WHERE teachers = (?) have you tried WHERE teachers = ? instead.
But...
bad sql syntax
Also that command as a whole doesnt make much sense, SQL syntax wise - you seem to be trying to insert where a teacher that doesn't exist (if you are inserting them) and values on an insert does not go with where and where needs a from. i.e. once you've solved your python error, sqlite is going to have a fit as well.
That's already covered by another answer.
But...
probably not what you should be doing
If you have an existing teacher, you only need to insert their teacherid into table bookings. You don't have to, and in fact, you can't insert into table teachers at this point, you'd get a duplicate data error.
So, rather than fixing your second query, just get rid of it entirely.
If you can get a command line or GUI SQL tool up, try running these queries by hardcoding them by hand before coding them in Python. the sqlite command should be able to do that for you.
(recommendation) don't use insert table values
Try being explicit with insert into table (<column list>) values .... The reason is that, as soon as the table changes in some way that affects column order (possibly an alter column) the values won't line up with the implied insert list. hard to debug, hard to know what was intended at time of writing. Been there, done that. And had to debug buncha folks' code who took this shortcut, it's never fun
I have a table using SQL Lite with Python. The size of the table always has 3 columns and could have many rows. Each of the cells are strings. Here is example table:
serial_num date_measured status
1234A 1-1-2015 passed
4321B 6-21-2015 failed
1423C 12-25-2015 passed
......
My program prompts me for a serial number. This is saved as a variable called serialNum. How can I delete (or overwrite) an entire row if serialNum equals any of the strings in the serial_num column in my table?
I've seen many examples on how to delete (or overwrite) a row in a table if I know all the values in each cell of that row, but my trouble is that the only cell that could ever be the same in each row would be the serial number. I need to so a search through the serial_number column and if any string in that column equals the current value of my serialNum variable, I need to delete (or overwrite) that row.
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
c.execute('''CREATE TABLE test (serial_num text, date_measured text, status text)''')
c.execute("INSERT INTO test VALUES ('1234A', '1-1-2015', 'passed')")
c.execute("INSERT INTO test VALUES ('4321B', '6-21-2015', 'failed')")
c.execute("INSERT INTO test VALUES ('1423C', '12-25-2015', 'passed')")
conn.commit()
Does anyone know a simple way to do this? I've seen others say that an ID must be used or a temporary table, but I would hope there might be an easier way to accomplish my task. Any advice would be great.
SQL suports this: simply use delete
"delete from test where serial_num=<some input>;"
or in this case
c.execute("delete from test where serial_num=%s;", serialNum);
There's no need to search through the list when using SQL. SQL is declarative: you tell it what to do using your query, not how to do it. Don't loop though all your rows to check which to delete: tell it what to delete and the database engine will find the best/fastest way to satisfy that goal.
Hope I well interpreted your question
for row in c.execute('SELECT * FROM test WHERE serial_num = ?', serialNum'):
# do whatever you want on row
print row
I was able to figure out a working solution:
sql = "DELETE FROM test WHERE serial_num = ?"
c.execute(sql, (serialNum,))
The comma after serialNum for some reason has to be there. Thank you #Michiel Arienfor the head start
I am trying to run some querys that needs to create some temporary tables and then returns a result set, but i am unable to do that with MySQLdb api.
I already dig something about this issue like here but without success.
My query is like this:
create temporary table tmp1
select * from table1;
alter tmp1 add index(somefield);
create temporary table tmp2
select * from table2;
select * from tmp1 inner join tmp2 using(somefield);
This returns immediatly an empty result set. If i go to the mysql client and do a show full processlist i can see my queries executing. They take some minutes to complete.
Why cursor returns immediatly and don't wait to query to run.
If i try to run another query i have a "Commands out of sync; you can't run this command now"
I already tried to put my connection with autocommit to True
db = MySQLdb.connect(host='ip',
user='root',
passwd='pass',
db='mydb',
use_unicode=True
)
db.autocommit(True)
Or put every statement in is own cursor.execute() and between them db.commit() but without success too.
Can you help me to figure what is the problem? I know mysql don't support transactions for some operations like alter table, but why the api don't wait until everything is finished like it does with a select?
By the way i'm trying to do this on a ipython notebook.
I suspect that you're passing your multi-statement SQL string directly to the cursor.execute function. The thing is, each of the statements is a query in its own right so it's unclear what the result set should contain.
Here's an example to show what I mean. The first case is passing a semicolon set of statements to execute which is what I presume you have currently.
def query_single_sql(cursor):
print 'query_single_sql'
sql = []
sql.append("""CREATE TEMPORARY TABLE tmp1 (id int)""")
sql.append("""INSERT INTO tmp1 VALUES (1)""")
sql.append("""SELECT * from tmp1""")
cursor.execute(';'.join(sql))
print list(cursor.fetchall())
Output:
query_single_sql
[]
You can see that nothing is returned, even though there is clearly data in the table and a SELECT is used.
The second case is where each statement is executed as an independent query, and the results printed for each query.
def query_separate_sql(cursor):
print 'query_separate_sql'
sql = []
sql.append("""CREATE TEMPORARY TABLE tmp3 (id int)""")
sql.append("""INSERT INTO tmp3 VALUES (1)""")
sql.append("""SELECT * from tmp3""")
for query in sql:
cursor.execute(query)
print list(cursor.fetchall())
Output:
query_separate_sql
[]
[]
[(1L,)]
As you can see, we consumed the results of the cursor for each query and the final query has the results we expect.
I suspect that even though you've issued multiple queries, the API only has a handle to the first query executed and so immediately returns when the CREATE TABLE is done. I'd suggest serializing your queries as described in the second example above.
I am trying to do a simple fetch using MySQLDB in Python.
I have 2 tables(Accounts & Products). I have to look up Accounts table, get acc_id from it & query the Products table using it.
The Products tables has more than 10 rows. But when I run this code it randomly returns between 0 & 6 rows each time I run it.
Here's the code snippet:
# Set up connection
con = mdb.connect('db.xxxxx.com', 'user', 'password', 'mydb')
# Create cursor
cur = con.cursor()
# Execute query
cur.execute("SELECT acc_id FROM Accounts WHERE ext_acc = '%s'" % account_num ) # account_num is alpha-numberic and is got from preceding part of the program
# A tuple is returned, so get the 0th item from it
acc_id = cur.fetchone()[0]
print "account_id = ", acc_id
# Close the cursor - I was not sure if I can reuse it
cur.close()
# Reopen the cursor
cur = con.cursor()
# Second query
cur.execute("SELECT * FROM Products WHERE account_id = %d" % acc_id)
keys = cur.fetchall()
print cur.rowcount # This prints incorrect row count
for key in keys: # Does not print all rows. Tried to directly print keys instead of iterating - same result :(
print key
# Closing the cursor & connection
cur.close()
con.close()
The weird part is, I tried to step through the code using a debugger(PyDev on Eclipse) and it correctly gets all rows(both the value stored in the variable 'keys' as well as console output are correct).
I am sure my DB has correct data since I ran the same SQL on MySQL console & got the correct result.
Just to be sure I was not improperly closing the connection, I tried using with con instead of manually closing the connection and it's the same result.
I did RTM but I couldn't find much in it to help me with this issue.
Where am I going wrong?
Thank you.
EDIT: I noticed another weird thing now. In the line
cur.execute("SELECT * FROM Products WHERE account_id = %d" % acc_id), I hard-coded the acc_id value, i.e made it
cur.execute("SELECT * FROM Products WHERE account_id = %d" % 322) and it returns all rows
This is not actually an answer, just an attempt to gather together all of the information from a chat with RBK that ruled out a bunch of potential problems, but still didn't come up with an explanation or a solution, in hopes that someone else can spot the problem or think of something else to try.
It's clearly something in this line:
cur.execute("SELECT * FROM Products WHERE account_id = %d" % acc_id)
Especially since putting 322 in place of acc_id fixes everything. (As proven below.)
There are actually two problems with that line, which could be getting in the way. You always want to use DB-API binding rather than string formatting (and the equivalent in any other language), to avoid SQL injection attacks, for correctness of escaping/conversion/etc., and for efficiency. Also, both DB-ABI binding and string formatting require a tuple of arguments, not a single argument. (For legacy reasons, a single argument often works, but sometimes it doesn't, and then it's just confusing to debug… better not to do it.) So, this should be:
cur.execute("SELECT * FROM Products WHERE account_id = %d", (acc_id,))
Unfortunately, after discussing this in chat, and having you try a bunch of things, we were unable to find what's actually wrong here. Summarizing what we tried:
So then, we tried:
cur.execute("SELECT COUNT(*) FROM Devices WHERE account_id = %s" , (333,))
print cur.fetchone()[0]
print 'account id =', acc_id
print type(acc_id)
cur.execute("SELECT COUNT(*) FROM Devices WHERE account_id = %s" , (acc_id,))
print cur.fetchone()[0]
The output was:
10
account id = 333
<type 'long'>
2
When run repeatedly, the last number varies from 0-6, while the first is always 10. There's no way using acc_id could be different from using 333, and yet it is. And just in case one query was somehow "infecting" the next one, without the first two lines, the rest works the same way.
So, there's no way using acc_id could possibly be different than using 333. And yet, it is.
At some point during the chat we apparently moved from Products to Devices, and from 322 to 333, but regardless, the tests shown above were definitely done exactly as shown, and returned different results.
Maybe he has a buggy or badly-installed version of MySQLDb. He's going to try looking for a newer version, or one of the other Python MySQL libraries, and see if it makes a difference.
My next best guess at this point is that RBK has inadvertently angered some technologically-sophisticated god of mischief, but I can't even think of one of those off the top of my head.
I sort of figured out the problem. It was silly at the end. It was a race condition!
This is how my actual code was organized :
Code Block 1
{code which calls an API which creates an entry in Accounts table &
Creates corresponding entries in Product table(10 entries)}
......
Code Block2
{The code I had posted in my question}
The problem was that the API(called in Code Block 1) took a few seconds to add 10 entries into the Product table.
When my code(Code Block 2) ran a fetch query, all the 10 rows were not added and hence fetched somewhere between 0 to 6 rows(how much ever was added at that time).
What I did to solve this was made the code sleep for 5 seconds before I did the SQL queries:
Code Block 1
time.sleep(5)
Code Block 2
The reason why it worked when I hard coded the acc_id was that, the acc_id which I hard-coded was from a precious execution(each run returns a new acc_id).
And the reason why it worked while stepping through a debugger was that manually stepping acted like giving it a sleep time.
It is a lesson for me to know a little about the inside working of APIs(even though they are supposed to be like a black box) and think about race conditions like this, the next time I come across similar issues.