We Keep Coding

sqlite: Calling database each time function is called

I got a function that is called to calculate a response every time the user inputs something. The function gets the response from a database. What I don't understand is, why I have to redefine my variable (I have called it intents_db) that contains all the data from the database each time the function is called? I have tried putting it outside the function, but then my program only works the first time, but the returns an empty answer the second time the user inputs something.
def response(sentence, user_id='1'):
results = classify_intent(sentence)
intents_db = c.execute("SELECT row_num, responses, tag, responses, intent_type, response_type, context_set,\
context_filter FROM intents")
if results:
# loop as long as there are matches to process
while results:
if results[0][1] > answer_threshold:
for i in intents_db:
# print('tag:', i[2])
if i[2] == results[0][0]:
print(i[6])
if i[6] != 'N/A':
if show_details:
print('context: ', i[6])
context[user_id] = i[6]
responses = i[1].split('&/&')
print(random.choice(responses))
if i[7] == 'N/A' in i or \
(user_id in context and i[7] in i and i[7] == context[
user_id]):
# a random response from the intent
responses = i[1].split('&/&')
print(random.choice(responses))
print(i[4], i[5])
print(results[0][1])
elif results[0][1] <= answer_threshold:
print(results[0][1])
for i in intents_db:
if i[2] == 'unknown':
# a random response from the intent
responses = i[1].split('&/&')
print(random.choice(responses))
initial_comm_output = random.choice(responses)
return initial_comm_output
else:
initial_comm_output = "Something unexpected happened when calculating response. Please restart me"
return initial_comm_output
results.pop(0)
return results
Also, I started getting into databases and sqlite3 because I want to make a massive database long term. Therefore it also seems inefficient that I have to load the whole database at all. Is there some way I can only load the row of data I need? I got a row_number column in my database, so if it was somehow possible to say like:
"SELECT WHERE row_num=2 FROM intents"
that would be great, but I can't figure out how to do it.

cursor.execute() returns an iterator, and you can only loop over it once.
If you want to reuse it, turn it into a list:
intents_db = list(c.execute("..."))

Therefore it also seems inefficient that I have to load the whole database at all. Is there some way I can only load the row of data I need? I got a row_number column in my database, so if it was somehow possible to say like: "SELECT WHERE row_num=2 FROM intents" that would be great, but I can't figure out how to do it.
You nearly got it: it is
intents_db = c.execute("SELECT row_num, responses, tag, responses, intent_type,
response_type, context_set, context_filter
FROM intents WHERE row_num=2")
But don't do the mistake other database beginners make and try to put in some variable from your program directly into that string. This makes the program prone to SQL injections.
Rather, do
row_num = 2
intents_db = c.execute("SELECT row_num, responses, tag, responses, intent_type,
response_type, context_set, context_filter
FROM intents WHERE row_num=?", (row_num,))
Of course, you can also set conditions for other fields.

Inserting data using subqueries and Python

I'm a bit new to the whole Python + DB interactions and I have encountered some issues while trying to insert into a table using subqueries. At this point I imagine it's only a problem of syntax/missing parentheses/missing exclamation marks/etc but I can't seem to be able to figure it out by myself so maybe a fresh eye could help with the issue.
This is the query I am trying to run :
self.cur.execute("INSERT INTO game_genre (id_game, id_genre) VALUES (( SELECT gd.id_game from game_details gd where gd.title like %s, ( "%" + x + "%" , )) , (SELECT g.id_genre from genres g where g.title_genre like %s, ("%" + genre + "%",))"))
where "x" and "genre" are variables
I have tested the queries independently (outside of the insert) and they return the expected result
Could someone shed some divine light onto this?? Thanks

It looks like you're trying to parameterize your query, but you've stuck two different parameter lists into your query as postgres code instead of just one after your query as python code. It's always a good idea to format your code so that the overall structure jumps out at you.
Try:
self.cur.execute("""
INSERT INTO game_genre (id_game, id_genre)
VALUES (
(SELECT gd.id_game FROM game_details gd WHERE gd.title LIKE %s),
(SELECT g.id_genre FROM genres g WHERE g.title_genre LIKE %s))
""", ("%" + x + "%", "%" + genre + "%"))
See http://initd.org/psycopg/docs/usage.html for more details on query parameterization. In particular, note that execute() takes a query with a bunch of %ss, and then a parameter list with one value for each %s. Using """ around every query is highly recommended, in part for readability, and in part because it will prevent you from having to escape " and ' characters.

ValueError: invalid literal for int() with base 10: 'NewFilmLikes' in Python

I am trying to get a value from an SQLite3 database and put that into a variable called FilmLikes, I then want to add one to it and I store that in the variable NewFilmLikes. I then try to change the value of this box in my database to the 'NewFilmLikes', however I always get this problem:
ValueError: invalid literal for int() with base 10: 'NewFilmLikes'
Here is the code showing that:
FilmLikes=i[2]
NewFilmLikes=(int(FilmLikes))+1
filmUpdate=("UPDATE HORRORALL SET likes = ? WHERE number = 1"(NewFilmLikes))
c.execute(filmUpdate)
conn.commit()
Here is the code for i:
for i in find_film:
print('')
print('Number',(i[0]))
print(i[1])
filmChoice=int(input('Please choose which number you would like to like: '))
if filmChoice==1:
find_film=c.execute("SELECT * FROM HORRORALL")
for i in find_film:
if i[0]==filmChoice:
## FilmLikes=i[2]
try:
x=(int(i[2]))+1
except:
pass
## FilmLikes=i[2]
## NewFilmLikes=(int(str(FilmLikes)))+1
filmUpdate=("UPDATE HORRORALL SET likes = ? WHERE number = 1"(x))
c.execute(filmUpdate)
conn.commit()

Without knowing much more about your setup, it's hard to answer this properly.
Mind you, the specific error you refer to (ValueError: invalid literal for int() with base 10: 'NewFilmLikes') often occurs when you get a decimal rather than an integer.
Here's a sort of general answer.
It's good practice not to use SELECT * because it hides what you are trying to retrieve. Specify, e.g SELECT film_id, file_name, no_of_likes FROM horrorall as this way you can see what you are retrieving and the positions.
You're retrieving the entire list of films with find_film and then looping through to find which one. SQL has a find function, it's called WHERE and you are using it in the film_update section. If you have the ID or Name inputted by the user, you can then call only the record you need. E.g. "SELECT film_id, file_name, no_of_likes FROM horrorall WHERE film_id = ?" (filmChoice,)
Note that parameterised queries use a tuple, even if there is only one parameter - hence it is (filmChoice,) with a comma (and for filmUpdate you would need (x,))
filmUpdate will only ever update the first film. I assume that because your if ... loop continues through all the possible options? You can shortly this by putting another ? in there for your WHERE e.g. filmUpdate=("UPDATE HORRORALL SET likes = ? WHERE number = ?"(x,i[0]))
How does the user know what film to select? You aren't retrieving the list until they give you a number!
Putting all of this together, here is an example. Note this is far from the ideal way of doing it, I don't want to throw too many ideas at you in one go!
# Get the whole list of films to show to the user
find_film = c.execute("SELECT film_id, film_name FROM HORRORALL")
# Present the list for selection
for films in find_film:
print(str(films[0]) + '. ' + films[1])
# Get the users input as an integer
filmChoice = int(input('Please choose which number you would like to like: '))
# Fetch the number of likes of the selected film
film_likes = c.execute('SELECT likes FROM HORRORALL WHERE film_id = ?' (filmChoice,))
# Send the update back to the database
c.execute("UPDATE HORRORALL SET likes = ? WHERE film_id = ?"(int(film_likes[0]) + 1, filmChoice)
conn.commit()

Python SQL select row from specific variable field

I'd like to grab a specific value from a row based on a random variable. Here's an example table the PID column is an "auto-increment primary key integer" and the other 2 columns are TEXT
example-table
PID NAME PHONE
--- ---- -----
1 bill 999-9999
2 joe 888-8888
I'd like to throw a random variable at the table
randomVariable = raw_input('Enter something: ')
> 1
and have the code return the name
> bill
I know I can use something like...
randomVariable = raw_input('Enter something: ')
sql = ("SELECT name FROM example_table WHERE pid='%s'" % randomVariable)
result = cursor.execute(sql)
print result
> bill
Apparently using '%s' isn't secure and it is suggested to use '?' in it's place.
randomVariable = raw_input('Enter something: ')
sql = ("SELECT name FROM example_table WHERE pid=?", randomVariable)
result = cursor.execute(sql)
print result
But this doesn't seem to work for me. I end up with...
"ValueError: operation parameter must be str or unicode"
I realize I could just grab all the rows and put them into a variable which I could then iterate over till I find what I'm looking for but I'm thinking that wouldn't be very efficient with a large database. can anyone help point me in the right direction with this?

I believe you're meant to use it like this
randomVariable = raw_input('Enter something: ')
sql = "SELECT name FROM example_table WHERE pid=?"
result = cursor.execute(sql, randomVariable)
print result

Validate the user input, and %s is fine. Storing your rows and putting them into a list is not a good idea at all, since the amount of rows will grow over time, taking up a huge amount of memory when not even in use. To guard against SQL injection, you could validate input using something like a typecast to an int, put that in a try/except block, and this would stop all malicious input such as ' OR 1=1--

App Engine Datastore IN Operator - how to use?

Reading: http://code.google.com/appengine/docs/python/datastore/gqlreference.html
I want to use:
:= IN
but am unsure how to make it work. Let's assume the following
class User(db.Model):
name = db.StringProperty()
class UniqueListOfSavedItems(db.Model):
str = db.StringPropery()
datesaved = db.DateTimeProperty()
class UserListOfSavedItems(db.Model):
name = db.ReferenceProperty(User, collection='user')
str = db.ReferenceProperty(UniqueListOfSavedItems, collection='itemlist')
How can I do a query which gets me the list of saved items for a user? Obviously I can do:
q = db.Gql("SELECT * FROM UserListOfSavedItems WHERE name :=", user[0].name)
but that gets me a list of keys. I want to now take that list and get it into a query to get the str field out of UniqueListOfSavedItems. I thought I could do:
q2 = db.Gql("SELECT * FROM UniqueListOfSavedItems WHERE := str in q")
but something's not right...any ideas? Is it (am at my day job, so can't test this now):
q2 = db.Gql("SELECT * FROM UniqueListOfSavedItems __key__ := str in q)
side note: what a devilishly difficult problem to search on because all I really care about is the "IN" operator.

Since you have a list of keys, you don't need to do a second query - you can do a batch fetch, instead. Try this:
#and this should get me the items that a user saved
useritems = db.get(saveditemkeys)
(Note you don't even need the guard clause - a db.get on 0 entities is short-circuited appropritely.)
What's the difference, you may ask? Well, a db.get takes about 20-40ms. A query, on the other hand (GQL or not) takes about 160-200ms. But wait, it gets worse! The IN operator is implemented in Python, and translates to multiple queries, which are executed serially. So if you do a query with an IN filter for 10 keys, you're doing 10 separate 160ms-ish query operations, for a total of about 1.6 seconds latency. A single db.get, in contrast, will have the same effect and take a total of about 30ms.

+1 to Adam for getting me on the right track. Based on his pointer, and doing some searching at Code Search, I have the following solution.
usersaveditems = User.Gql(“Select * from UserListOfSavedItems where user =:1”, userkey)
saveditemkeys = []
for item in usersaveditems:
#this should create a list of keys (references) to the saved item table
saveditemkeys.append(item.str())
if len(usersavedsearches > 0):
#and this should get me the items that a user saved
useritems = db.Gql(“SELECT * FROM UniqueListOfSavedItems WHERE __key__ in :1’, saveditemkeys)