We Keep Coding

Dynamically search for null in sqlite select query using python

I'm new to python and I want to do a similar query to this one:
_c.execute('select * from cases where bi = ? and age = ? and
shape = ? and margin = ? and density = ?',(obj['bi'],
obj['age'], obj['margin'], obj['density']))
When some of the parameters are None, for example obj['bi'] = None, the query searches for the row when bi = 'None'. But I want it to search for the row when: 'bi is NULL'
A possible solution is to verify the values of the parameters one by one in a sequence of if-elses. For example:
query = 'select * from cases where'
if obj['bi'] is None:
query += ' bi is null'
else:
query += ' bi = ' + str(obj['bi']) + ' and '
...
# do the same if-else for the other parameters
...
_c.execute(query)
But, it doesn't seem to me as the best solution.
The question is, what is the best solution to the given problem and how to avoid SQL injections.

Okay, after firing up a python REPL and playing around with it a bit, it's simpler than I thought. The Python sqlite bindings turn a Python None into a SQL NULL, not into a string 'None' like it sounded like from your question. In SQL, = doesn't match NULL values, but IS will. So...
Given a table foo looking like:
a | b
--------------
NULL | 1
Dog | 2
Doing:
c = conn.cursor()
c.execute('SELECT * FROM foo WHERE a IS ?', (None,))
print(c.fetchone())
will return the (NULL, 1) row, and
c.execute('SELECT * FROM foo WHERE a IS ?', ('Dog',))
print(c.fetchone())
will return the ('Dog', 2) row.
In other words, use IS not = in your query.

Insert list or tuple into table without iteration into postgresql

I am new to python. What I am trying to achieve is to insert values from my list/tuple into my redshift table without iteration.I have around 1 million rows and 1 column. Below is the code I am using to create my list/tuple.
cursor1.execute("select domain from url limit 5;")
for record, in cursor1:
ext = tldextract.extract(record)
mylist.append(ext.domain + '.' + ext.suffix)
mytuple = tuple(mylist)
I am not sure what is best to use, tuple or list. output of print(mylist) and print(mytuple) are as follows.
List output
['friv.com', 'steep.tv', 'wordpress.com', 'fineartblogger.net',
'v56.org'] Tuple Output('friv.com', 'steep.tv', 'wordpress.com',
'fineartblogger.net', 'v56.org')
Now, below is the code I am using to insert the values into my redshift table but I am getting an error:
cursor2.execute("INSERT INTO sample(domain) VALUES (%s)", mylist) or
cursor2.execute("INSERT INTO sample(domain) VALUES (%s)", mytuple)
Error - not all arguments converted during string formatting
Any help is appreciated. If any other detail is required please let me know, I will edit my question.
UPDATE 1:
Tried using below code and getting different error.
args_str = ','.join(cur.mogrify("(%s)", x) for x in mylist)
cur.execute("INSERT INTO table VALUES " + args_str)
ERROR - INSERT has more expressions than target columns

I think you're looking for Fast Execution helpers:
mylist=[('t1',), ('t2',)]
execute_values(cursor2, "INSERT INTO sample(domain) %s", mylist, page_size=100)
what this does is it replaces the %s with 100 VALUES. I'm not sure how high you can set page_size, but that should be far more performant.

Finally found a solution. For some reason cur.mogrify was not giving me proper sql string for insert. Created my own SQl string and it works alot faster than cur.executeall()
list_size = len(mylist)
for len in range(0,list_size):
if ( len != list_size-1 ):
sql = sql + ' ('+ "'"+ mylist[len] + "'"+ ') ,'
else:
sql = sql + '('+ "'"+ mylist[len] + "'"+ ')'
cursor1.execute("INSERT into sample(domain) values " + sql)
Thanks for your help guys!

In python, How to update variable dynamically? [duplicate]

This question already has answers here:
How do I put a variable’s value inside a string (interpolate it into the string)?
(9 answers)
Python: best practice and securest way to connect to MySQL and execute queries
(2 answers)
Reduce tuple of one item (singleton) to its value
(3 answers)
I'm getting a TypeError. How do I fix it?
(2 answers)
Closed last month.
I have some data that I want to change(unixtime -> human date time).
After then, How can I update result on mysql dynamically.
>>> print data
((1424794931452.0,), (1424794931645.0,), (1424794931821.0,), (1424794932014.0,), (1424794932189.0,)
for i in data:
dt = datetime.fromtimestamp(i[0] // 1000)
s = dt.strftime('%Y-%m-%d %H:%M:%S')
sql2 = "UPDATE accelerometer SET test = "+ s +"WHERE _id="+i
cursor.execute(sql2)
data = cursor.fetchall()
print (s)
this is error message.
TypeError: cannot concatenate 'str' and 'tuple' objects
I want to update those below result data on Mysql dynamically. But have some problem. What is the problem?
2015-02-24 11:22:11
2015-02-24 11:22:11
2015-02-24 11:22:11
2015-02-24 11:22:12
2015-02-24 11:22:12

You are looping over tuples with an id, not a list of ids:
((1424794931452.0,), (1424794931645.0,), (1424794931821.0,), (1424794932014.0,), (1424794932189.0,)
So each i is set to one of those tuples. Extract the id, by using indexing or by adding a comma to the for loop assignment:
for i in data:
dt = datetime.fromtimestamp(i[0] // 1000)
s = dt.strftime('%Y-%m-%d %H:%M:%S')
sql2 = "UPDATE accelerometer SET test = " + s + "WHERE _id=" + i[0]
cursor.execute(sql2)
or
for i, in data:
dt = datetime.fromtimestamp(i // 1000)
s = dt.strftime('%Y-%m-%d %H:%M:%S')
sql2 = "UPDATE accelerometer SET test = " + s + "WHERE _id=" + i
cursor.execute(sql2)
You should really use SQL parameters instead of string concatenation here; you can then reuse the SQL statement:
sql2 = "UPDATE accelerometer SET test = ? WHERE _id=?"
for i, in data:
dt = datetime.fromtimestamp(i // 1000)
cursor.execute(sql2, (i, dt))
where I made two assumptions: that your database driver uses ? as the placeholder syntax (it could be %s instead) and that it natively supports datetime objects (most can these days), so you don't need to use datetime.strftime() to produce a string first.
The statement reuse can go further, in that the database only has to parse the query once, and only has to produce one query plan; this speeds up repeated executions.
Using SQL parameters has another very important advantage: it prevents SQL injection attacks (where an attacker adds quoting and additional SQL statements). For your specific inputs that is not so much of a concern, but should always be kept in mind.

Your core problem is that you're trying to put together a self-sufficient string (with string concatenation in your case, problems with string interpolation are more common, but it's pretty much the same issue).
Rather, use placeholders:
sql2 = "UPDATE accelerometer SET test = %s WHERE _id= %s"
then pass the data to execute:
cursor.execute(sql2, (s, i))
which will do the right thing for you.
It is, of course, possible, though tricky, to build the right stand-alone string -- but it's still very wrong to do so, see https://xkcd.com/327/ and don't expose yourself, ever!, to SQL-injection risks: use placeholders and data in execute, instead!

Your data is stored in tuples. And the error message says that strings and tuples can't be combined as a string, as a tuple isn't a string.
Instead of:
sql2 = "UPDATE accelerometer SET test = "+ s +"WHERE _id="+i
Try:
sql2 = "UPDATE accelerometer SET test = "+ s +"WHERE _id=%f" % i
I wouldn't recommend to use %s to build the string, as that could change the format of your floating point values.

database field value that matches to every query

I would like to insert records into a sqlite database with fields such that every query that specifies a value for that field does not disqualify the record.
Make Model Engine Parameter
Ford * * 1
Ford Taurus * 2
Ford Escape * 3
So a query = (database.table.Make == Ford') & (database.table.Model == 'Taurus') would return the first two records
EDIT: thanks to woot, I decided to use the following: (database.table.Make.belongs('Ford','')) & (database.table.Model.belongs('Taurus','')) which is the syntax for the IN operator in web2py

Are you looking for something like this? It won't perform well due to the ORs if you have a lot of rows.
SELECT *
FROM Cars
WHERE ( Cars.Make = 'Ford' OR Cars.Make = '*' )
AND ( Cars.Model = 'Taurus' OR Cars.Model = '*' )
Here is a SQL Fiddle example.
If you meant to use NULL, you can just replace that and replace the OR condition with OR Cars.Make IS NULL, etc.
Or to make it maybe a little less verbose:
SELECT *
FROM Cars
WHERE Cars.Make IN ('Ford','*')
AND Cars.Model IN ('Taurus','*')
But you wouldn't be able to use NULL in this case and would have to use the * token.
SQL Fiddle

Python + Mysql = Incorrect integer value:

The problem I have is to identify the type of data entering the database, I think everyone as IntegerField model in Django and Python code only inserts it all in one list and then insert it into the base data.
On the other hand I have no very clear writing Python code length according to the rules of the line, what I do is just see that in the end the code is very long as separated with spaces to align the next line below do not know if this good in this way and that the code will not fail.
The data that has to enter ip_unidad is ('186 .99.41.000 ', 3333) found in' self.addr [0] 'and the data date is '091211' which is in 'self.Cadenapura [17] '
and try "self.Cadenapura [17] = int (self.Cadenapura [17])" but nothing
It records the input data in the database but the two spaces are 0.
any ideas would be grateful.
The console error is:
Warning: Incorrect integer value: 'self.addr[0]' for column 'ip_unidad' at row 1
('self.addr[0]','self.Cadenapura[17]')
Warning:Incorrect integer value: 'self.Cadenapura[17]' for column 'fecha' at row 1
('self.addr[0]','self.Cadenapura[17]')
The code. Py used is:
sql = """INSERT INTO carro ( ip_unidad , hora ) VALUES (%s,%s)"""
db = MySQLdb.Connect(host="localhost", user="root",passwd="--------",db="gprslmgs")
cursor = db.cursor()
try :
cursor.execute(sql,('self.addr[0]','self.Cadenapura[17]'))
db.commit()
except:
db.rollback()
Django model used to create the database is:
class Carro(models.Model):
ip_unidad = models.IntegerField(max_length=15)
fecha = models.IntegerField(max_length=6)
Thank you.

I am not very familiar with Django but what i see is that you specify object within ' - think you don't need to do this. Have you tried something like:
cursor.execute(sql % (self.addr[0], self.Cadenapura[17]))
Or:
cursor.execute(sql, (self.addr[0], self.Cadenapura[17],))
While browsing i found the following MySQLdb sample code:
import MySQLdb
db = MySQLdb.connect(passwd="moonpie",db="thangs")
c = db.cursor()
c.executemany(
"""INSERT INTO breakfast (name, spam, eggs, sausage, price)
VALUES (%s, %s, %s, %s, %s)""",
[
("Spam and Sausage Lover's Plate", 5, 1, 8, 7.95 ),
("Not So Much Spam Plate", 3, 2, 0, 3.95 ),
("Don't Wany ANY SPAM! Plate", 0, 4, 3, 5.95 )
] )
So i think it should work the second way i mentioned.

I see two main problems - first is that '186.99.41.000' is not an integer, this is a string. Django provides an ip address field which is designed for this.
Your second problem is similar to the first, in that '09876' is a string, and your column type is IntegerField, however when you convert '09876' to an integer, you'll get 9876 because in Python, a number starting from 0 is an octal literal:
>>> print 127
127
>>> print 0177
127
>>> print 0x7f
127
So you need to store '09876' as a string in your database, and to do that you need to change your column type to CharField.

Like I said on your newer question, you should use Django's model api to handle the SQL for you.
Carro.objects.create(ip_unidad=self.addr[0], fecha=self.Cadenapura[17])
Furthermore, you should also revise your model. Instead of using IntegerFields, you should instead use
class Carro(models.Model):
ip_unidad = models.IPAddressField()
fecha = models.DateField()
Then when saving your data(with the model's objects.create), you need to make sure that your date is a python datetime.date object. Using the IPAddressField also means that you don't need to bother trying to convert the IP address to an int.