Unique Keys and Python Dictionary - python

Here is the code for the relevant function:
def populateSubscribers(csvfile, db):
conn = sqlite3.connect(db)
conn.text_factory = str #bugger 8-bit bytestrings
cur = conn.cursor()
subscriber_dict = {}
# read values from tab-delimited csv file
reader = csv.reader(open(csvfile, "rU"), delimiter = '\t')
for Number, Name, Message, Datetime, Type in reader:
if str(Number)[:1] == '1':
tmpNumber = str(Number)[1:]
Number = int(tmpNumber)
# check to ensure name/number not null
if Number and Name:
# add unique subscribers to dictionary
subscriber_dict[Number] = Name
else:
print 'Subscriber missing name or number'
# insert unique subscribers into subscriber table
for number, name in subscriber_dict.items():
cur.execute('INSERT OR IGNORE INTO subscriber (name, phone_number) VALUES (?,?)', (name, number))
conn.commit()
cur.close()
conn.close()
print '...Successfully populated subcriber table.'
It is reading subscriber names and phone numbers from a csv file and then it is supposed to write an entry for each unique subscriber into the database. I wanted the phone number to be the key in key/value pairs since it is unique. But for some reason, it is not reading all of the numbers from the data, it is missing some subscribers. If I make the name the key it misses as would be anticipated (de-duplicates all the Unknowns), but the phone number as key is missing some numbers altogether. Any ideas on how to fix the logic here?

Looks to me like if str(Number)[:1] == '1': is probably filtering out some of your data.
Add an else to that and print out any it's rejecting. Those are probably the ones which are going wrong.
Either way, pare down your input data and find which one's aren't being used. Without seeing the data and the alternative you've said which does work, it's hard to pin down the exact cause here.

Related

How do I allow placeholders to be empty in SQLite3 in Python?

I am writing a "Sales Order" database right now and I am supposed to implement a search function that allows the user to enter in data like, customer first name, last name, order ID, customer ID, etc. The problem I am having is that the customer does not have to enter anything for the fields and instead just press enter to leave it blank to skip. But SQL seems to be taking that as an empty input and looking for an empty field. How can I make it so that the program instead allows anything to be accepted by the placeholder rather than looking for a blank. Thanks!
initialSearchInformation = [input("Enter Customer Last Name (To skip, press ENTER)"),
input("Enter Customer Email (To skip, press ENTER)"),
input("Enter Customer Number (To skip, press ENTER)"),
input("Enter Order Number (To skip, press ENTER)"),
input("Enter Order Amount Greater Than (To skip, press ENTER)"),
input("Enter Order Amount Less Than (To skip, press ENTER)")]
conn = create_connection()
cur = conn.cursor()
cur.execute("""SELECT *
FROM SalesOrder
INNER JOIN Customer
ON Customer.CustomerID = SalesOrder.CustomerID
WHERE SalesOrderID=?
AND SalesOrder.CustomerID=?
AND Amount > ?
AND Amount < ?
AND Customer.CustomerName LIKE ?""", (initialSearchInformation[3], initialSearchInformation[2], initialSearchInformation[4], initialSearchInformation[5], '%'+initialSearchInformation[0]+'%',))
rows = cur.fetchall()
print("\nInfo Found: ")
for row in rows:
print(row)
Check for an empty value in your SQL. E.g. replace
SalesOrderId = ?
with
(? = '' OR SalesOrderID = ?)
Since there are now two placeholders, you'll need to duplicate the corresponding element in the parameter tuple:
initialSearchInformation[3], initialSearchInformation[3]
The full query becomes
cur.execute("""SELECT *
FROM SalesOrder
INNER JOIN Customer ON Customer.CustomerID = SalesOrder.CustomerID
WHERE (? = '' OR SalesOrderID=?)
AND (? = '' OR SalesOrder.CustomerID=?)
AND (? = '' OR Amount > ?)
AND (? = '' OR Amount < ?)
AND (? = '' OR Customer.CustomerName LIKE ?)""",
(initialSearchInformation[3], initialSearchInformation[3],
initialSearchInformation[2], initialSearchInformation[2],
initialSearchInformation[4], initialSearchInformation[4],
initialSearchInformation[5], initialSearchInformation[5],
initialSearchInformation[0], '%'+initialSearchInformation[0]+'%',))

Python SQLite :IndexError: list index out of range

conn = database_connect()
if(conn is None):
return None
cur = conn.cursor()
try:
# Try executing the SQL and get from the database
sql = """SELECT *
FROM user
WHERE user_id**strong text** =%s AND password =%s"""
cur.execute(sql, (employee_id, password))
r = cur.fetchone()# Fetch the first row
rr = cur.fetchall()
cur.close() # Close the cursor
conn.close() # Close the connection to the db
except:
# If there were any errors, return a NULL row printing an error to the debug
print("Error Invalid Login")
cur.close() # Close the cursor
conn.close() # Close the connection to the db
return None
user_info = []
if rr is None:
print("worry")
return []
for n in rr:
user_info.append(n)
test = {
'info1': user_info[0],
'info2': user_info[1],
'info3': user_info[2],
'info4': user_info[3],
}
return test
Here is my code. First Implement the login function, and then get the user information, but there is a IndexError: list index out of range. How do I fix this?
Here:
r = cur.fetchone()# Fetch the first row
rr = cur.fetchall()
The call to fetchone() will consume the first row, so rr will contain n-1 rows.
Also, if your database allow duplicated user_id then you have a serious db design issue - whether user_id is (suppoed to be) the primary key or the "username" (loging name), it should really be unique. If it isn't, then you want to change your schema, and if it is indeed unique then obviously your query can only return (at most !) one single row, in which case rr is garanteed to always be empty (since you consumed the first row with the fetchone() call.
As a side note, here are some possible improvements:
this :
for n in rr:
user_info.append(n)
is completely useless - you are just building a shallow copy of rr, wo just work directly with rr instead.
Then do not assume you will always have four rows in your query result, so at least build your test dict dynamically:
test = {}
# enumerate(seq) yields an index, value tuple
# for each value in `seq`.
for num, row in enumerate(rr, 1):
key = "info{}.format(num)
test[key] = row
or more succintly:
test = {"info{}".format(num):row for num, row in enumerate(rr, 1)}
Note that this dict doesn't add much value compared to rr - you have "info" keys instead of numeric indexes and that's about all, so you could just directly use the rr list instead.
Last but not least, your try/except clause is too broad (too much code in the try block), the except clause will eat very valuable debugging information (the exact error message and the full traceback) and even worse, the error message you display assumes way too much about what really happens. Actually you probably shouldn't even have an except clause here (at least not until you really know what errors can happen here AND be properly handled at this point) so better to let the error propagate (so you get the full error message and traceback), and use the finally clause to close your connection instead:
sql = """SELECT *
FROM user
WHERE user_id**strong text** =%s AND password =%s"""
try:
cur.execute(sql, (employee_id, password))
r = cur.fetchone()# Fetch the first row
rr = cur.fetchall()
finally:
# this will always be executed whatever happens in the try block
cur.close() # Close the cursor
conn.close() # Close the connection to the db

perform validation or raise exception for incorrect user input in mysql python

I'm trying to perform validation in Crud operations in MySQL. My table has name,idno,division and I have wrote the MySQL output into a csv file. I'm giving a user input and if the user input matches with the idno in table then it should print the entire row.
This works fine when I give the idno in the table but it doesn't raises exception when an incorrect value is given as input. Can anyone help me with raising exception or performing validation?
Here are my codes:
def search (self):
try:
with open("test.csv", 'rb') as inputfile:
reader = csv.DictReader(inputfile)
user_input=int(raw_input("Enter the Idno to search:"))
rows = [row for row in reader if row['Idno']==str(int(user_input)
for row in rows:
print rows
except ValueError:
print "Enter correct idno "
search()
Here, to validate your user input from raw_input you can consider two ways
- checking your value with regular expression
- Checking with the actual database idno.
for example, before fetching actual data from database, check whether given idno is available or not using select query
Query: select idno from <table_name>
and check for the existence. If not available, you can raise value error
exception : raise ValueError
I take it that this line posted in your question
rows = rows = [row for row in reader if row['Idno'] == str(int(user_input)
should really be
rows = [row for row in reader if row['Idno'] == str(int(user_input))]
You are checking this for a ValueError. Your exception will trap situations where the user inputs a value that is not an integer. But I take it that for an id number to be valid, it must also appear in your .csv file. Your code doesn't test for that. You need something like
rows = [row for row in reader if row['Idno'] == str(int(user_input))]
if not rows:
raise ValueError("idno %r not found")
Or if, as your comments suggest, you have an objection to checking if the list is empty:
rows = [row for row in reader if row['Idno'] == str(int(user_input))]
if user_input not in rows:
raise ValueError("idno %r not found")
Tried to perform validation using MySQL db and it works. But I dont know what should replace if result==() to make it more meaningful and perfect as () None doesn't works for me.
def search(self):
user_input=raw_input('Enter idno:')
cursor.execute("select * from table where Idno='{0}';".format(user_input))
result=cursor.fetchall()
if result==():
print "User_input didnt match"
else:
print result
search()

Cannot validate query results

query = "SELECT serialno from registeredpcs where ipaddress = "
usercheck = query + "'%s'" %fromIP
#print("query"+"-"+usercheck)
print(usercheck)
rs = cursor.execute(usercheck)
print(rs)
row = rs
#print(row)
#rs = cursor.rowcount()
if int(row) == 1:
query = "SELECT report1 from registeredpcs where serialno = "
firstreport = query + "'%s'" %rs
result = cursor.execute(firstreport)
print(result)
elif int(row) == 0:
query_new = "SELECT * from registeredpcs"
cursor.execute(query_new)
newrow = cursor.rowcount()+1
print(new row)
What I am trying to do here is fetch the serialno values from the db when it matches a certain ipaddress. This query if working fine. As it should the query result set rs is 0. Now I am trying to use that value and do something else in the if else construct. Basically I am trying to check for unique values in the db based on the ipaddress value. But I am getting this error
error: uncaptured python exception, closing channel smtpd.SMTPChannel connected
192.168.1.2:3630 at 0x2e47c10 (**class 'TypeError':'int' object is not
callable** [C:\Python34\lib\asyncore.py|read|83]
[C:\Python34\lib\asyncore.py|handle_read_event|442]
[C:\Python34\lib\asynchat.py|handle_read|171]
[C:\Python34\lib\smtpd.py|found_terminator|342] [C:/Users/Dev-
P/PycharmProjects/CR Server Local/LRS|process_message|43])
I know I am making some very basic mistake. I think it's the part in bold thats causing the error. But just can't put my finger on to it. I tried using the rowcount() method didn't help.
rowcount is an attribute, not a method; you shouldn't call it.
"I know I am making some very basic mistake" : well, Daniel Roseman alreay adressed the cause of your main error, but there are a couple other mistakes in your code:
query = "SELECT serialno from registeredpcs where ipaddress = "
usercheck = query + "'%s'" % fromIP
rs = cursor.execute(usercheck)
This part is hard to read (you're using both string concatenation and string formatting for no good reason), brittle (try this with `fromIP = "'foo'"), and very very unsafe. You want to use paramerized queries instead, ie:
# nb check your exact db-api module for the correct placeholder,
# MySQLdb uses '%s' but some other use '?' instead
query = "SELECT serialno from registeredpcs where ipaddress=%s"
params = [fromIP,]
rs = cursor.execute(query, params)
"As it should the query result set rs is 0"
This is actually plain wrong. cursor.execute() returns the number of rows affected (selected, created, updated, deleted) by the query. The "resultset" is really the cursor itself. You can fetch results using cursor.fetchone(), cursor.fetall(), or more simply (and more efficiently if you want to work on the whole resultset with constant memory use) by iterating over the cursor, ie:
for row in cursor:
print row
Let's continue with your code:
row = rs
if int(row) == 1:
# ...
elif int(row) == 0:
# ...
The first line is useless - it only makes row an alias of rs, and badly named - it's not a "row" (one line of results from your query), it's an int. Since it's already an int, converting it to int is also useless. And finally, unless 'ipadress' is a unique key in your table, your query might return more than one row.
If what you want is the effective value(s) for the serialno field for records matching fromIP, you have to fetch the row(s):
row = cursor.fetchone() # first row, as a tuple
then get the value, which in this case will be the first item in row:
serialno = row[0]

Python SQL select row from specific variable field

I'd like to grab a specific value from a row based on a random variable. Here's an example table the PID column is an "auto-increment primary key integer" and the other 2 columns are TEXT
example-table
PID NAME PHONE
--- ---- -----
1 bill 999-9999
2 joe 888-8888
I'd like to throw a random variable at the table
randomVariable = raw_input('Enter something: ')
> 1
and have the code return the name
> bill
I know I can use something like...
randomVariable = raw_input('Enter something: ')
sql = ("SELECT name FROM example_table WHERE pid='%s'" % randomVariable)
result = cursor.execute(sql)
print result
> bill
Apparently using '%s' isn't secure and it is suggested to use '?' in it's place.
randomVariable = raw_input('Enter something: ')
sql = ("SELECT name FROM example_table WHERE pid=?", randomVariable)
result = cursor.execute(sql)
print result
But this doesn't seem to work for me. I end up with...
"ValueError: operation parameter must be str or unicode"
I realize I could just grab all the rows and put them into a variable which I could then iterate over till I find what I'm looking for but I'm thinking that wouldn't be very efficient with a large database. can anyone help point me in the right direction with this?
I believe you're meant to use it like this
randomVariable = raw_input('Enter something: ')
sql = "SELECT name FROM example_table WHERE pid=?"
result = cursor.execute(sql, randomVariable)
print result
Validate the user input, and %s is fine. Storing your rows and putting them into a list is not a good idea at all, since the amount of rows will grow over time, taking up a huge amount of memory when not even in use. To guard against SQL injection, you could validate input using something like a typecast to an int, put that in a try/except block, and this would stop all malicious input such as ' OR 1=1--

Categories