I have a select query like this :
SELECT * FROM simple_tbl WHERE a_name != " " AND d_name != " " AND s_time != "0000-00-00 00:00:00" AND e_time != "0000-00-00 00:00:00"
I have to use this in a python script, I used the select inside the cursor.execute("query"), but its failing everytime.
Can anyone help me using it ?
I have used below 2 ways:
1st:
cursor.execute ("SELECT * FROM maintenance_simple_tbl WHERE acc_name != " " AND device_name != " " AND start_time != '0000-00-00 00:00:00' AND end_time != '0000-00-00 00:00:00'")
Error:
Traceback (most recent call last):
File "sync.py", line 42, in <module>
cursor.execute ("SELECT * FROM simple_tbl WHERE a_name != " " AND d_name != " " AND s_time != '0000-00-00 00:00:00' AND e_time != '0000-00-00 00:00:00'")
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND d_name !=
ND s_time != '0000-00-00 00:00:00' AND e_time != '000' at line 1")
2nd:
cursor.execute ("SELECT * FROM simple_tbl WHERE a_name != " " AND d_name != " " AND start_time !='%s' AND e_time !='%s'" % (0000-00-00 00:00:00, 0000-00-00 00:00:00))
Error:
File "sync.py", line 43
cursor.execute ("SELECT * FROM simple_tbl WHERE a_name != " " AND d_name != " " AND s_time !='%s' AND e_time !='%s'" % (0000-00-00 00:00:00, 0000-00-00 00:00:00))
^
SyntaxError: invalid syntax
Here is how you can proceed to query MySQL in Python:
import MySQLdb as sql
dbParams = {
'host': # your database host,
'port': # your database port,
'user': # your database user,
'passwd': # your database password,
'db': # your database name
}
query = 'SELECT * FROM simple_tbl WHERE a_name != " " AND d_name != " " AND s_time != "0000-00-00 00:00:00" AND e_time != "0000-00-00 00:00:00"'
database = sql.connect(**dbParams) # initiate a connection to your database
cursor = database.cursor() # create a new cursor that you will use to query your database
cursor.execute(query) # this will execute your query
result = cursor.fetchall()
Instead of fetchall() function you could just use fetchone() if you want to retrieve the first returned row.
Although the other answer will work for executing a query and fetching the results, I think that the DictCursor object is more in line with what you would want when dealing with MySQL tables. You can use it as follows:
import MySQLdb
import MySQLdb.cursors
def get_db_cursor():
db = MySQLdb.connect(host="host",
user="user",
passwd="passwd",
db="db_name",
cursorclass=MySQLdb.cursors.DictCursor)
cur = db.cursor()
return cur
Now, you can use this function as follows to get a cursor:
cursor = get_db_cursor()
sql = "SELECT * simple_tbl WHERE a_name != ' ' AND d_name != ' ' AND s_time != '0000-00-00 00:00:00' AND e_time != '0000-00-00 00:00:00' " # Note that I replaced the double quotes with single quotes
cursor.execute(sql)
data = cursor.fetchall()
This returns a collection of dictionaries - one for each row in the resulting table. You can iterate over this collection as follows:
for row in data:
if len(row['a_name']) > 5:
# do something
Related
I've tried making an update to my sqlite3 table but it doesn't seem to work.
marks = "My long name here"
conn = sqlite3.connect('mydb.db')
cur=conn.cursor()
cur.execute("UPDATE '" + str(marks) +"' SET (ENG,KIS,MAT,BIO) = (-1,-1,-1,-1) WHERE (ENG,KIS,MAT,BIO) =('nan','nan','nan','nan')")
conn.commit()
conn.close()
I can't see nay error in my code.
You don't need 4 separate statements.
You can do it in 1 statement with CASE expressions:
UPDATE tablename
SET ENG = CASE WHEN ENG = 'nan' THEN -1 ELSE ENG END,
KIS = CASE WHEN KIS = 'nan' THEN -1 ELSE KIS END,
MAT = CASE WHEN MAT = 'nan' THEN -1 ELSE MAT END,
BIO = CASE WHEN BIO = 'nan' THEN -1 ELSE BIO END
WHERE 'nan' IN (ENG,KIS,MAT,BIO)
This worked, separating them.
marks = "My long name here"
conn = sqlite3.connect('mydb.db')
cur=conn.cursor()
cur.execute("UPDATE '" + str(marks) + "' SET ENG=-1 WHERE ENG='nan'")
cur.execute("UPDATE '" + str(marks) + "' SET KIS=-1 WHERE KIS='nan'")
cur.execute("UPDATE '" + str(marks) + "' SET MAT=-1 WHERE MAT='nan'")
cur.execute("UPDATE '" + str(marks) + "' SET BIO=-1 WHERE BIO='nan'")
conn.commit()
conn.close()
Rather than this.
marks = "My long name here"
conn = sqlite3.connect('mydb.db')
cur=conn.cursor()
cur.execute("UPDATE '" + str(marks) +"' SET (ENG,KIS,MAT,BIO) = (-1,-1,-1,-1) WHERE (ENG,KIS,MAT,BIO) =('nan','nan','nan','nan')")
conn.commit()
conn.close()
I need to write a program which can,firstly, for ip adresses of people who saw my campaign on google, and then give me detailed information about each of these persons.
I have all information in postgres database and use python 3.5
Here is my code:
def get_connection(cursor_factory=None):
conn_string_pg = "host= '" + host + "' dbname = '" + dbname + "' user = '" + user + \
"' password = '" + password + "'"
if cursor_factory is None:
conn_pg = psycopg2.connect(conn_string_pg)
else:
conn_pg = psycopg2.connect(conn_string_pg,
cursor_factory=cursor_factory)
return conn_pg
def find_logs():
select = """ select ip_address from log_files o where o.url like
'%my_campaign'
"""
conn = get_connection(cursor_factory = RealDictCursor)
cur = conn.cursor()
cur.execute(select)
records = cur.fetchone()
for item in records:
select_2 = "select * from log_files where ip_address = %(item)s "
cur.execute(select_2)
logs = cur.fetchone()
return logs
print(find_logs())
cur.close()
Unfortunately I get this error:
psycopg2.ProgrammingError: syntax error at or near "%" LINE 1:
...elect * from web_logs.log_data where ip_address = %(item)s o...
Your string interpolation is incorrect. You're trying to insert the value of item into your select_2 statement, but you don't actually do the string interpolation, so you pass psycopg2 an invalid SQL statement. You want to do something like
select_2 = "select * from log_files where ip_address = {}".format(item)
It's because ip_address = %(item)s it's not a valid sql syntax. You should make string formatting before:
select_2 = "select * from log_files where ip_address = %(item)s " % {'item': item}
And the better way to do it is to give all the transformation to the postgres driver
select_2 = "select * from log_files where ip_address = %s "
cur.execute(select_2, (item, ))
So I have the following error:
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'I, Alexander_Bernard16#milton.edu, D0NBT6)' at line 1")
Here's my code:
cnx = MySQLdb.connect(
user=username, passwd=password, host=hostname, db=databaseName)
cursor = cnx.cursor()
cursor.execute("CREATE TABLE if not exists gotchaTable(id int(11) PRIMARY KEY "
"AUTO_INCREMENT, selfFirstName TEXT NOT NULL, selfLastName TEXT NOT NULL, "
"selfGrade TEXT NOT NULL, selfCode TEXT NOT NULL, targetCode TEXT NOT "
"NULL);")
cnx.commit()
add_data = (
"INSERT INTO gotchaTable (selfFirstName, selfLastName, selfGrade, selfCode, targetCode) VALUES ({0}, {1}, {2}, {3}, {4});"
)
studentlist = []
with open('Gotcha.csv', 'rb') as csvfile:
gotchaData = csv.DictReader(csvfile)
for row in gotchaData:
student = Student(
row['First'], row['Last'], row['Class'], row['Email'])
studentlist.append(student)
studentlist = randomList(studentlist)
for x in xrange(1, len(studentlist)):
studentlist[x].target = studentlist[
x + 1] if x < len(studentlist) - 1 else studentlist[0]
cursor.execute(add_data.format(studentlist[x].first, studentlist[x].last,
studentlist[x].grade, studentlist[x].email,
studentlist[x].code, studentlist[x].target.code))
cnx.commit()
print studentlist[x].getData()
And here's my student class:
class Student(object):
"""docstring for Student"""
def __init__(self, f, l, c, e):
self.first = f
self.last = l
self.grade = c
self.email = e
self.code = id_generator()
self.target = None
def getData(self):
return self.first + ' ' + self.last + ' ' + self.grade + ' ' + self.email + ' ' + self.code
Im trying to make a program that gets data from a csv (which already works) and puts it into a SQL table. How do i fix the error 1064, i've tried using "%s" instead of '{0}' but i get the same error. Any suggestions?
the id_generator() method returns a string of random characters.
randomList(a) makes the array random.
Don't use string formatting to parameterize an SQL query - this is dangerous and, as you can see, error-prompt. Instead, let the MySQL driver worry about it:
add_data = """
INSERT INTO
gotchaTable
(selfFirstName, selfLastName, selfGrade, selfCode, targetCode)
VALUES
(%s, %s, %s, %s, %s)
"""
Then, when you call execute() pass parameters in a separate argument:
cursor.execute(add_data, [
studentlist[x].first,
studentlist[x].last,
studentlist[x].grade,
# studentlist[x].email, ALSO WATCH THIS ONE (there are only 5 placeholders in the query)
studentlist[x].code,
studentlist[x].target.code
])
I'm trying to get the following Python MYSQL update statement correct(With Variables):
try:
connection = mysql.connector.connect\
(host = "localhost", user = "root", passwd ="", db = "crawling")
except:
print("Keine Verbindung zum Server")
sys.exit(0)
cursor = connection.cursor()
cursor.execute("TRUNCATE meta;")
connection.commit()
cursor.execute("ALTER TABLE meta AUTO_INCREMENT =1;")
connection.commit()
for j in range(1, int(outerElements)):
for i in range(1, int(innerElements)):
partner_ID = 6
location_ID = 20
headline = driver.find_element_by_xpath("//div[#id='productList']/div["+str(j)+"]/div["+str(i)+"]/div/div[2]/h2/a").text
price = driver.find_element_by_xpath("//div[#id='productList']/div["+str(j)+"]/div["+str(i)+"]/div/div[2]/div[2]/span[2]").text[:-1]
deeplink = driver.find_element_by_xpath("//div[#id='productList']/div["+str(j)+"]/div["+str(i)+"]/div/div[2]/h2/a").get_attribute("href")
print("Header: " + headline + " | " + "Price: " + price + " | " + "Deeplink: " + deeplink + " | " + "PartnerID: " + str(partner_ID) + " | " + "LocationID: " + str(location_ID))
cursor.execute('''UPDATE meta SET (price_id, Header, Price, Deeplink, PartnerID, LocationID) \
VALUES(%s, %s, %s, %s, %s, %s)''', ['None'] + [headline] + [price] + [deeplink] + [partner_ID] + [location_ID])
connection.commit()
cursor.close()
connection.close()
When conducting the code, I´m getting the following error message:
Traceback (most recent call last):
File "C:/Users/hmattu/PycharmProjects/untitled1/localhost_crawl.py", line 97, in test_sel
VALUES(%s, %s, %s, %s, %s, %s)''', ['None'] + [headline] + [price] + [deeplink] + [partner_ID] + [location_ID])
File "C:\Python34\lib\site-packages\mysql\connector\cursor.py", line 507, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Python34\lib\site-packages\mysql\connector\connection.py", line 722, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Python34\lib\site-packages\mysql\connector\connection.py", line 640, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '= (price_id, Header, Price, Deeplink, PartnerID, LocationID)
Can anybody help me out? Any feedback is appreciated.
Try to change
UPDATE meta SET (price_id, Header, Price, Deeplink, PartnerID, LocationID)
for
UPDATE meta SET (`price_id`, `Header`, `Price`, `Deeplink`, `PartnerID`, `LocationID`)
I am moving data from Mysql to Postgres and my code is like below -
import os, re, time, codecs, glob, sqlite3
from StringIO import StringIO
import psycopg2, MySQLdb, datetime, decimal
from datetime import date
import gc
tables = (['table1' , 27],)
conn = psycopg2.connect("dbname='xxx' user='xxx' host='localhost' password='xxx' ")
curpost = conn.cursor()
db = MySQLdb.connect(host="127.0.0.1", user="root", passwd="root" , unix_socket='/var/mysql/mysql.sock', port=3306 )
cur = db.cursor()
cur.execute('use xxx;')
for t in tables:
print t
curpost.execute( "truncate table " + t[0] )
cur.execute("select * from "+ t[0] )
a = ','.join( '%s' for i in range(t[1]) )
qry = "insert into " + t[0] + " values ( " + a +" )"
print qry
i = 0
while True:
rows = cur.fetchmany(5000)
if not rows: break
string = ''
for row in rows:
string = string + ('|'.join([str(x) for x in row])) + "\n"
curpost.copy_from(StringIO(string), t[0], sep="|", null="None" )
i += curpost.rowcount
print i , " loaded"
curpost.connection.commit()
del string, row, rows
gc.collect()
curpost.close()
cur.close()
For small tables, the code runs fine. However the larger ones (3.6 million records), the moment the mysql execute (cur.execute("select * from "+ t[0] )) runs, the memory utilization on the machine zooms. This is even though i have used fetchmany and records should only come in batches of 5000. I have tried with 500 records also and its the same. For large tables it seems that fetchmany is not working as documented..
Edit - I added garbage collection and del statements. Still the memory keeps on bloating till all records are not processed.
Any ideas?
Sorry if I am wrong, you've said that you don't want to change query
But just in case if you have no choice you can try:
replace this fragment:
cur.execute("select * from "+ t[0] )
a = ','.join( '%s' for i in range(t[1]) )
qry = "insert into " + t[0] + " values ( " + a +" )"
print qry
i = 0
while True:
rows = cur.fetchmany(5000)
to this one:
a = ','.join( '%s' for i in range(t[1]) )
qry = "insert into " + t[0] + " values ( " + a +" )"
print qry
i = 0
while True:
cur.execute("select * from "+ t[0]+" LIMIT "+i+", 5000")
rows = cur.fetchall()