We Keep Coding

I can't get Python to work with mySQL

Okay, so I'll preface with, 'I am new to Python'. I am using PyCharm to run this programme I am building to dump to a Database (locally hosted mySQL).
Unfortunately I am running into problems getting data into the database with Python, but I am able to insert data in SQL running through the phpMyAdmin web gui. I must be missing something obvious. I am using 'mysql.connector' addon for PyCharm. Here is the code relevant.
So, at the start I import the module? like so;
import mysql.connector
And this code is referenced inside a couple while loops;
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
c = cnx.cursor()
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
cnx.commit()
Please dont eat me alive either for the way I write, I am new :)
Other important factors;
I was able to get it to enter data a couple times (but it came through to SQL as a NULL value. My guess is I was passing the wrong format of number? or something to do with the db Collation?)
I have also tried this with a number of different addons for PyCharm (All of them seem to be implemented in similar ways).
I have tried it with and without quote marks on the variable names being inserted, I have tried so much, I am at my wits end.
As for the error codes, this is what it looks like currently;
C:\Users\logge\PycharmProjects\untitled3\venv\Scripts\python.exe C:/Users/logge/PycharmProjects/untitled3/main.py
Traceback (most recent call last):
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 111, in <module>
weather(1)
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 101, in weather
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\cursor.py", line 561, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 525, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 427, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir' at line 1
Process finished with exit code 1
The 'line 101' refered to in the error codes is talking about the line starting with 'c.execute... INSERT' in the second section of code I posted.
Here is the code I have also tried, just to see if I could get any results (I still get the same errors though):
from __future__ import print_function
from datetime import date, datetime, timedelta
import mysql.connector
dtime1 = 1
tmp1 = 5
apptmp1 = 2
dewpoint1 = 64
relhum1 = 3
delta_t1 = 4
wind_dir1 = 5
wind_spd_kmh1 = 6
wind_gust_kmh1 = 7
wind_spd_kts1 = 8
wind_gust_kts1 = 9
press_qnh1 = 10
press_msl1 = 11
rainsince9am1 =12
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
cursor = cnx.cursor()
tomorrow = datetime.now().date() + timedelta(days=1)
add_employee = ("INSERT INTO stations "
"(dtime, tmp, apptmp, dewpoint, relhum, delta_t, wind_dir, wind_spd_kmh, wind_gust_kmh, wind_spd_kts, wind_gust_kts, press_qnh, press_msl, rainsince9am) "
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,)")
data_employee = (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)
# Insert new employee
cursor.execute(add_employee, data_employee)
emp_no = cursor.lastrowid
# Make sure data is committed to the database
cnx.commit()
cursor.close()
cnx.close()
Any help or suggestions would be greatly appreciated. I am happy to post the entire code if required also. I just didn't want to clutter this whole thing up.

I found your error. You have referenced the input data to be strings (%s). But you are putting in integers (numbers).
If you want to put the numbers as strings, in the first chunk of code (yours), you need to put in str(variable) instead of just variable. In the second chunk of code (copied from MySQL), you can declare the variables as variable = "0000" with the quotes.
If you want to work with numbers in your code, change the table structure to be INTEGER or DOUBLE instead of VARCHAR or CHAR.
I would assume that the table is present and all fields are present as there is no error connecting to the database and finding the table.

python & MySQLdb TypeError('not all arguments converted during string formatting',) [duplicate]

Upon running this script:
#! /usr/bin/env python
import MySQLdb as mdb
import sys
class Test:
def check(self, search):
try:
con = mdb.connect('localhost', 'root', 'password', 'recordsdb');
cur = con.cursor()
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
ver = cur.fetchone()
print "Output : %s " % ver
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
finally:
if con:
con.close()
test = Test()
test.check("test")
I get an error of:
./lookup
Traceback (most recent call last):
File "./lookup", line 27, in <module>
test.check("test")
File "./lookup", line 11, in creep
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not all arguments converted during string formatting
I have zero idea why. I'm trying to do parameterized querys, but it's been nothing but a pain. I'm somewhat new to Python, so it's probably an obvious problem.

Instead of this:
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
Try this:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", [search] )
See the MySQLdb documentation. The reasoning is that execute's second parameter represents a list of the objects to be converted, because you could have an arbitrary number of objects in a parameterized query. In this case, you have only one, but it still needs to be an iterable (a tuple instead of a list would also be fine).

You can try this code:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
You can see the documentation

'%' keyword is so dangerous because it major cause of 'SQL INJECTION ATTACK'.
So you just using this code.
cursor.execute("select * from table where example=%s", (example,))
or
t = (example,)
cursor.execute("select * from table where example=%s", t)
if you want to try insert into table, try this.
name = 'ksg'
age = 19
sex = 'male'
t = (name, age, sex)
cursor.execute("insert into table values(%s,%d,%s)", t)

cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
I do not why, but this works for me . rather than use '%s'.

The accepted answer by #kevinsa5 is correct, but you might be thinking "I swear this code used to work and now it doesn't," and you would be right.
There was an API change in the MySQLdb library between 1.2.3 and 1.2.5. The 1.2.3 versions supported
cursor.execute("SELECT * FROM foo WHERE bar = %s", 'baz')
but the 1.2.5 versions require
cursor.execute("SELECT * FROM foo WHERE bar = %s", ['baz'])
as the other answers state. I can't find the change in the changelogs, and it's possible the earlier behavior was considered a bug.
The Ubuntu 14.04 repository has python-mysqldb 1.2.3, but Ubuntu 16.04 and later have python-mysqldb 1.3.7+.
If you're dealing with a legacy codebase that requires the old behavior but your platform is a newish Ubuntu, install MySQLdb from PyPI instead:
$ pip install MySQL-python==1.2.3

I don't understand the first two answers. I think they must be version-dependent. I cannot reproduce them on MySQLdb 1.2.3, which comes with Ubuntu 14.04LTS. Let's try them. First, we verify that MySQL doesn't accept double-apostrophes:
mysql> select * from methods limit 1;
+----------+--------------------+------------+
| MethodID | MethodDescription | MethodLink |
+----------+--------------------+------------+
| 32 | Autonomous Sensing | NULL |
+----------+--------------------+------------+
1 row in set (0.01 sec)
mysql> select * from methods where MethodID = ''32'';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '9999'' ' at line 1
Nope. Let's try the example that Mandatory posted using the query constructor inside /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py where I opened "con" as a connection to my database.
>>> search = "test"
>>> "SELECT * FROM records WHERE email LIKE '%s'" % con.literal(search)
"SELECT * FROM records WHERE email LIKE ''test''"
>>>
Nope, the double apostrophes cause it to fail. Let's try Mike Graham's first comment, where he suggests leaving off the apostrophes quoting the %s:
>>> "SELECT * FROM records WHERE email LIKE %s" % con.literal(search)
"SELECT * FROM records WHERE email LIKE 'test'"
>>>
Yep, that will work, but Mike's second comment and the documentation says that the argument to execute (processed by con.literal) must be a tuple (search,) or a list [search]. You can try them, but you'll find no difference from the output above.
The best answer is ksg97031's.

According PEP8，I prefer to execute SQL in this way:
cur = con.cursor()
# There is no need to add single-quota to the surrounding of `%s`,
# because the MySQLdb precompile the sql according to the scheme type
# of each argument in the arguments list.
sql = "SELECT * FROM records WHERE email LIKE %s;"
args = [search, ]
cur.execute(sql, args)
In this way, you will recognize that the second argument args of execute method must be a list of arguments.
May this helps you.

I encountered this error while executing
SELECT * FROM table;
I traced the error to cursor.py line 195.
if args is not None:
if isinstance(args, dict):
nargs = {}
for key, item in args.items():
if isinstance(key, unicode):
key = key.encode(db.encoding)
nargs[key] = db.literal(item)
args = nargs
else:
args = tuple(map(db.literal, args))
try:
query = query % args
except TypeError as m:
raise ProgrammingError(str(m))
Given that I am entering any extra parameters, I got rid of all of "if args ..." branch. Now it works.

Live output from Python database query

I have a piece of code looking like this:
conn=sybpydb.connect(user=args.u, password=args.p,servername=args.s)
cur = conn.cursor()
cur.execute(sql)
print(cur.connections.messages)
The execute takes maybe 5 minutes to run, and then it prints my output.
I was wondering if there is any way to print the output line for line while the execute is running instead of waiting until it is done and getting it all as one big batch?

filmor is right : retrieve results running on the cursor.
The sybpydb package is a PEP 249 implementation provided by Sybase. So, the Cursor object you retrieve have some methods, for example fetchone and fetchall.
Your code, to retrieve results line by line and do something with it, would be:
# Preparation your connection and cursor for fetching results
conn = sybpydb.connect(user=args.u, password=args.p, servername=args.s)
cur = conn.cursor()
cur.execute(sql)
# Reading all result one by one
for row in cur.fetchall():
# Do something with your current results
print(row)
# Closing cursor and connection
cur.close()
conn.close()

Any examples of SQLalchemy 0.7, UPDATE using from_statement()

I'm writing a quick one-off migration script that updates a single field in a table with half a million rows.
Since I hadn't planned on writing out full models for the joins I'm doing to fetch the initial ~25000 rows of data, I've been trying to figure out how to do an UPDATE statement using a from_statement() call and using my own raw sql, but I can't find any examples.
Along with that, SQLalchemy is throwing an error. Here's an example of my call and error:
mydb = self.session()
mydb.query().from_statement(
"""
UPDATE my_table
SET settings=mysettings
WHERE user_id=myuserid AND setting_id=123
""").params(mysettings=new_settings, myuserid=user_id).all()
The error I get:
Traceback (most recent call last):
File "./sample_script.py", line 111, in <module>
main()
File "./sample_script.py", line 108, in main
migrate.set_migration_data()
File "./sample_script.py", line 100, in set_migration_data
""").params(mysettings=new_settings, myuserid=user_id).all()
File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 1267, in all
return list(self)
File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 1361, in __iter__
return self._execute_and_instances(context)
File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 1364, in _execute_and_instances
result = self.session.execute(querycontext.statement, params=self._params, mapper=self._mapper_zero_or_none())
File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 251, in _mapper_zero_or_none
if not getattr(self._entities[0], 'primary_entity', False):
IndexError: list index out of range
UPDATE
I'm using MySQL.
Per Samy's suggestion, I tried this:
mydb.execute(
"UPDATE mytable SET settings=:mysettings WHERE user_id=:userid AND setting_id=123",
{'userid': user_id, 'mysettings': new_settings}
)
This had no effect. I don't get any errors, but the statement doesn't seem to actually execute, as the row does not change. If I manually cut and paste the query that gets logged from the echo=True option, the row updates in the database just fine.
UPDATE - SOLVED
Samy's suggestion was correct but the .execute() call only works on 'engine', not 'session', so this worked just fine:
self.engine.execute(
"UPDATE mytable SET settings=:mysettings WHERE user_id=:userid AND setting_id=123",
{'userid': user_id, 'mysettings': new_settings}
)

Well this is rather strange, according to the docs, the from_statement is used for SELECT statements.
Execute the given SELECT statement and return results.
I could be looking at the wrong function, or it may be possible to use other type of statements, Im not really sure.
You could just use execute since it can do any type of statement, heres a quick example.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
session = sessionmaker(bind = create_engine('sqlite://'), autocommit = True)()
_ = session.execute('CREATE TABLE my_table (user_id int, setting_id int, settings string)')
for id in xrange(200):
_ = session.execute('INSERT INTO my_table (user_id, setting_id) VALUES (:user_id, :setting_id)',
{'user_id':id, 'setting_id':id})
_ = session.execute(
"""
UPDATE my_table
SET settings = :mysettings
WHERE user_id = :user_id AND setting_id = 123
""", {'user_id':123, 'mysettings':'test'})
r = session.execute('SELECT * FROM my_table WHERE user_id = :user_id', {'user_id':123}).fetchall()
print r
[(123, 123, u'test')]
note that this isn't really the best way to use sqlalchemy, which was designed to create a dry environment, decoupled from a specific db backend, though you probably have your reasons for using raw sql versus its ORM.

You need to use the proper parameter syntax; the format depends entirely on your database adapter. For example, some adapters support :name paramaters, in which case you are missing those colons in your query:
mydb.query().from_statement(
"""
UPDATE my_table
SET settings=:mysettings
WHERE user_id=:myuserid AND setting_id=123
""").params(mysettings=new_settings, myuserid=user_id).all()
The DBAPI 2.0 spec supports several formats, including positional parametrs with ? and %s placeholders, and named parameters in the above form and as %(name)s formatting. You need to review your database adapter documentation to find out what is supported.

To insert to Pg by Psycopg

How can you fix the SQL-statement in Python?
The db connection works. However, cur.execute returns none which is false.
My code
import os, pg, sys, re, psycopg2
try:
conn = psycopg2.connect("dbname='tk' host='localhost' port='5432' user='naa' password='123'")
except: print "unable to connect to db"
cur = conn.cursor()
print cur.execute("SELECT * FROM courses") # problem here
The SQL-command in Psql returns me the correct output.
I can similarly run INSERT in Psql, but not by Python's scripts.
I get no warning/error to /var/log.
Possible bugs are
cursor(), seems to be right however
the syntax of the method connect(), seems to be ok however

You have to call one of the fetch methods on cur (fetchone, fetchmany, fetchall) to actually get the results of the query.
You should probably have a read through the a tutorial for DB-API.

You have to call cur.fetchall() method (or one of other fetch*() methods) to get results from query.