I am trying to iterate a table, fetch the rows in which a field has a pattern, then update the same row with a match group.
The following code runs without error, the two print lines before update clause output correct values. I have followed similar answers to come up the update clause, the logic seems right to me. However the code does not work, i.e., no rows updated. Where did I do wrong? Thanks,
# -*- coding: utf-8 -*-
import re
import MySQLdb
pattern = re.compile('#(.*)#.*$')
conn = MySQLdb.connect(
host='localhost', user='root',
passwd='password', db='j314', charset='utf8')
cursor = conn.cursor()
cursor.execute(
"""select `id`, `created_by_alias` from w0z9v_content where `catid` = 13 AND `created_by_alias` regexp "^#.*#.*$" limit 400""")
aliases = cursor.fetchall()
for alias in aliases:
newalias = pattern.match(alias[1])
if newalias.group(1) is not None:
# print alias[0]
# print newalias.group(1)
cursor.execute("""
update w0z9v_content set created_by_alias = %s where id = %s""", (newalias.group(1), alias[0]))
conn.close
autocommit is probably globally disabled on the server.
Execute either COMMIT after your updates, or SET autocommit=1 at the beginning of the session.
http://dev.mysql.com/doc/refman/5.0/en/commit.html
Also, you're not actually closing the connection, you forgot to call close:
conn.close()
Related
I'm trying to use sqlite3 in python to delete a selection of rows from a table. My attempt fails, but I can't work out why.
The sql query works ok, but I can't implement it within the python code.
I have a set of records that are moved from current_table to archive_table after a period of time.
I'd like to clean up the current_table by removing those rows that are in the archive_table (matched on id).
Intended SQL query:
DELETE FROM current_table WHERE id IN ( SELECT id FROM archive_table);
Attempted python code:
import sqlite3
def clean_up(current_table, archive_table):
db = sqlite3.connect(sqlite_db)
cursor = db.cursor()
sql_query_delete = '''DELETE FROM %s WHERE id IN ( SELECT id FROM %s);''' % (current_table, archive_table)
try:
cursor.execute(sql_query_delete)
db.commit()
db.close()
except:
print("error deleting")
Now working. The database file was locked by another process. Removing the pointless try/except led me to the detailed error message.
Code is follow. How to get replaced ? by value of variables [table, url]?
Expected SQL command is select * from OTHER_URL where url="http://a.com/a.jpg"
This SQL command occurs no error on the sqlite3 command line interface.
import sqlite3
from contextlib import closing
dbname = "ng.db"
with closing(sqlite3.connect(dbname)) as conn:
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS OTHER_URL (url TEXT)")
conn.commit()
table = "OTHER_URL"
url = "http://a.com/a.jpg"
with closing(sqlite3.connect(dbname)) as conn:
c = conn.cursor()
c.execute('select * from ? where url="?"', [table, url])
print c.fetchone()
There are two errors here. Firstly, you can't use parameter substitution for table names (or column names), only for values. You need to use string interpolation for anything else.
Secondly, you don't need quotes around the value parameter; the substitution will take care of that.
So:
c.execute('select * from {} where url=?'.format(table), [url])
I am currently trying to use pyodbc to insert data from a .csv into an Azure SQL Server database. I found a majority of this syntax on Stack Overflow, however for some reason I keep getting one of two different errors.
1) Whenever I use the following code, I get an error that states 'The SQL contains 0 parameter markers, but 7 parameters were supplied'.
import pyodbc
import csv
cnxn = pyodbc.connect('driver', user='username', password='password', database='database')
cnxn.autocommit = True
cursor = cnxn.cursor()
csvfile = open('CSV File')
csv_data = csv.reader(csvfile)
SQL="insert into table([Col1],[Col2],[Col3],[Col4],[Col5],[Col6],[Col7]) values ('?','?','?','?','?','?','?')"
for row in csv_data:
cursor.execute(SQL, row)
time.sleep(1)
cnxn.commit()
cnxn.close()
2) In order to get rid of that error, I am defining the parameter markers by adding '=?' to each of the columns in the insert statement (see code below), however this then gives the following error: ProgrammingError: ('42000'"[42000] [Microsoft] [ODBC SQL Server Driver][SQL Server] Incorrect syntax near '=').
import pyodbc
import csv
cnxn = pyodbc.connect('driver', user='username', password='password', database='database')
cnxn.autocommit = True
cursor = cnxn.cursor()
csvfile = open('CSV File')
csv_data = csv.reader(csvfile)
SQL="insert into table([Col1]=?,[Col2]=?,[Col3]=?,[Col4]=?,[Col5]=?,[Col6]=?,[Col7]=?) values ('?','?','?','?','?','?','?')"
for row in csv_data:
cursor.execute(SQL, row)
time.sleep(1)
cnxn.commit()
cnxn.close()
This is the main error I am haveing trouble with, I have searched all over Stack Overflow and can't seem to find a solution. I know this error is probably very trivial, however I am new to Python and would greatly appreciate any advice or help.
Since SQL server can import your entire CSV file with a single statement this is a reinvention of the wheel.
BULK INSERT my_table FROM 'CSV_FILE'
WITH ( FIELDTERMINATOR=',', ROWTERMINATOR='\n');
If you want to persist with using python, just execute the above query with pyodbc!
If you would still prefer to execute thousands of statements instead of just one
SQL="insert into table([Col1],[Col2],[Col3],[Col4],[Col5],[Col6],[Col7]) values (?,?,?,?,?,?,?)"
note that the ' sorrounding the ? shouldn't be there.
# creating column list for insertion
colsInsert = "["+"],[".join([str(i) for i in mydata.columns.tolist()]) +']'
# Insert DataFrame recrds one by one.
for i,row in mydata.iterrows():
sql = "INSERT INTO Test (" +colsInsert + ") VALUES (" + "%?,"*(len(row)-1) + "%?)"
cursor.execute(sql, tuple(row))
# cursor.execute(sql, tuple(row))
# the connection is not autocommitted by default, so we must commit to save our changes
c.commit()
I have a script (see below) that runs perfectly in Windows that I'm trying to move to an Ubuntu environment. I have setup the PostgreSQL database exactly the same, with the exact same tables and usernames, etc. However, when I try to run the script in Ubunu it fails when it parses the "with" statement.
Here is the "with" statement:
with con:
cur = con.cursor()
cur.executemany(final_str, symbols)
I get the following error:
INSERT INTO symbol (ticker, instrument, name, sector, currency, created_date, last_updated_date) VALUES (%s, %s, %s, %s, %s, %s, %s) 502
Traceback (most recent call last):
File "loadSPX.py", line 60, in <module>
insert_snp500_symbols(symbols)
File "loadSPX.py", line 54, in insert_snp500_symbols
with con:
AttributeError: __exit__
However, if I remove the "with" and change it to the following it works perfectly:
cur = con.cursor()
cur.executemany(final_str, symbols)
con.commit()
Any ideas what is causing this? Here is the full script below:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import datetime
import lxml.html
import psycopg2 as mdb
import psycopg2.extras
from math import ceil
def obtain_parse_wiki_snp500():
"""Download and parse the Wikipedia list of S&P500
constituents using requests and libxml.
Returns a list of tuples for to add to database."""
# Stores the current time, for the created_at record
now = datetime.datetime.utcnow()
# Use libxml to download the list of S&P500 companies and obtain the symbol table
page = lxml.html.parse('http://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
symbolslist = page.xpath('//table[1]/tr')[1:503]
# Obtain the symbol information for each row in the S&P500 constituent table
symbols = []
for symbol in symbolslist:
tds = symbol.getchildren()
sd = {'ticker': tds[0].getchildren()[0].text,
'name': tds[1].getchildren()[0].text,
'sector': tds[3].text}
# Create a tuple (for the DB format) and append to the grand list
symbols.append( (sd['ticker'], 'stock', sd['name'],
sd['sector'], 'USD', now, now) )
return symbols
def insert_snp500_symbols(symbols):
"""Insert the S&P500 symbols into the database."""
# Connect to the PostgreSQL instance
db_host = 'localhost'
db_user = 'sec_user'
db_pass = 'XXXXXXX'
db_name = 'securities_master'
con = mdb.connect(host=db_host, dbname=db_name, user=db_user, password=db_pass)
# Create the insert strings
column_str = "ticker, instrument, name, sector, currency, created_date, last_updated_date"
insert_str = ("%s, " * 7)[:-2]
final_str = "INSERT INTO symbol (%s) VALUES (%s)" % (column_str, insert_str)
print final_str, len(symbols)
# Using the MySQL connection, carry out an INSERT INTO for every symbol
with con:
cur = con.cursor()
cur.executemany(final_str, symbols)
if __name__ == "__main__":
symbols = obtain_parse_wiki_snp500()
insert_snp500_symbols(symbols)
Your psycopg2 library on Ubuntu is too old; you need to upgrade to version 2.5 or newer. In older versions connections do not yet support being used as context managers.
See the Psycopg 2.5 release announcement:
Connections and cursors as context managers
A recent DBAPI extension has standardized the use of connections and cursors as context managers: it is now possible to use an idiom such as:
with psycopg2.connect(DSN) as conn:
with conn.cursor() as curs:
curs.execute(SQL)
with the intuitive behaviour: when the cursor block exits the cursor is closed; when the connection block exits normally the current transaction is committed, if it exits with an exception instead the transaction is rolled back, in either case the connection is ready to be used again
If you installed the python-psycopg2 system package you are most likely using 2.4.5; only Utopic Unicorn (14.10) has a more recent version (2.5.3). To install the newer version from soure, you'll need to install the Python development headers (python-dev) plus the PostgreSQL client library headers (libpq-dev).
My simple test code is listed below. I created the table already and can query it using the SQLite Manager add-in on Firefox so I know the table and data exist. When I run the query in python (and using the python shell) I get the no such table error
def TroyTest(self, acctno):
conn = sqlite3.connect('TroyData.db')
curs = conn.cursor()
v1 = curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
print v1
conn.close()
When you pass SQLite a non-existing path, it'll happily open a new database for you, instead of telling you that the file did not exist before. When you do that, it'll be empty and you'll instead get a "No such table" error.
You are using a relative path to the database, meaning it'll try to open the database in the current directory, and that is probably not where you think it is..
The remedy is to use an absolute path instead:
conn = sqlite3.connect('/full/path/to/TroyData.db')
You need to loop over the cursor to see results:
curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
for row in curs:
print row[0]
or call fetchone():
print curs.fetchone() # prints whole row tuple
The problem is the SQL statment. you must specify the db name and after the table name...
'''SELECT * FROM db_name.table_name WHERE acctno = ? '''