We Keep Coding

Postgres fails fetching data in Python

I am using Python with psycopg2 module to get data from Postgres database.
The database is quite large (tens of GB).
Everything appears to be working, I am creating objects from the fetched data.
However, after ~160000 of created objects I get the following error:
I suppose the reason is the amount of data, but I could not get anywhere searching for a solution online. I am not aware of using any proxy and have never used any on this machine before, the database is on localhost.

It's interesting how often the "It's a local server so I'm not open to SQL injection" stance leads to people thinking that string interpolation is somehow easier than a parameterized query. In your case it's ended up with:
'... cookie_id = \'{}\''.format(cookie)
So you've ended up with something that's less legible and also fails (though from the specific error I don't know exactly how). Use parameterization:
cursor.execute("SELECT user_id, created_at FROM cookies WHERE cookie_id = %s ORDER BY created_at DESC;", (cookie,))
Bottom line, do it the correct way all the time. Note, there are cases where you must use string interpolation, e.g. for table names:
cursor.execute("SELECT * FROM %s", (table_name,)) # Not valid
cursor.execute("SELECT * FROM {}".format(table_name)) # Valid
And in those cases, you need to take other precautions if someone else can interact with the code.

pyodbc stored procedures / Microsoft SQL Server error 42000

I'm working on a script to automate a file load procedure. So, naturally I need to perform some stored procedures that already exist. I'm using pyodbc to connect to my database. I can SELECT perfectly fine from the database, but when I try to execute from the database I get this error:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 10.0]
Syntax error, permission violation, or other nonspecific error (0) (SQLExecDirectW)')
I can't figure out what the problem here is - the user has full DB admin permissions, the syntax is correct based off what the pyodbc official documentation says.
print("Executing SP")
conn.execute('{EXEC TEMP.s_p_test}')
print("SP Executed.")
Here, TEMP is the schema for the type of stored procedure in that specific database. I.e., it's the full name of the stored procedure. I feel like it's probably something stupidly obvious that I'm just missing.

I tried a couple of things to fix it. As #Brian Pendleton suggested, I had tried to change from an explicit database user defined via UID and PWD to trusted_connection=True. Unfortunately that did not change anything.
However, out of curiosity I decided to see what taking the curly braces out of the function call would do. The execution worked immediately and produced the desired output. It would seem that the documentation at pyodbc's wiki either shows bad examples or I found a bug I don't know how to replicate because I don't know what makes my situation abnormal.
Or, in other words, instead of
conn.execute('{EXEC TEMP.s_p_test}')
I used
conn.execute('EXEC TEMP.s_p_test')

Is it possible to use pyodbc to read Paradox tables that are open in the Paradox gui?

I'm working in a environment with a very poorly managed legacy Paradox database system. (I'm not the administrator.) I've been messing around with using pyodbc to interact with our tables, and the basic functionality seems to work. Here's some (working) test code:
import pyodbc
LOCATION = "C:\test"
cnxn = pyodbc.connect(r"Driver={{Microsoft Paradox Driver (*.db )\}};DriverID=538;Fil=Paradox 5.X;DefaultDir={0};Dbq={0};CollatingSequence=ASCII;".format(LOCATION), autocommit=True, readonly=True)
cursor = cnxn.cursor()
cursor.execute("select last, first from test")
row = cursor.fetchone()
print row
The problem is that most of our important tables are going to be open in someone's Paradox GUI at pretty much all times. I get this error whenever I try to do a select from one of those tables:
pyodbc.Error: ('HY000', "[HY000] [Microsoft][ODBC Paradox Driver] Could not lock
table 'test'; currently in use by user '(unknown)' on machine '(unknown)'. (-1304)
(SQLExecDirectW)")
This is, obviously, because pyodbc tries to lock the table when cursor.execute() is called on it. This behavior makes perfect sense, since cursor.execute() runs arbitary SQL code and could change the table.
However, Paradox itself (through its gui) seems to handle multiple users fine. It only gives you similar errors if you try to restructure the table while people are using it.
Is there any way I can get pyodbc to use some sort of read-only mode, such that it doesn't have to lock the table when I'm just doing select and such? Or is locking a fundamental part of how it works that I'm not going to be able to get around?
Solutions that would use other modules are also totally fine.

Ok, I finally figured it out.
Apparently, odbc dislikes Paradox tables which have no primary key. You cannot update tables with no primary key under any circumstances, and you cannot read from tables with no primary key unless you are the only user trying to access that table.
Unrelatedly, you get essentially the same error messages from password-protected tables if you don't supply a password.
So I was testing my script on two different tables, one of which has both a password and a primary key, and one of which had neither. I assumed the error messages had the same root cause, but it was actually two different problems, with different solutions.
There still seems to be no way to get access to tables without primary keys if they are open in someone's GUI, but that's a smaller issue.

Make sure that you have the latest version of pyobdc (3.0.6)here, according to them, they
Added Cursor.commit() and Cursor.rollback(). It is now possible to use
only a cursor in your code instead of keeping track of a connection
and a cursor.
Added readonly keyword to connect. If set to True, SQLSetConnectAttr
SQL_ATTR_ACCESS_MODE is set to SQL_MODE_READ_ONLY. This may provide
better locking semantics or speed for some drivers.
Fixed an error reading SQL Server XML data types longer than 4K.
Also, i have tested this on a paradox server using readonly and it does works.
Hope this helps!

I just published a Python library for reading Paradox database files via the pxlib C-library: https://github.com/mherrmann/pypxlib. This operates on the file-level so should also let you read the database independently of who else is currently accessing it. Since it does not synchronize read/write accesses, you do have to be careful though!

Python | MySQLdb -> I cant insert

I have an odd problem.
There are two databases. One has all products in it for an online shop. The other one has the online-shop software on it and only the active products.
I want to take the values from db1, change some values in python and insert into db2.
db1.execute("SELECT * FROM table1")
for product in db1.fetchall():
# ...
db2.execute("INSERT INTO table2 ...")
print "Check: " + str(db2.countrow)
So I can get the values via select, even selecting from db2 is no problem. My check always gives me 1 BUT there are no new rows in table2. The autoincrement value grows but the there is no data.
And I dont even get an error like "couldnt insert"
So does anybody has an idea what could be wrong? (if i do an insert manually via phpmyadmin it works... and if i just take the sql from my script and do it manually the sql-statements work aswell)
EDIT: Found the answer here How do I check if an insert was successful with MySQLdb in Python?
Is there a way to make these executes without committing everytime?
I have a wrapper around the mysqldb and it works perfectly for me, changing the behaviour with commit I need to make some big changes.
EDIT2: ok i found out that db1 is MyISAM (which would work without commiting) and db2 is InnoDB (which actually seems only to work with commiting) I guess I have to change db2 to MyISAM aswell.

Try adding db2.commit() after the inserts if you're using InnoDB.
Starting with 1.2.0, MySQLdb disables autocommit by default, as
required by the DB-API standard (PEP-249). If you are using InnoDB
tables or some other type of transactional table type, you'll need to
do connection.commit() before closing the connection, or else none of
your changes will be written to the database.
http://mysql-python.sourceforge.net/FAQ.html#my-data-disappeared-or-won-t-go-away

Execute a prepared statement in sqlalchemy

I have to run 40K requests against a username:
SELECT * from user WHERE login = :login
It's slow, so I figured I would just use a prepared statement.
So I do
e = sqlalchemy.create_engine(...)
c = e.connect()
c.execute("PREPARE userinfo(text) AS SELECT * from user WHERE login = $1")
r = c.execute("EXECUTE userinfo('bob')")
for x in r:
do_foo()
But I have a:
InterfaceError: (InterfaceError) cursor already closed None None
I don't understand why I get an exception

Not sure how to solve your cursor related error message, but I dont think a prepared staement will solve your performance issue - as long as your using SQL server 2005 or later the execution plan for SELECT * from user WHERE login = $login will already be re-used and there will be no performance gain from the prepared statement. I dont know about MySql or other SQL database servers, but I suspect they too have similar optimisations for Ad-Hoc queries that make the prepared statement redundant.
It sounds like the cause of the performance hit is more down to the fact that you are making 40,000 round trips to the database - you should try and rewrite the query so that you are only executing one SQL statement with a list of the login names. Am I right in thinking that MySql supports an aray data type? If it doesnt (or you are using Microsoft SQL) you should look into passing in some sort of delimited list of usernames.

From this discussion, it might be a good idea to check your paster debug logs in case there is a better error message there.