I am using python and sqlite3 and would like to use the memory for temp files. According to the docs, https://www.sqlite.org/compile.html, SQLITE_TEMP_STORE=3 means "Always use memory". I can check the current value with:
import sqlite3
conn = sqlite3.connect("test.db")
cur = conn.cursor()
check_db = conn.execute( """ select * from pragma_compile_options where compile_options like 'TEMP_STORE=%' """).fetchall()
print("check_db:", check_db)
When I attempt to update:
sq_update = """ update pragma_compile_options set compile_options = 'TEMP_STORE=3' where compile_options like 'TEMP_STORE=1' """
conn.execute(sq_update) conn.commit()
The following error is returned.
INTERNALERROR> sqlite3.OperationalError: table pragma_compile_options may not be modified
My goal is to set tell sqlite to use the memory for temp files.
You need to examine the content of pragma_compile_options output to see the value of TEMP_STORE. You can only change the run-time setting if TEMP_STORE was explicitly set to non-zero value. In that case, use PRAGMA temp_store = 2 to achieve your goal. See https://www.sqlite.org/pragma.html#pragma_temp_store.
Related
Now, I have a study about python sqlite3 database. I think it is very simple problem but not allow next step. Could help me?
There is print OK on vscode terminal, but not revised to DB file. I'm searching several times but I can not fix it.
If I execute the code, it not sorting on DB files.
import sqlite3
conn = sqlite3.connect('sqliteDB1.db')
cursor = conn.cursor()
cursor.execute("SELECT * FROM member")
temp123 = cursor. fetchall()
print(temp123)
cursor.execute("SELECT * FROM member ORDER BY -code")
temp321 = cursor.fetchall()
conn.commit
print(temp321)
conn.close()
A select statement just returns data from a database, it will not modify it. Moreover, tables in SQL databases are inherently unordered sets. They have no intrinsic value, and you should never rely on the order of the rows that happens to be returned unless you explicitly sort it with an order by clause.
For some odd reason I can't get results from a callproc call in a Python test app. The stored procedure in MqSQL 5.2.47 looks like this:
CREATE PROCEDURE `mytestdb`.`getperson` (IN personid INT)
BEGIN
select person.person_id,
person.person_fname,
person.person_mi,
person.person_lname,
person.persongender_id,
person.personjob_id
from person
where person.person_id = personid;
END
Now, using PyCharm with Python 3.3, I can't seem to retrieve anything when calling this stored procedure. This code gets me the desired results:
import mysql.connector
cnx = mysql.connector.connect(user='root', host='127.0.0.1', database='mytestdb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.execute("select * from person where person.person_id = 1")
people = cursor.fetchall()
for person in people:
print(person)
cnx.close()
But this code with either cursor.fetchall() or cursor.fetchone()...
import mysql.connector
cnx = mysql.connector.connect(user='root', host='127.0.0.1', database='mytestdb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.callproc("getperson", [1])
people = cursor.fetchall()
for person in people:
print(person)
cnx.close()
... returns "mysql.connector.errors.InterfaceError: No result set to fetch from." There's an additional odd behavior using the cursor.execute() method like so...
import mysql.connector
cnx = mysql.connector.connect(user='root', host='127.0.0.1', database='mytestdb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.execute("call getperson(1)")
people = cursor.fetchall()
for person in people:
print(person)
cnx.close()
... because it yields "mysql.connector.errors.InterfaceError: Use cmd_query_iter for statements with multiple queries" followed by "mysql.connector.errors.InterfaceError: Use multi=True when executing multiple statements" despite the fact that I'm only returning one query result rather than multiple result sets. Is the MySQL Python connector treating the execute call on the stored procedure as a double query? How can I just call the stored procedure and get my results back? I really don't want dynamic SQL in my code. Thanks ahead for any advice!
Have you tried picking one of the resultsets?
for result in cursor.stored_results():
people = result.fetchall()
It could be that it's allocating for multiple resultsets even though you only have one SELECT stmt. I know in PHP's MySQLi stored procedures do this to allow for INOUT and OUT variable returns (which again, you have none of, but maybe it's allocating anyways).
The complete code I'm using (which is working) is:
import mysql.connector
cnx = mysql.connector.connect(user='me',password='pw',host='localhost',database='mydb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.callproc("getperson",[1])
for result in cursor.stored_results():
people=result.fetchall()
for person in people:
print person
cnx.close()
Getting the result of a stored procedure after calling cursor.callproc depends on these factors:
whether the result of calling the procedure is assigned to an INOUT or OUT parameter
whether the result consists of a single row or a result set (or result sets)
the python package used to make the call
The DBAPI spec has this to say on cursor.callproc:
Call a stored database procedure with the given name. The sequence of parameters must contain one entry for each argument that the procedure expects. The result of the call is returned as modified copy of the input sequence. Input parameters are left untouched, output and input/output parameters replaced with possibly new values.
The procedure may also provide a result set as output. This must then be made available through the standard .fetch*() methods.
In practice, using the return value of cursor.callproc can only work if the procedure returns a single row, with the number of columns matching the number of INOUT and OUT parameters, so there is some variation in how the results are handled.
Here is how these cases are handled by the principal MySQL Python connector packages - MySQL Connector, mysqlclient (MySQLdb) and PyMySQL.
Single row result, returned via INOUT or OUT parameters
MySQL Connector returns a modified copy of the input sequence as the return value of cursor.callproc; the value is a tuple.
params = [in_param, out_param1, out_param2]
in_, out1, out2 = cursor.callproc("test_proc", params)
mysqlclient and PyMySQL require that the database is queried for the output parameters, and the results then fetched via the cursor; the value is a tuple of tuples. The parameter names to be queried are of the form '#_{procedure_name}_{params.index(param)}'
cursor.callproc("test_proc", params)
cursor.execute("""SELECT #_test_proc_0, #_test_proc_1""")
result = cursor.fetchall()
One or more rows in a single result set, no INOUT or OUT parameters defined
MySQL Connector exposes the result via the cursor's stored_results method (cursor.stored_results is not part of the DBAPI spec)
cursor.callproc("test_proc", params)
results = [r.fetchall() for r in cursor.stored_results()]
mysqlclient and PyMySQL expose the result via the cursor's fetch* methods
cursor.callproc("test_proc", params)
results = cursor.fetchall()
Multiple result sets, no INOUT or OUT parameters defined
MySQL Connector exposes the result via the cursor's stored_results method
cursor.callproc("test_proc", params)
results = [r.fetchall() for r in cursor.stored_results()]
mysqlclient and PyMySQL require that each result set be fetched via the cursor, while calling cursor.nextset to advance to the next result set. Note that an extra, empty result set may be returned, a result of calling the procedure (this would also happen in the previous examples, if the result set were retrieved via cursor.nextset instead of only calling cursor.fetchall once).
cursor.callproc("test_proc", params)
results = [cursor.fetchall()]
while cursor.nextset():
results.append(cursor.fetchall())
Version Info
$ mysql --version
mysql Ver 15.1 Distrib 10.1.41-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
$ pip list | grep -i mysql
mysql-connector-python 8.0.18
mysqlclient 1.4.6
PyMySQL 0.9.3
Why not try it this way
cursor.callproc("getperson", ['1'])
I'm having trouble finding any information on how to do error checking on MySQLdb. I have been trying to do a simple update command for a MySQL database and it simply is not working. No matter how I change the terms, or the type of variables I submit to it.
Here are some of my (commented out) attempts:
timeid = twitseek['max_id']
#timeup = "UPDATE `timeid` set `timestamp`='" + str(timeid) + "';"
#print timeup
#c.execute(timeup)
#timeup = "UPDATE timeid SET timestamp=\"" + str(timeid) + "\"";
#timeup = "UPDATE timeid set timestamp = '500';"
timeup = 500
c.execute("""UPDATE timeid SET timestamp = %d;""", timeup)
#c.execute(timeup)
All I want to do is upload the value of timeid to the timestamp column's first value (or any value) in the table timeid.
Nothing I do seems to work and I've been sitting here for literally hours trying countless iterations.
You seem to be missing an obligatory call to .commit() on your connection object to commit your change.
# Your cursor is c
# We don't see your connection object, but assuming it is conn...
c.execute("""UPDATE timeid SET timestamp = %d;""", timeup)
conn.commit()
The above method will produce valid SQL, but you don't get the security benefit of prepared statements this way. The proper method to pass in parameters is to use %s, and pass in a tuple of parameters:
c.execute("UPDATE timeid SET timestamp = %s;", (timeup,))
conn.commit()
From the MySQLdb FAQ:
Starting with 1.2.0, MySQLdb disables autocommit by default, as
required by the DB-API standard (PEP-249). If you are using InnoDB
tables or some other type of transactional table type, you'll need to
do connection.commit() before closing the connection, or else none of
your changes will be written to the database.
Conversely, you can also use connection.rollback() to throw away any
changes you've made since the last commit.
As far as error checking goes, a failed connection or a syntactically invalid query will throw an exception. So you would want to wrap it in a try/except as is common in Python.
This isn't a question, so much as a pre-emptive answer. (I have gotten lots of help from this website & wanted to give back.)
I was struggling with a large bit of SQL query that was failing when I tried to run it via python using pymssql, but would run fine when directly through MS SQL. (E.g., in my case, I was using MS SQL Server Management Studio to run it outside of python.)
Then I finally discovered the problem: pymssql cannot handle temporary tables. At least not my version, which is still 1.0.1.
As proof, here is a snippet of my code, slightly altered to protect any IP issues:
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB)
cur = conn.cursor()
cur.execute(testQuery)
The above code FAILS (returns no data, to be specific, and spits the error "pymssql.OperationalError: No data available." if you call cur.fetchone() ) if I call it with testQuery defined as below:
testQuery = """
CREATE TABLE #TEST (
[sample_id] varchar (256)
,[blah] varchar (256) )
INSERT INTO #TEST
SELECT DISTINCT
[sample_id]
,[blah]
FROM [myTableOI]
WHERE [Shipment Type] in ('test')
SELECT * FROM #TEST
"""
However, it works fine if testQuery is defined as below.
testQuery = """
SELECT DISTINCT
[sample_id]
,[blah]
FROM [myTableOI]
WHERE [Shipment Type] in ('test')
"""
I did a Google search as well as a search within Stack Overflow, and couldn't find any information regarding the particular issue. I also looked under the pymssql documentation and FAQ, found at http://code.google.com/p/pymssql/wiki/FAQ, and did not see anything mentioning that temporary tables are not allowed. So I thought I'd add this "question".
Update: July 2016
The previously-accepted answer is no longer valid. The second "will NOT work" example does indeed work with pymssql 2.1.1 under Python 2.7.11 (once conn.autocommit(1) is replaced with conn.autocommit(True) to avoid "TypeError: Cannot convert int to bool").
For those who run across this question and might have similar problems, I thought I'd pass on what I'd learned since the original post. It turns out that you CAN use temporary tables in pymssql, but you have to be very careful in how you handle commits.
I'll first explain by example. The following code WILL work:
testQuery = """
CREATE TABLE #TEST (
[name] varchar(256)
,[age] int )
INSERT INTO #TEST
values ('Mike', 12)
,('someone else', 904)
"""
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB) ## obviously setting up proper variables here...
conn.autocommit(1)
cur = conn.cursor()
cur.execute(testQuery)
cur.execute("SELECT * FROM #TEST")
tmp = cur.fetchone()
tmp
This will then return the first item (a subsequent fetch will return the other):
('Mike', 12)
But the following will NOT work
testQuery = """
CREATE TABLE #TEST (
[name] varchar(256)
,[age] int )
INSERT INTO #TEST
values ('Mike', 12)
,('someone else', 904)
SELECT * FROM #TEST
"""
conn = pymssql.connect(host=sqlServer, user=sqlID, password=sqlPwd, \
database=sqlDB) ## obviously setting up proper variables here...
conn.autocommit(1)
cur = conn.cursor()
cur.execute(testQuery)
tmp = cur.fetchone()
tmp
This will fail saying "pymssql.OperationalError: No data available." The reason, as best I can tell, is that whether you have autocommit on or not, and whether you specifically make a commit yourself or not, all tables must explicitly be created AND COMMITTED before trying to read from them.
In the first case, you'll notice that there are two "cur.execute(...)" calls. The first one creates the temporary table. Upon finishing the "cur.execute()", since autocommit is turned on, the SQL script is committed, the temporary table is made. Then another cur.execute() is called to read from that table. In the second case, I attempt to create & read from the table "simultaneously" (at least in the mind of pymssql... it works fine in MS SQL Server Management Studio). Since the table has not previously been made & committed, I cannot query into it.
Wow... that was a hassle to discover, and it will be a hassle to adjust my code (developed on MS SQL Server Management Studio at first) so that it will work within a script. Oh well...
I have a list of ids of rows to fetch from database. I'm using python and psycopg2, and my problem is how to effectively pass those ids to SQL? I mean that if I know the length of that list, it is pretty easy because I can always manually or automatically add as many "%s" expressions into query string as needed, but here I don't know how much of them I need. It is important that I need to select that rows using sql "id IN (id1, id2, ...)" statement. I know that it is possible to check the length of the list and concatenate suitable number of "%s" into query string, but I'm afraid that it would be very slow and ugly. Does anyone have an idea on how to solve it? And please don't ask why I need to do it with "IN" statement - it is a benchmark which is a part of my class assignment. Thanks in advance!
Python tuples are converted to sql lists in psycopg2:
cur.mogrify("SELECT * FROM table WHERE column IN %s;", ((1,2,3),))
would output
'SELECT * FROM table WHERE column IN (1,2,3);'
For Python newcomers: It is unfortunately important to use a tuple, not a list here. Here's a second example:
cur.mogrify("SELECT * FROM table WHERE column IN %s;",
tuple([row[0] for row in rows]))
this question is old and maybe there is a newer one out there, but the answer my colleagues are going with right now is this:
sql = "SELECT * FROM table WHERE column = ANY(%(parameter_array)s)"
cur.execute(sql,{"parameter_array": [1, 2, 3]})
Now sql module of psycopg2 (https://www.psycopg.org/docs/sql.html) can be used to safeguard against errors and injections, like e.g.:
import psycopg2
from psycopg2 import sql
params = config()
conn = psycopg2.connect(**params)
cur = conn.cursor()
ids = ['a','b','c']
sql_query = sql.SQL('SELECT * FROM {} WHERE id IN ({});').format(
sql.Identifier('table_name'),
sql.SQL(',').join(map(sql.Literal, ids))
)
print (sql_query.as_string(cur)) # for debug
cur.execute(sql_query)
from configparser import ConfigParser
def config(filename='database.ini', section='postgresql'):
# create a parser
parser = ConfigParser()
# read config file
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception('Section {0} not found in the {1} file'.format(section, filename))
return db
Note: sql.Identifier will add quotes if needed so it will work if you use quoted identifiers in PostgreSQL also (they have to be used to allow e.g. case sensitive naming).
Example and structure of database.ini:
[postgresql]
host=localhost
port=5432
database=postgres
user=user
password=mypass