I have a weird issue, which is probably easy to resolve.
I have a class Database with an __init__ and an executeDictMore method (among others).
class Database():
def __init__(self, database, server,login, password ):
self.database = database
my_conv = { FIELD_TYPE.LONG: int }
self.conn = MySQLdb.Connection(user=login, passwd=password, db=self.database, host=server, conv=my_conv)
self.cursor = self.conn.cursor()
def executeDictMore(self, query):
self.cursor.execute(query)
data = self.cursor.fetchall()
if data == None :
return None
result = []
for d in data:
desc = self.cursor.description
dict = {}
for (name, value) in zip(desc, d) :
dict[name[0]] = value
result.append(dict)
return result
Then I instantiate this class in a file db_functions.py :
from Database import Database
db = Database()
And I call the executeDictMore method from a function of db_functions :
def test(id):
query = "SELECT * FROM table WHERE table_id=%s;" %(id)
return db.executeDictMore(query)
Now comes the weird part.
If I import db_functions and call db_functions.test(id) from a python console:
import db_functions
t = db_functions.test(12)
it works just fine.
But if I do the same thing from another python file I get the following error :
AttributeError: Database instance has no attribute 'executeDictMore'
I really don't understand what is going on here. I don't think I have another Database class interfering. And I append the folder where the modules are in sys.path, so it should call the right module anyway.
If someone has an idea, it's very welcome.
You have another Database module or package in your path somewhere, and it is getting imported instead.
To diagnose where that other module is living, add:
import Database
print Database.__file__
before the from Database import Database line; it'll print the filename of the module. You'll have to rename one or the other module to not conflict.
You could at least try to avoid SQL injection. Python provides such neat ways to do so:
def executeDictMore(self, query, data=None):
self.cursor.execute(query, data)
and
def test(id):
query = "SELECT * FROM table WHERE table_id=%s"
return db.executeDictMore(query, id)
are the ways to do so.
Sorry, this should rather be a comment, but an answer allows for better formatting. Iam aware that it doesn't answer your question...
You should insert (not append) into your sys.path if you want it first in the search path:
sys.path.insert(0, '/path/to/your/Database/class')
Im not too sure what is wrong but you could try passing the database object to the function as an argument like
db_functions.test(db, 12) with db being your Database class
Related
Here is some custom code I wrote that I think might be problematic for this particular use case.
class SQLServerConnection:
def __init__(self, database):
...
self.connection_string = \
"DRIVER=" + str(self.driver) + ";" + \
"SERVER=" + str(self.server) + ";" + \
"DATABASE=" + str(self.database) + ";" + \
"Trusted_Connection=yes;"
self.engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"mssql+pyodbc", \
query={'odbc_connect': self.connection_string}
)
)
# Runs a command and returns in plain text (python list for multiple rows)
# Can be a select, alter table, anything like that
def execute(self, command, params=False):
# Make a connection object with the server
with self.engine.connect() as conn:
# Can send some parameters along with a plain text query...
# could be single dict or list of dict
# Doc: https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#sending-multiple-parameters
if params:
output = conn.execute(sqlalchemy.text(command,params))
else:
output = conn.execute(sqlalchemy.text(command))
# Tell SQL server to save your changes (assuming that is applicable, is not with select)
# Doc: https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#committing-changes
try:
conn.commit()
except Exception as e:
#pass
warn("Could not commit changes...\n" + str(e))
# Try to consolidate select statement result into single object to return
try:
output = output.all()
except:
pass
return output
If I try:
cnxn = SQLServerConnection(database='MyDatabase')
cnxn.execute("SELECT * INTO [dbo].[MyTable_newdata] FROM [dbo].[MyTable] ")
or
cnxn.execute("SELECT TOP 0 * INTO [dbo].[MyTable_newdata] FROM [dbo].[MyTable] ")
Python returns this object without error, <sqlalchemy.engine.cursor.LegacyCursorResult at 0x2b793d71880>, but upon looking in MS SQL Server, the new table was not generated. I am not warned about the commit step failing with the SELECT TOP 0 way; I am warned ('Connection' object has no attribute 'commit') in the above way.
CREATE TABLE, ALTER TABLE, or SELECT (etc) appears to work fine, but SELECT * INTO seems to not be working, and I'm not sure how to troubleshoot further. Copy-pasting the query into SQL Server and running appears to work fine.
As noted in the introduction to the 1.4 tutorial here:
A Note on the Future
This tutorial describes a new API that’s released in SQLAlchemy 1.4 known as 2.0 style. The purpose of the 2.0-style API is to provide forwards compatibility with SQLAlchemy 2.0, which is planned as the next generation of SQLAlchemy.
In order to provide the full 2.0 API, a new flag called future will be used, which will be seen as the tutorial describes the Engine and Session objects. These flags fully enable 2.0-compatibility mode and allow the code in the tutorial to proceed fully. When using the future flag with the create_engine() function, the object returned is a subclass of sqlalchemy.engine.Engine described as sqlalchemy.future.Engine. This tutorial will be referring to sqlalchemy.future.Engine.
That is, it is assumed that the engine is created with
engine = create_engine(connection_url, future=True)
You are getting the "'Connection' object has no attribute 'commit'" error because you are creating an old-style Engine object.
You can avoid the error by adding future=True to your create_engine() call:
self.engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"mssql+pyodbc",
query={'odbc_connect': self.connection_string}
),
future=True
)
Use this recipe instead:
#!python
from sqlalchemy.sql import Select
from sqlalchemy.ext.compiler import compiles
class SelectInto(Select):
def __init__(self, columns, into, *arg, **kw):
super(SelectInto, self).__init__(columns, *arg, **kw)
self.into = into
#compiles(SelectInto)
def s_into(element, compiler, **kw):
text = compiler.visit_select(element)
text = text.replace('FROM',
'INTO TEMPORARY TABLE %s FROM' %
element.into)
return text
if __name__ == '__main__':
from sqlalchemy.sql import table, column
marker = table('marker',
column('x1'),
column('x2'),
column('x3')
)
print SelectInto([marker.c.x1, marker.c.x2], "tmp_markers").\
where(marker.c.x3==5).\
where(marker.c.x1.in_([1, 5]))
This needs some tweaking, hence it will replace all subquery selects as select INTOs, but test it for now, if it worked it would be better than raw text statments.
Have you tried this from this answer by #Michael Berkowski:
INSERT INTO assets_copy
SELECT * FROM assets;
The answer states that MySQL documentation states that SELECT * INTO isn't supported.
Having self.con = sqlite3.connect (db) and self.cur = self.con.cursor() in the db.py file, how can I call them in the main.py file to use cursor.execute? I wrote db.self.cur.execute('SELECT Name FROM TableExample'), but obviously I was wrong, there is an error of course
P.S: In db.py do I have to insert the path of the database inside db of the sqlite3.connect(db)?
I had tried like this:
DB.PY
import sqlite3
class Database:
def __init__(self, db):
self.con = sqlite3.connect(db)
self.cur = self.con.cursor()
sql = """
CREATE TABLE IF NOT EXISTS employees(
ID Integer Primary Key,
example1 integer,
example2 integer
)
"""
self.cur.execute(sql)
self.con.commit()
MAIN.PY
from db import Database
db = Database('/home/xxxx/database.db')
def example():
db.self.cur.execute('SELECT Name FROM TableExample') #error
result=[row[0] for row in cursor]
return result
UPDATE: New example in main.py
def example():
db.cur.execute('SELECT Name FROM TableExample')
result=[row[0] for row in db.cur]
return result
There are a few problems to fix here. The error on this line:
db.self.cur.execute('SELECT Name FROM TableExample') #error
is because of the use of self. You'll want to read up on how to use classes, but in short self (named that by convention, not requirement) is how an object instance refers to itself, such as where you set self.cur in __init__(). You don't refer to self outside the object.
Instead, you want to refer to the cur attribute of the db object like this:
db.cur.execute('SELECT Name FROM TableExample')
The next issue is the list expression that reads from the database. This:
result=[row[0] for row in cursor]
is going to cause an error, because cursor is not defined. You probably want to refer to the cur attribute in the db object:
result=[row[0] for row in db.cur]
However, while this works, it's not the best way to do it. You're implicitly using the results of the last query executed on the cursor, which does work, but is less obvious.
Better would be to create a new cursor object from the query and iterate that:
query = db.cur.execute('SELECT Name FROM TableExample')
result = [row[0] for row in query]
I'm trying to add a block of text into a sqlAlchemy table, which I want to compress to save space with it. Looking through various answers I came up with what I think should be working, but is not. I'm working with a sqlite database.
Updated: Was pointed out I was attempting to use mysql on sqlite which I wasn't aware that was what was happening. I adjusted to use zlib instead and it works to a degree, which gives me a new error that I do not understand.
# proper imports and stuff to make this work
from sqlalchemy import func
class Data(Base):
__tablename__ = 'data'
# ...
text_blobbed = Column('text', BLOB)
#hybrid_property
def text(self):
# return func.decompress(self.text_blobbed)
return self.text_blobbed.decode("zlib")
#text.setter
def text(self, stuff):
# self.text_blobbed = func.compress(stuff)
self.text_blobbed = stuff.encode("zlib")
old error from func.
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such function: compress [SQL: ...... ]
I can now add in the text via Data.text = "a really big block of text"
But when I go to query for this like
session.query(Data.text).filter(Data.id.like(2)).first()
I get an error:
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object associated with Data.text_blobbed has an attribute 'decode'
Doing this is fine.
r = session.query(Data).filter(Data.id.like(2)).first()
print r.text
I've also looked at the text_blobbed which is a set(). And I can do this that works:
r = session.query(Data.text_blobbed.filter( ... ).first()[0].decode("zlib")
print r
But if I move that [0] into the hybrid_property for
...
return self.text_blobbed[0].decode("zlib")
and query:
r = session.query(Data.text).filter( ... ).first()
I get the error:
NotImplementedError: Operator 'getitem' is not supported on this expression
So, I'm a bit confused still.
I've been looking at these things:
SQLAlchemy - Writing a hybrid method for child count
mysql Compress() with sqlalchemy
SELECT UNCOMPRESS(text) FROM with sqlalchemy
http://docs.sqlalchemy.org/en/latest/orm/mapped_sql_expr.html?highlight=descriptor
Using python Mongoengine I am trying to create databases and add documents to different databases. Here's how I am trying to do it :
from mongoengine import *
class MongoDataAccessObject():
# method to connect to the database and initialize the tables etc.
def __init__(self, my_env, helperObj):
print "initializing db for the environment ", my_env
self.con = None
self.dbName = my_env
self.helper_obj = helperObj
try:
self.con = connect(db=self.dbName)
except Exception as e:
print e
def addRecord(self, document_object):
document_object.save()
Now, I pass the names of different databases that I want created while creating the object of the above class, and add the documents like this :
for my_env in list_of_envs:
dao = MongoDataAccessObject(my_env, helper_object)
dao.addRecord(myDocument)
Now there are 2 questions here:
For some reason all my documents keep getting added to the same DB (the first one being passed while MongoDataAccessObject object creation. I would assume that when I am creating a new object every time, while passing a different db name each time, a new connection should get created to the new db passed and documents should get added to the db which is currently connected to.
To verify if I am actually connected to the DB in question or not, I could not find a method like get_database_name() on the connection object. Is there a way to verify if I am getting connected to the DB name being passed or not ?
Ok did some more research and found this:
https://github.com/MongoEngine/mongoengine/issues/605
Tried it out like this in iptyhon:
from mongoengine import *
import datetime
class Page(Document):
title = StringField(max_length=200, required=True)
date_modified = DateTimeField(default=datetime.datetime.now)
def switch(model, db):
model._meta['db_alias'] = db
# must set _collection to none so it is re-evaluated
model._collection = None
return model
register_connection('default', name='testing')
register_connection('mycon', name='db1')
page = Page(title="Test Page")
page = switch(page, 'mycon')
page.save()
This works and creates a db named db1 and stores the document there.
Now I do this again:
register_connection('mycon2', name='db2')
page = Page(title="Test Page")
page = switch(page, 'mycon2')
page.save()
Contrary to my expectation this time db2 was not created (checked from both mongo client and from Robomongo), however the document was saved successfully. Wonder where exactly did the document get saved then ??
So to figure that out repeated the above exercise with a small change as below:
register_connection('mycon2', name='db2')
page = Page(title="Test Page")
page = switch(page, 'mycon2')
x = page.save()
# did a dir(x) and found that there is _get_db, so tried it out as below
x._get_db()
and the output was :
Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, read_preference=Primary()), u'db2')
which I guess means that the document got saved in a database named db2. But where on earth is this db2 ???? Why can't I see it through either mongo client or even Robomongo etc. ?
Finally the only way I could find to achieve the above was through the context manager with statement provided by MongoEngine, which also is well documented here : http://docs.mongoengine.org/guide/connecting.html#switch-database
The way I did it in my code above is something like this :
In the first call to the db, a default db needs to be created, which should have the alias as default. Only then can another alias be created, else Mongoengine throws an error saying that no default db found.
So to do this, when the very first db object is created, a False flag is sent to the __init__of MongoDataAccessObject
So to do this, the MongoDataAccessObject was also changed to something like this:
class MongoDataAccessObject():
# method to connect to the database and initialize the tables etc.
def __init__(self, my_env, helperObj, is_default_db_set):
print "initializing db for the environment ", my_env
self.con = None
self.dbName = my_env
self.helper_obj = helperObj
self.db_alias = my_env
self.is_default_db_set = is_default_db_set
try:
# all of this part with the is_default_db_set and register_connection() is needed because MongoEngine does not
# provide a simple way of changing dbs or switching db connections. The only way to do it is through the switch_db()
# context manager they provide (which has been used in the addRecord() below)
if not self.is_default_db_set:
self.con = connect(db=self.dbName, alias='default')
else:
# register_connection(alias_name, db_name)
register_connection(self.db_alias, self.dbName)
except Exception as e:
print e
And the addRecord() was also modified as :
def addRecord(self, document_object):
with switch_db(model_object, self.db_alias) as model_object:
document_object = model_object()
document_object.save()
And this part above:
for my_env in list_of_envs:
dao = MongoDataAccessObject(my_env, helper_object)
dao.addRecord(myDocument)
was also modified as:
for my_env in list_of_envs:
dao = MongoDataAccessObject(my_env,helper_object,mongo_default_db_flag)
dao.addRecord(myDocument)
And this seemed to do the job for me.
I recently had a Python 2.7x project where I needed to use mysql.connector to execute multiple, semicolon delineated statements in one query. This is explained nicely in the this post..
However, I needed to use mysql.connector with Twisted for my current project, which means using Twisted's excellent enterprise.adbapi module to make my new blocking database connection non-blocking.
config = {"user": username, "password": password, "host": hostname,
"database": database_name, "raise_on_warnings": True}
cp = adbapi.ConnectionPool("mysql.connector", **config)
my test statements are defined below. I apologize that they are a bit of a frivolous example, but I know the results that I expect, and it should be enough to verify that I'm getting results for multiple statements.
statement1 = "SELECT * FROM queue WHERE id = 27;"
statement2 = "SELECT * FROM order WHERE id = 1;"
statement_list = [statement1, statement2]
statements = " ".join(statement_list)
The problem comes when I now try to execute the ConnectionPool method .runQuery()
def _print_result(result):
if result:
print("this is a result")
print(result)
else:
print("no result")
reactor.stop()
d = cp.runQuery(statements, multi=True)
d.addBoth(_print_result)
this gets me the following result:
this is a result [Failure instance: Traceback: : No result set to fetch from.
How can I use Twisted's adbapi module to get the results that I know are there?
So, it turns out that when using adbapi.ConnectionPool.runQuery(), the default behavior is to send the result of the database interrogation to the cursor.fetchall() method. However, when using mysql.connector, this doesn't work, even without twisted. Instead one needs to iterate over the result set, and call fetchall() on each member of the set.
So, the way I solved this was with the following subclass.
from twisted.enterprise import adbapi
class NEWadbapiConnectionPool(adbapi.ConnectionPool):
def __init__(self, dbapiName, *connargs, **connkw):
adbapi.ConnectionPool.__init__(self, dbapiName, *connargs, **connkw)
def runMultiQuery(self, *args, **kw):
return self.runInteraction(self._runMultiQuery, *args, **kw)
def _runMultiQuery(self, trans, *args, **kw):
result = trans.execute(*args, **kw)
result_list = []
for item in result:
if item.with_rows:
result_list.append(item.fetchall())
return result_list
so now I create the following:
def _print_result(result):
if result:
print("this is a result")
print(result)
else:
print("no result")
reactor.stop()
cp = NEWadbapiConnectionPool("mysql.connector", **config)
d = cp.runMultiQuery(statements, multi=True)
d.addBoth(_print_result)
and get a list of the results for each statement.
I hope someone else finds this useful.
RunQuery always expects results. The right way to do it is to call runOperation() which does not fetch results.
If you want to use .runQuery, it expects results to fetch so you need to return something
dbpool.runQuery(
"UPDATE something SET col1=true WHERE some_id=123 RETURNING *"
)
.runOperation does not expect results
dbpool.runOperation(
"UPDATE something SET col1=true WHERE some_id=123"
)