In my app, I have a model folder / package.
In that model folder, I generally have a whole bunch of import statements (most of them from SQLAlchemy, such as String, Integer, Session, etc.
I find myself writing the same import statements again and again for stuff like User, Blog, Article, Etc. and have become pretty sloppy at it.
Understanding that explicit is better than implicit and then going ahead and disregarding that, is there a way to do this once?
this probably isnt what you want but who knows
common_imports.py
from sqlalchemy import Model,String,...
in other scripts just do
from common_imports import *
[edit]
actually I came up with a really awful and hacky way to do what you want
a really hacky way to do it might be something like
\root_of_package
-main.py
\models
-__init__.py
-model1.py
-model2.py
then in models\__init__.py
from sqlalchemy import String,Integer,...
from glob import glob
Base = ...
for fname in glob("%s/*.py"%dirname(__file__)):
with open(fname) as f:
eval(f.read())
__all__ = locals()
then in models/model1.py
#just use all the imports/locals from __init__.py as if they already existed
#this will be hellish if you use an IDE that checks for errors for you
class Model1(Base):
id = Column(Integer,primary_key=True)
...
then in main.py
from models import Model1
I do this in one of my applications.
# approot/m.py
from models.model import String, Integer, Session, (etc)
# Otherwise blank!
Then in everything else:
import m
Then you can do m.String, m.Integer, m.Session and etc. This also means I can further encapsulate my code so that later when I write up user models I can just add to m.py
from models.user import User, Group, PermissionSet
and continue to access m.User, m.Group and etc.
Related
I have a module models.py with some data logic:
db = PostgresqlDatabase(database='database', user='user')
# models logic
and flask app which actually interacts with database:
from models import db, User, ...
But I want to move initializing all setting from one config file in flask app:
So I could separate importing db from other stuff (I need this for access to module variable db in models):
import models
from models import User, ...
app.config.from_object(os.environ['APP_SETTINGS'])
models.db = PostgresqlDatabase(database=app.config['db'],
user=app.config['db'])
and use further db as models.db
But seems it is kinda ugly. Duplicating imports, different usage of module stuff..
Is there any better way how to handle this situation?
I'd recommend 1 level of indirection, so that your code becomes like this:
import const
import runtime
def foo():
runtime.db.execute("SELECT ... LIMIT ?", (..., const.MAX_ROWS))
You get:
clear separation of leaf module const
mocking and/or reloading is possible
uniform and concise access in all user modules
To get rich API on runtime, use the "replace module with object at import" trick ( see __getattr__ on a module )
The docs say to put this somewhere:
from sqlalchemy import event
from colanderalchemy import setup_schema
event.listen(mapper, 'mapper_configured', setup_schema)
Where should this go in Pyramid? Should I be using Pyramid events instead of SQLAlchemy's?
When I tried putting it at the top of the models.py file, it complained about mapper not existing; should I still be using that?
You need to use the SQLAlchemy events as they tell what is happening inside the SQLAlchemy (they do not relate to the pyramid events at all).
The documentation for the ColanderAlchemy is confusing; what they call for a mapper here is your model class (it is not a mapper).
Thus in your models you should be doing something like:
class MyModelClass(Base):
...
event.listen(
MyModelClass,
"mapper_configured",
setup_schema)
The test suite shows it working like this:
from sqlalchemy import event
from colanderalchemy import setup_schema
from sqlalchemy.orm import mapper
event.listen(mapper, 'mapper_configured', setup_schema)
Please let me know if that fixes it for you and I can go update the documentation accordingly.
I'm somewhat new to Python (but not at all to programming, already fluent in Perl, PHP, & Ruby).
I'm running into a major difference between all my previous languages and Python: Mainly that I can easily read in data from another file into an entire project.
there are two major examples of this that I would like to solve:
1) I would like to have a settings/config file (in YAML) that gets read in and parsed in a config.py file. I then want a resulting dict() to be accessible to all my other files in the project. I have figured out how to do this by from lib.project.config import cfg but that means that for each page that is importing the configs the system has to REparse the yaml. That seems just silly to me. Is there no way to have that file parsed just once and then have the results accessible to any other file in my project?
2) I would like to import a database.py file that then looks at my configs to see if we need to import a sqlite3, mysql, or postgresql version of a database class. Again, I can manage this by putting the logic directly in each page that I need the database class for. But I hate having to paste code like
if cfg.get('db_type') == 'sqlite':
from lib.project.databases.sqlite3 import database
elif cfg.get('db_type') == 'mysql':
from lib.project.databases.mysql import database
at the top of each file that needs the database class. I'd much rather just add:
import lib.project.database
Any help would be very much appreciated.
I've done a good deal of googling and SO searches and not found my answers. Hopefuly one of you wiz's out there can help.
Thanks.
UPDATE:
The reason I'm doing things this way (for #2) is because I'm also trying to make other classes inherit the database class.
So lib/project/databases/sqlite.py is a definition of a database class.
And so is lib/project/databases/mysql.py.
The idea being that after this import is complete I can import classes like the users class and define it like so:
class user(database):
...
And thus inherit all of the structure and methods of the database class.
Your suggestions to simply create an instance based on the sqlite/mysql logic/decision and then pass that where it needs to be is a good solution for that. But I need a bit more...
Ideas?
Thank you all for your help in understanding more about Python and how to get what I'm looking for done.
Here is what I ended up doing:
1) The config file issue:
This apparently was mostly solved to begin with. I confirmed what everyone was saying: that you can "import" a file as many times as you like, but it is only ever parsed/processed/compiled once. As for making it reachable to any file that needs it:
Config class:
import os
import yaml
class Config:
def __init__(self):
self.path = os.getcwd()
stream = open(self.path+"/conf/config.yaml", 'r')
data = yaml.load(stream)
config_keys = data.keys()
for k in config_keys:
setattr(self, k, data.get(k))
if (os.path.isfile(self.path+"/conf/config-override.yaml") ):
stream = open(self.path+"/conf/config-override.yaml", 'r')
data = yaml.load(stream)
config_keys = data.keys()
for k in config_keys:
setattr(self, k, data.get(k))
config = Config()
And then any file that wants to use it:
from lib.project.Config import config
This is all working swimmingly so far.
2) Dynamic database type for Database class:
I just altered my overall design a tiny bit to make a Database class (mostly empty) inherit from either the Sqlite or Mysql classes (both custom builds which are wrappers to existing Sqlite3 and mysql-connector classes). This way there is always a solid Database class to inherit from, I only load in what files I need to, and it's all defined by my config file. Example:
Database class:
from lib.project.Config import config
if config.db_type == 'sqlite' :
from lib.project.databases.Sqlite import Sqlite
elif config.db_type == 'mysql':
from lib.project.databases.Mysql import Mysql
class Database(Sqlite if config.db_type == 'sqlite' else Mysql):
''' documentation '''
I'd still love to hear people's feedback on this code/method.
As I said, I'm still newish to Python and could still be missing something.
Thanks again everyone.
1) you're all set - the file will only be read once.
2) Agreed you don't want to copy any paste code like that - if you want to use 1 instance of the database throughout your project, you'd do something like:
import lib.project
db = lib.project.database
where db is just a local variable used to access your already created database. You would instantiate your database as database (resolving as you've done with the if/elif code whether to use sqlite3 or mysql) in lib/project/databases.py and then in your lib/project/__init__.py file you would do a from .databases import database.
if you want to use multiple databases (of the same sqlite3/mysql type) throughout your project, you would resolve which constructor Database is bound to (or inherits from) in databases.py:
from lib.project.Config import config
if config.db_type == 'sqlite' :
import lib.project.databases.Sqlite as Db
elif config.db_type == 'mysql':
import lib.project.databases.Mysql as Db
class Database(Db):
'''docs'''
Ok, so I'm trying to change this:
app/
- lib.py
- models.py
- blah.py
Into this:
app/
- __init__.py
- lib.py
- models/
- __init__.py
- user.py
- account.py
- banana.py
- blah.py
And still be able to import my models using from app.models import User rather than having to change it to from app.models.user import User all over the place. Basically, I want everything to treat the package as a single module, but be able to navigate the code in separate files for development ease.
The reason I can't do something like add for file in __all__: from file import * into init.py is I have circular references between the model files. A fix I don't want is to import those models from within the functions that use them. But that's super ugly. Let me give you an example:
user.py
...
from app.models import Banana
...
banana.py
...
from app.models import User
...
I wrote a quick pre-processing script that grabs all the files, re-writes them to put imports at the top, and puts it into models.py, but that's hardly an improvement, since now my stack traces don't show the line number I actually need to change.
Any ideas? I always though init was probably magical but now that I dig into it, I can't find anything that lets me provide myself this really simple convenience.
It depends on what your circular references are for. If you have a class in user that inherits from Banana and a class in banana that inherits from User, you can't do this. You also can't do it if each class defines a decorator that gets used in the other or anything else that gets called during the actual import.
You can, however, if you are just mutually referencing helper functions, or if your User object has a method to create new instances of Banana and your Banana object has a method that creates new instances of User. As long as the mutual reference doesn't actually get used until something in the module is called from outside it, then in your model folder, in __init__.py, you can just do something like:
import user
import banana
#etc...
user.banana = banana
banana.user = user
#etc...
User = user.User
Banana = banana.Banana
Then for sake of clarity and not trying to figure out what's going on
I struggled with the title for this question so let me just lay out the code:
File A:
class SomeClass(Base):
__tablename__ = 'some_classes'
id = Column(Integer, primary_key=True)
my_awesome_property = Column(Unicode(255))
other_class = relationship('OtherClass', backref='some_class', uselist=False)
File B:
class OtherClass(Base):
__tablename__ = 'other_classes'
id = Column(Integer, primary_key=True)
my_sweet_property = Column(Unicode(255))
some_class_id = Column(ForeignKey('some_classes.id'))
Now, in many cases I would refer to both of these files from a "higher-order" file containing some functions like so:
Higher Order File:
from model.alpha import SomeClass
from model.bravo import OtherClass
from sqlalchemy.orm import sessionmaker
session = sessionmaker(bind=some_engine)()
def some_random_query():
return session.query(SomeClass).join(OtherClass).filter(OtherClass.my_sweet_property=='Mike Bayer\'s cat speaks SQL.').first()
So that's pretty normal, nothing wrong with that... until... I decide I want to a put a function into one of the lower-level files like File A (and avoid circular imports)
Back to File A:
# pretend I imported a session here
def frustrating_situation():
session.query(SomeClass).join(SomeClass.other_class).filter(SomeClass.other_class.my_sweet_property=='Get ready for an exception!').first()
This will throw this bad boy right here:
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object associated with SomeClass.other_class has an attribute 'my_sweet_property'
Now I suppose that makes sense given what I know of the internals of SQLAlchemy, but I also think from an API standpoint that statement should really work.
Here is how I worked around it:
session.query(SomeClass).join(SomeClass.other_class).filter(SomeClass.other_class.property.mapper.c.my_sweet_property == 'verbose, yet it works as desired').first()
So after all that, my question is really quite simple: Does anybody know a better / more idiomatic / proper / less dirty feeling way of doing this?
Suggestions welcome.
Side Note
For anyone who is wondering:
"Why not just import the class you want the reference to for the join/filter operation?"
There are a couple reasons why you might not want to / be able to import the class into the module where you are writing the query.
You have split up your class definitions across many files and decided to avoid circular imports by strictly not importing across same-level modules
You have decided to place functions that operate on 1 or more classes not currently defined or imported in the current module and do not wish to import them because they are not used for any other reason in the module (and see reason 1 again).
SomeClass.other_class.my_sweet_property
doesn't work in sqlalchemy. sorry.
You are referring to OtherClass, in this .filter() clause. how you arrive at that name is your business, but the clearest way, from the point of view of what each statement means and where the arguments come from is still just to import things.
edit: A common cause of problems with circular imports occurs when you try to import the names out of modules directly instead of just importing the modules. If you turn code that looks like:
from foo import Bar
def baz():
Bar.quux()
you'll have an import problem if foo is also trying to import this module (say, because it wants to use baz).
Fix it by importing only the module:
import foo
def baz()
foo.Bar.quux()
since foo.Bar is resolved later, only when baz() is called, you don't have any trouble when this module gets imported, since it doesn't actually try to use the contents of any of the modules it imports.