I have an application that has some unmanaged models that pull from a table in a different DB.
I've created Factories for all my models and in most cases factory_boy has worked great!
The problem I'm running into has to do with a specific unmanaged model that uses a manager on one of the fields.
class DailyRoll(StaffModel):
id = models.IntegerField(db_column='ID', primary_key=True)
location_id = models.IntegerField(db_column='locationID')
grade = models.CharField(db_column='grade', max_length=10)
end_year = models.IntegerField(db_column='endYear')
run_date = models.DateField(db_column='runDate')
person_count = models.IntegerField(db_column='personCount')
contained_count = models.IntegerField(db_column='containedCount')
double_count = models.IntegerField(db_column='doubleCount')
aide_count = models.IntegerField(db_column='aideCount')
one_count = models.IntegerField(db_column='oneCount')
bi_count = models.IntegerField(db_column='biCount')
last_run_date = managers.DailyRollManager()
class Meta(object):
managed = False
db_table = '[DATA].[position_dailyRoll]'
The manager looks like this:
class DailyRollManager(models.Manager):
def get_queryset(self):
last_run_date = super().get_queryset().using('staff').last().run_date
return super().get_queryset().using('staff').filter(
run_date=last_run_date
)
My factory_boy setup looks like this (it's very basic at the moment, because I'm just trying to get it to work):
class DailyRollFactory(factory.Factory):
class Meta:
model = staff.DailyRoll
id = 1
location_id = 1
grade = 1
end_year = 2019
run_date = factory.fuzzy.FuzzyDateTime(timezone.now())
person_count = 210
contained_count = 1
double_count = 2
aide_count = 3
one_count = 4
bi_count = 5
last_run_date = managers.DailyRollManager()
I'm sure the last_run_date is setup improperly - but I don't know how best to handle using a manager in factory_boy.
When I run my tests - it creates a default sqlite3 database named 'defaultdb' to run tests against (we use SQL Server in production - but because of security we cannot give django control over master to manage testing). To accomplish this I had to create a workaround, but it works.
My settings look like this:
if 'test' in sys.argv or 'test_coverage' in sys.argv:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'defaultdb.sqlite3'),
},
'staff': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'staff.sqlite3'),
},
}
My test is simple like this:
def test_index_returns_200_and_correct_template(self):
"""
index page returns a 200
"""
# Load test data which index needs
# if this is not included the test fails with
# 'no such table: [DATA].[position_dailyRoll]'
f.DailyRollFactory()
response = self.client.get(reverse('position:index'))
self.assertEqual(response.status_code, 200)
self.assertTemplateUsed(response, 'position/index.html')
If I don't include f.DailyRollFactory() in the test it fails with this error:
no such table: [DATA].[position_dailyRoll]
When I include f.DailyRollFactory() it gives this error:
TypeError: 'last_run_date' is an invalid keyword argument for this function
I think that ultimately that manager is trying to look at the 'staff' database - which I have being created, but it's empty and contains no data.
I am trying to figure out:
A) How can I pre-populate that staff database with data like what I have in my factory-boy model? I think that may resolve the issue I'm running into. Will it? I am not sure.
B) Is that the right way to handle the manager in a factory_boy Factory? I'm not sure exactly how to handle that.
I ended up using Pytest-Django as that testing framework actually allows for doing exactly what I wanted.
Using the --nomigrations flag, it takes my models which are only managed by django in tests and creates the appropriate table name for them (using their db_table attribute) in the test database. Then I can use factory_boy to create mock data and test it up!
Related
I have a mongodb database named as world, which has two collection from before city, languages. I want to show the data of my collection on the web, how can i do it.
currently i know to create collection in models.py and migrate it. like;
first we have to edit databases[] in setting.py
DATABASES = {
'default': {
'ENGINE': 'djongo',
'NAME': 'world',
}
}
in models.py i creted a class and migrated it using python manage.py migrate
class Destination(models.Model):
name= models.CharField(max_length=100)
img=models.ImageField(upload_to='pics')
desc=models.TextField()
and i'm able to retrieve data from Destination by using below code in views.py
from django.shortcuts import render
from .models import Destination
def index(request):
dests=Destination.objects.all()
return render(request,'index.html',{'dests':dests})
My question is my collection/class is already available in the database ( city, language) and i'm not creating it contrary to Destination which was defined by me. then how to show data of city collection of world database on the front-end.
kindly looking for help.
If I got you properly, then you have a MongoDB database called world.
There you stored city and languages before you started to set up Django.
Then you added a Destination model, thus created a new collection.
For now, you're looking for a way how to get city and languages collections data similar way as you do with Destination.
So there are multiple ways how you could handle it:
Create Django models for city and languages collections (define fields that you have in existing collections):
class City(models.Model):
field1 = ...
field2 = ...
class Meta:
db_table = 'city' # important, should be your existing collection name
class Language(models.Model):
field3 = ...
field4 = ...
class Meta:
db_table = 'languages' # important, should be your existing collection name
Now you're ready to use City and Language the same way as you do with the Destination model.
Use PyMongo (this is already installed as you're using Djongo). So your snipped will look something like:
from django.shortcuts import render
from .models import Destination
import pymongo
# default localhost connection URL
MONGO_URL = 'mongodb://localhost:27017'
connection = pymongo.MongoClient(MONGO_URL)
mongo_db = connection.world
def index(request):
collection_city = db['city']
collection_languages = db['languages']
cities = list(collection_city.find())
languages = list(collection_languages.find())
dests=Destination.objects.all()
return render(
request,
'index.html',
{
'dests': dests,
'cities': cities,
'languages': languages
}
)
I'd use option 1, as it allows you to keep the project coherent.
I have a web application in Python django. I need to import users and display data about them from another database, from another existing application. All I need is the user to be able to login and display information about them. What solutions are?
You can set 2 DATABASES in settings.py.
DATABASES = {
'default': {
...
},
'user_data': {
...
}
}
Then in one database store User models with authentication and stuff, in another rest information. You can connect information about specific User with a field that is storing id of User from another database.
If you have multiple databases and create a model, you should declare on which db it is going to be stored. If you didn't, it will be in default one (if you have it declared).
class UserModel(models.Model):
class Meta:
db_table = 'default'
class UserDataModel(models.Model):
class Meta:
db_table = 'user_data'
the answer from #NixonSparrow was wrong.
_meta.db_table defined only table_name in database and not the database self.
for switch database you can use manager.using('database_name'), for every model, it is good declared here: https://docs.djangoproject.com/en/4.0/topics/db/multi-db/#topics-db-multi-db-routing
in my project i use multiple router.
https://docs.djangoproject.com/en/4.0/topics/db/multi-db/#topics-db-multi-db-routing
it help don't override every manager with using. But in your case:
DATABASES = {
'default': {
...
},
'other_users_data': {
...
}
}
and somethere in views:
other_users = otherUserModel.objects.using('other_users_data')
Probably, otherUserModel should define in meta, which table you want to use db_table = 'other_users_table_name' and also probably it should have managed=False, to hide this model from migration manager.
I need to programmatically generate the CREATE TABLE statement for a given unmanaged model in my Django app (managed = False)
Since i'm working on a legacy database, i don't want to create a migration and use sqlmigrate.
The ./manage.py sql command was useful for this purpose but it has been removed in Django 1.8
Do you know about any alternatives?
As suggested, I post a complete answer for the case, that the question might imply.
Suppose you have an external DB table, that you decided to access as a Django model and therefore have described it as an unmanaged model (Meta: managed = False).
Later you need to be able to create it in your code, e.g for some tests using your local DB. Obviously, Django doesn't make migrations for unmanaged models and therefore won't create it in your test DB.
This can be solved using Django APIs without resorting to raw SQL - SchemaEditor. See a more complete example below, but as a short answer you would use it like this:
from django.db import connections
with connections['db_to_create_a_table_in'].schema_editor() as schema_editor:
schema_editor.create_model(YourUnmanagedModelClass)
A practical example:
# your_app/models/your_model.py
from django.db import models
class IntegrationView(models.Model):
"""A read-only model to access a view in some external DB."""
class Meta:
managed = False
db_table = 'integration_view'
name = models.CharField(
db_column='object_name',
max_length=255,
primaty_key=True,
verbose_name='Object Name',
)
some_value = models.CharField(
db_column='some_object_value',
max_length=255,
blank=True,
null=True,
verbose_name='Some Object Value',
)
# Depending on the situation it might be a good idea to redefine
# some methods as a NOOP as a safety-net.
# Note, that it's not completely safe this way, but might help with some
# silly mistakes in user code
def save(self, *args, **kwargs):
"""Preventing data modification."""
pass
def delete(self, *args, **kwargs):
"""Preventing data deletion."""
pass
Now, suppose you need to be able to create this model via Django, e.g. for some tests.
# your_app/tests/some_test.py
# This will allow to access the `SchemaEditor` for the DB
from django.db import connections
from django.test import TestCase
from your_app.models.your_model import IntegrationView
class SomeLogicTestCase(TestCase):
"""Tests some logic, that uses `IntegrationView`."""
# Since it is assumed, that the `IntegrationView` is read-only for the
# the case being described it's a good idea to put setup logic in class
# setup fixture, that will run only once for the whole test case
#classmethod
def setUpClass(cls):
"""Prepares `IntegrationView` mock data for the test case."""
# This is the actual part, that will create the table in the DB
# for the unmanaged model (Any model in fact, but managed models will
# have their tables created already by the Django testing framework)
# Note: Here we're able to choose which DB, defined in your settings,
# will be used to create the table
with connections['external_db'].schema_editor() as schema_editor:
schema_editor.create_model(IntegrationView)
# That's all you need, after the execution of this statements
# a DB table for `IntegrationView` will be created in the DB
# defined as `external_db`.
# Now suppose we need to add some mock data...
# Again, if we consider the table to be read-only, the data can be
# defined here, otherwise it's better to do it in `setUp()` method.
# Remember `IntegrationView.save()` is overridden as a NOOP, so simple
# calls to `IntegrationView.save()` or `IntegrationView.objects.create()`
# won't do anything, so we need to "Improvise. Adapt. Overcome."
# One way is to use the `save()` method of the base class,
# but provide the instance of our class
integration_view = IntegrationView(
name='Biggus Dickus',
some_value='Something really important.',
)
super(IntegrationView, integration_view).save(using='external_db')
# Another one is to use the `bulk_create()`, which doesn't use
# `save()` internally, and in fact is a better solution
# if we're creating many records
IntegrationView.objects.using('external_db').bulk_create([
IntegrationView(
name='Sillius Soddus',
some_value='Something important',
),
IntegrationView(
name='Naughtius Maximus',
some_value='Whatever',
),
])
# Don't forget to clean after
#classmethod
def tearDownClass(cls):
with connections['external_db'].schema_editor() as schema_editor:
schema_editor.delete_model(IntegrationView)
def test_some_logic_using_data_from_integration_view(self):
self.assertTrue(IntegrationView.objects.using('external_db').filter(
name='Biggus Dickus',
))
To make the example more complete... Since we're using multiple DB (default and external_db) Django will try to run migrations on both of them for the tests and as of now there's no option in DB settings to prevent this. So we have to use a custom DB router for testing.
# your_app/tests/base.py
class PreventMigrationsDBRouter:
"""DB router to prevent migrations for specific DBs during tests."""
_NO_MIGRATION_DBS = {'external_db', }
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""Actually disallows migrations for specific DBs."""
return db not in self._NO_MIGRATION_DBS
And a test settings file example for the described case:
# settings/test.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'db_name',
'USER': 'username',
'HOST': 'localhost',
'PASSWORD': 'password',
'PORT': '1521',
},
# For production here we would have settings to connect to the external DB,
# but for testing purposes we could get by with an SQLite DB
'external_db': {
'ENGINE': 'django.db.backends.sqlite3',
},
}
# Not necessary to use a router in production config, since if the DB
# is unspecified explicitly for some action Django will use the `default` DB
DATABASE_ROUTERS = ['your_app.tests.base.PreventMigrationsDBRouter', ]
Hope this detailed new Django user user-friendly example will help someone and save their time.
unfortunately there seems to be no easy way to do this, but for your luck I have just succeeded in producing a working snippet for you digging in the internals of the django migrations jungle.
Just:
save the code to get_sql_create_table.py (in example)
do $ export DJANGO_SETTINGS_MODULE=yourproject.settings
launch the script with python get_sql_create_table.py yourapp.yourmodel
and it should output what you need.
Hope it helps!
import django
django.setup()
from django.db.migrations.state import ModelState
from django.db.migrations import operations
from django.db.migrations.migration import Migration
from django.db import connections
from django.db.migrations.state import ProjectState
def get_create_sql_for_model(model):
model_state = ModelState.from_model(model)
# Create a fake migration with the CreateModel operation
cm = operations.CreateModel(name=model_state.name, fields=model_state.fields)
migration = Migration("fake_migration", "app")
migration.operations.append(cm)
# Let the migration framework think that the project is in an initial state
state = ProjectState()
# Get the SQL through the schema_editor bound to the connection
connection = connections['default']
with connection.schema_editor(collect_sql=True, atomic=migration.atomic) as schema_editor:
state = migration.apply(state, schema_editor, collect_sql=True)
# return the CREATE TABLE statement
return "\n".join(schema_editor.collected_sql)
if __name__ == "__main__":
import importlib
import sys
if len(sys.argv) < 2:
print("Usage: {} <app.model>".format(sys.argv[0]))
sys.exit(100)
app, model_name = sys.argv[1].split('.')
models = importlib.import_module("{}.models".format(app))
model = getattr(models, model_name)
rv = get_create_sql_for_model(model)
print(rv)
For Django v4.1.3, the above get_create_sql_for_model soruce code changed like this:
from django.db.migrations.state import ModelState
from django.db.migrations import operations
from django.db.migrations.migration import Migration
from django.db import connections
from django.db.migrations.state import ProjectState
def get_create_sql_for_model(model):
model_state = ModelState.from_model(model)
table_name = model_state.options['db_table']
# Create a fake migration with the CreateModel operation
cm = operations.CreateModel(name=model_state.name, fields=model_state.fields.items())
migration = Migration("fake_migration", "app")
migration.operations.append(cm)
# Let the migration framework think that the project is in an initial state
state = ProjectState()
# Get the SQL through the schema_editor bound to the connection
connection = connections['default']
with connection.schema_editor(collect_sql=True, atomic=migration.atomic) as schema_editor:
state = migration.apply(state, schema_editor, collect_sql=True)
sqls = schema_editor.collected_sql
items = []
for sql in sqls:
if sql.startswith('--'):
continue
items.append(sql)
return table_name,items
#EOP
I used it to create all tables (like the command syncdb of old Django version):
for app in settings.INSTALLED_APPS:
app_name = app.split('.')[0]
app_models = apps.get_app_config(app_name).get_models()
for model in app_models:
table_name,sqls = get_create_sql_for_model(model)
if settings.DEBUG:
s = "SELECT COUNT(*) AS c FROM sqlite_master WHERE name = '%s'" % table_name
else:
s = "SELECT COUNT(*) AS c FROM information_schema.TABLES WHERE table_name='%s'" % table_name
rs = select_by_raw_sql(s)
if not rs[0]['c']:
for sql in sqls:
exec_by_raw_sql(sql)
print('CREATE TABLE DONE:%s' % table_name)
The full soure code can be found at Django syncdb command came back for v4.1.3 version
I'm writing a simple Flask app, with the sole purpose to learn Python and MongoDB.
I've managed to reach to the point where all the collections are defined, and CRUD operations work in general. Now, one thing that I really want to understand, is how to refresh the collection, after updating its structure. For example, say that I have the following model:
user.py
class User(db.Document, UserMixin):
email = db.StringField(required=True, unique=True)
password = db.StringField(required=True)
active = db.BooleanField()
first_name = db.StringField(max_length=64, required=True)
last_name = db.StringField(max_length=64, required=True)
registered_at = db.DateTimeField(default=datetime.datetime.utcnow())
confirmed = db.BooleanField()
confirmed_at = db.DateTimeField()
last_login_at = db.DateTimeField()
current_login_at = db.DateTimeField()
last_login_ip = db.StringField(max_length=45)
current_login_ip = db.StringField(max_length=45)
login_count = db.IntField()
companies = db.ListField(db.ReferenceField('Company'), default=[])
roles = db.ListField(db.ReferenceField(Role), default=[])
meta = {
'indexes': [
{'fields': ['email'], 'unique': True}
]
}
Now, I already have entries in my user collection, but I want to change companies to:
company = db.ReferenceField('Company')
How can I refresh the collection's structure, without having to bring the whole database down?
I do have a manage.py script that helps me and also provides a shell:
#!/usr/bin/python
from flask.ext.script import Manager
from flask.ext.script.commands import Shell
from app import factory
app = factory.create_app()
manager = Manager(app)
manager.add_command("shell", Shell(use_ipython=True))
# manager.add_command('run_tests', RunTests())
if __name__ == "__main__":
manager.run()
and I have tried a couple of commands, from information that I could recompile and out of my basic knowledge:
>>> from app.models import db, User
>>> import mongoengine
>>> mongoengine.Document(User)
field = iter(self._fields_ordered)
AttributeError: 'Document' object has no attribute '_fields_ordered'
>>> mongoengine.Document(User).modify() # well, same result as above
Any pointers on how to achieve this?
Update
I am asking all of this, because I have updated my user.py to match my new requests, but anytime I interact with the db its self, since the table's structure was not refreshed, I get the following error:
FieldDoesNotExist: The field 'companies' does not exist on the
document 'User', referer: http://local.faqcolab.com/company
Solution is easier then I expected:
db.getCollection('user').update(
// query
{},
// update
{
$rename: {
'companies': 'company'
}
},
// options
{
"multi" : true, // update all documents
"upsert" : false // insert a new document, if no existing document match the query
}
);
Explanation for each of the {}:
First is empty because I want to update all documents in user collection.
Second contains $rename which is the invoking action to rename the fields I want.
Last contains aditional settings for the query to be executed.
I have updated my user.py to match my new requests, but anytime I interact with the db its self, since the table's structure was not refreshed, I get the following error
MongoDB does not have a "table structure" like relational databases do. After a document has been inserted, you can't change it's schema by changing the document model.
I don't want to sound like I'm telling you that the answer is to use different tools, but seeing things like db.ListField(db.ReferenceField('Company')) makes me think you'd be much better off with a relational database (Postgres is well supported in the Flask ecosystem).
Mongo works best for storing schema-less documents (you don't know before hand how your data is structured, or it varies significantly between documents). Unless you have data like that, it's worth looking at other options. Especially since you're just getting started with Python and Flask, there's no point in making things harder than they are.
In Django I would like to use Unit Test for testing a MS-SQL Server legacy database. The database is using stored procedures for adding data. The situation is as follow:
The MS-SQL database has the following settings in Django:
DATABASES['vadain_import'] = {
'ENGINE': 'sql_server.pyodbc',
'USER': 'xx',
'PASSWORD': 'xxx',
'HOST': '192.168.103.102',
'PORT': '',
'NAME': 'Vadain_import',
'OPTIONS': {
'driver': 'SQL Server Native Client 11.0',
'MARS_Connection': True,
}
}
The models of the database are made with inspectdb, example:
class WaOrders(models.Model):
order_id = models.IntegerField(primary_key=True, db_column='intOrderId')
type = models.TextField(db_column='chvType', blank=True)
class Meta:
db_table = 'WA_ORDERS'
database_name = 'vadain_import'
managed = False
# (There's a lot more of properties and models)
In models is executing the stored procedures. I cann't use the save
functionality of Django, like WAOrders.save(), because in the MS-SQL
database the primary key's are generated in the stored procedure.
#classmethod
def export(cls, **kwargs):
# Stored procedure for adding data into the db
sql = "declare #id int \
set #id=0\
declare #error varchar(1000)\
set #error=''\
exec UspWA_OrderImport\
#intOrderId=#id out\
,#chvType=N'%s'" % kwargs['type'] + " \
,#chvErrorMsg=#error output\
select #id as id, #error as 'error' \
"
# Connection Vadain db
cursor = connections['vadain_import'].cursor()
# Execute sql stored procedure, no_count is needed otherwise it's returning an error
# Return value primary key of WAOrders
try:
cursor.execute(no_count + sql)
result = cursor.fetchone()
# Check if primary key is set and if there are no errors:
if result[0] > 1 and result[1] == '':
# Commit SP
cursor.execute('COMMIT')
return result[0]
There is a mapping for creating the models, because the MS-SQL
database expect different data then the normal objects, like ‘order’.
def adding_data_into_vadain(self, order):
for curtain in order.curtains.all():
order_id = WaOrders.export(
type=format(curtain.type)
)
# relation with default and vadain db.
order.exported_id = order_id
order.save()
The function is working proper by running the program, but by running ‘manage.py test’ will be created a test databases. This is given the following problems:
By creating test database is missing the south tables (this is also not needed in the legacy database)
By changing the SOUTH_TESTS_MIGRATE to False I’m getting the error message that the tables are already exists of the default database.
My test is as follow:
class AddDataServiceTest(TestCase):
fixtures = ['order']
def test_data_service(self):
# add data into vadain_import data
for order in Order.objects.all():
AddingDataService.adding_data_into_vadain(order)
# test the exported values
for order in Order.objects.all():
exported_order = WaOrders.objects.get(order_exported_id=order.exported_id)
self.assertEqual(exported_order.type, 'Pleat curtain')
Can somebody advise me how I can test the situation?
maybe inside your WaOrders you can extend save() method and call there export function.