We Keep Coding

Showing data on frontend from sql databases using django

For fetching data from databases, I usually see people create a class in models.py and they migrate it, and then they fetch data from databases and show it on the front-end.
my question is, i already have a database which has student_info table and has lots of data.
I don't want to define student_info class in models.py and do migration. Instead, since i already have student_info data in sql database. so I want to fetch it directly. can i do so?
one method i know of doing the same, is directly connecting sql server in views.py as;
mydb=mysql.connector.connect(host="localhost",user="root",password="******",database=database_name)
and fetching the data and passing it as variable in index.html. But the problem is, it fetches data only once, and if i refresh the page where local server is running, all the content will disappear, and server will report Broken pipe from ('127.0.0.1', 59579) and to see the data, again i need to restart the sever.
Kindly looking for a better approach to show data on front-end from the existing databases

What you could do is a simple thing for the solution.
Create a StudentData(models.Model) class which will store your data in them as model objects And define all the attributes/ columns in models according to the data and which is in the student_info table.
Write a simple script ( using pymysql ) to retrive all the data from the student_table , store it in text file . ( This part is easy )
take that text file and now in your django project , run a shell command or create a new script to transfer the text_file student data to model objects and save them .
Now you have your data .
I think this will work fine .
If you also want code . I'll include it.
The code for the solution :
An Update : I have used pymysql for this scripts , i think mysqlconnector will also do the work.
Create a folder inside your project named "Creating Database model" ( you can choose whatever you want but I created the files and all script inside this folder )
first_file : "data retreiving.py"
import pymysql as psql
import os
connection = psql.connect(
user = "root" ,
password = os.environ.get("MYSQL_PASSWORD") ,
database = "college" ,
)
cursor = connection.cursor()
cursor.execute("select * from student") ;
data_text_file = open("Creating Database model/text_data.txt" , "w")
for row in cursor :
temp_string = ""
for data in row :
temp_string+= "{} ".format(str(data))
data_text_file.write(temp_string+"\n")
data_text_file.close()
Enter your password for connection. Here I am using college database which has student table with some student data .
Then I am creating a file text_data.txt which will have the text data of the table .
I used string formatting here for convinence and better retrival of data on the other side.
My student table schema has five fields :
id , first_name, last_name , age , branch
so my models.py according to this database is here . You create your own according to your requirement.
from django.db import models
class StudentData(models.Model) :
first_name = models.CharField(max_length = 100 , required = True)
last_name = models.CharField(max_length = 100 , required = True)
age = models.IntegerField(max_length = 100 , required = True)
branch = models.CharField(max_length = 100 , required = True)
def __str__(self) :
return "{} {}".format(first_name , last_name)
second file : "data entry in models.py"
This file takes data and then saves it to the model . And you get your database in django .
with open("Creating Database model/text_data.txt") as f :
data = f.read()
new_data_list = []
data = data.split("\n")
new_data_list = []
for i in data :
temp_list = i.split(" ")[:-1]
new_data_list.append(temp_list)
data = new_data_list[:-1]
for i in data :
student_info = StudentModel(
first_name = i[1] ,
last_name = i[2] ,
age = i[3] ,
branch = i[4]
)
studnet_info.save()
These all are within the "Creating Database model" folder.
This will do all the work you require. Make sure to debug and understand before committing to database.

Do i need to call create tables every time in peewee?

I working in a project in which i have different projects with the same database architecture,
so i used peewee Model in which:
dynamic_db = SqliteDatabase(None)
class BaseModel(Model):
class Meta:
database = dynamic_db
class KV (BaseModel):
key = TextField()
value = IntegerField()
And whenever i new project is created i will call a function
dynamic_db.init(r'{}\database.db'.format(ProjectName.upper()))
dynamic_db.connect()
dynamic_db.create_tables([KV])
dynamic_db.close()
The problem is that once this database is created, i can't access with peewee.
When i try to create a record:
KV.create(key = 'Saul', value = 123)
I get this error:
peewee.InterfaceError: Error, database must be initialized before opening a connection.
I would appreciate any help or cookbook for peewee.

I believe something is incorrect, either in your question description, or in the error you are receiving. The call you are making to .init() is what initializes the database. After that, you should have no problems using it.
Full example which works fine:
from peewee import *
db = SqliteDatabase(None)
class Base(Model):
class Meta:
database = db
class KV(Base):
key = TextField()
value = IntegerField()
db.init('foo.db') # database is now initialized
db.connect()
db.create_tables([KV]) # no problems.
db.close()

I was finally able to create a record.
I didn't mention that i was trying to create them in another file, but the procedure is the same as the one coleifer posted on the answer.
The file in which i create the peewee models is databases.py, so in the other file i do the following:
import databases
databases.db.init('foo.db')
databases.KV.create(name = 'Saul', value= 123)
Thanks!

How to programmatically generate the CREATE TABLE SQL statement for a given model in Django?

I need to programmatically generate the CREATE TABLE statement for a given unmanaged model in my Django app (managed = False)
Since i'm working on a legacy database, i don't want to create a migration and use sqlmigrate.
The ./manage.py sql command was useful for this purpose but it has been removed in Django 1.8
Do you know about any alternatives?

As suggested, I post a complete answer for the case, that the question might imply.
Suppose you have an external DB table, that you decided to access as a Django model and therefore have described it as an unmanaged model (Meta: managed = False).
Later you need to be able to create it in your code, e.g for some tests using your local DB. Obviously, Django doesn't make migrations for unmanaged models and therefore won't create it in your test DB.
This can be solved using Django APIs without resorting to raw SQL - SchemaEditor. See a more complete example below, but as a short answer you would use it like this:
from django.db import connections
with connections['db_to_create_a_table_in'].schema_editor() as schema_editor:
schema_editor.create_model(YourUnmanagedModelClass)
A practical example:
# your_app/models/your_model.py
from django.db import models
class IntegrationView(models.Model):
"""A read-only model to access a view in some external DB."""
class Meta:
managed = False
db_table = 'integration_view'
name = models.CharField(
db_column='object_name',
max_length=255,
primaty_key=True,
verbose_name='Object Name',
)
some_value = models.CharField(
db_column='some_object_value',
max_length=255,
blank=True,
null=True,
verbose_name='Some Object Value',
)
# Depending on the situation it might be a good idea to redefine
# some methods as a NOOP as a safety-net.
# Note, that it's not completely safe this way, but might help with some
# silly mistakes in user code
def save(self, *args, **kwargs):
"""Preventing data modification."""
pass
def delete(self, *args, **kwargs):
"""Preventing data deletion."""
pass
Now, suppose you need to be able to create this model via Django, e.g. for some tests.
# your_app/tests/some_test.py
# This will allow to access the `SchemaEditor` for the DB
from django.db import connections
from django.test import TestCase
from your_app.models.your_model import IntegrationView
class SomeLogicTestCase(TestCase):
"""Tests some logic, that uses `IntegrationView`."""
# Since it is assumed, that the `IntegrationView` is read-only for the
# the case being described it's a good idea to put setup logic in class
# setup fixture, that will run only once for the whole test case
#classmethod
def setUpClass(cls):
"""Prepares `IntegrationView` mock data for the test case."""
# This is the actual part, that will create the table in the DB
# for the unmanaged model (Any model in fact, but managed models will
# have their tables created already by the Django testing framework)
# Note: Here we're able to choose which DB, defined in your settings,
# will be used to create the table
with connections['external_db'].schema_editor() as schema_editor:
schema_editor.create_model(IntegrationView)
# That's all you need, after the execution of this statements
# a DB table for `IntegrationView` will be created in the DB
# defined as `external_db`.
# Now suppose we need to add some mock data...
# Again, if we consider the table to be read-only, the data can be
# defined here, otherwise it's better to do it in `setUp()` method.
# Remember `IntegrationView.save()` is overridden as a NOOP, so simple
# calls to `IntegrationView.save()` or `IntegrationView.objects.create()`
# won't do anything, so we need to "Improvise. Adapt. Overcome."
# One way is to use the `save()` method of the base class,
# but provide the instance of our class
integration_view = IntegrationView(
name='Biggus Dickus',
some_value='Something really important.',
)
super(IntegrationView, integration_view).save(using='external_db')
# Another one is to use the `bulk_create()`, which doesn't use
# `save()` internally, and in fact is a better solution
# if we're creating many records
IntegrationView.objects.using('external_db').bulk_create([
IntegrationView(
name='Sillius Soddus',
some_value='Something important',
),
IntegrationView(
name='Naughtius Maximus',
some_value='Whatever',
),
])
# Don't forget to clean after
#classmethod
def tearDownClass(cls):
with connections['external_db'].schema_editor() as schema_editor:
schema_editor.delete_model(IntegrationView)
def test_some_logic_using_data_from_integration_view(self):
self.assertTrue(IntegrationView.objects.using('external_db').filter(
name='Biggus Dickus',
))
To make the example more complete... Since we're using multiple DB (default and external_db) Django will try to run migrations on both of them for the tests and as of now there's no option in DB settings to prevent this. So we have to use a custom DB router for testing.
# your_app/tests/base.py
class PreventMigrationsDBRouter:
"""DB router to prevent migrations for specific DBs during tests."""
_NO_MIGRATION_DBS = {'external_db', }
def allow_migrate(self, db, app_label, model_name=None, **hints):
"""Actually disallows migrations for specific DBs."""
return db not in self._NO_MIGRATION_DBS
And a test settings file example for the described case:
# settings/test.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'db_name',
'USER': 'username',
'HOST': 'localhost',
'PASSWORD': 'password',
'PORT': '1521',
},
# For production here we would have settings to connect to the external DB,
# but for testing purposes we could get by with an SQLite DB
'external_db': {
'ENGINE': 'django.db.backends.sqlite3',
},
}
# Not necessary to use a router in production config, since if the DB
# is unspecified explicitly for some action Django will use the `default` DB
DATABASE_ROUTERS = ['your_app.tests.base.PreventMigrationsDBRouter', ]
Hope this detailed new Django user user-friendly example will help someone and save their time.

unfortunately there seems to be no easy way to do this, but for your luck I have just succeeded in producing a working snippet for you digging in the internals of the django migrations jungle.
Just:
save the code to get_sql_create_table.py (in example)
do $ export DJANGO_SETTINGS_MODULE=yourproject.settings
launch the script with python get_sql_create_table.py yourapp.yourmodel
and it should output what you need.
Hope it helps!
import django
django.setup()
from django.db.migrations.state import ModelState
from django.db.migrations import operations
from django.db.migrations.migration import Migration
from django.db import connections
from django.db.migrations.state import ProjectState
def get_create_sql_for_model(model):
model_state = ModelState.from_model(model)
# Create a fake migration with the CreateModel operation
cm = operations.CreateModel(name=model_state.name, fields=model_state.fields)
migration = Migration("fake_migration", "app")
migration.operations.append(cm)
# Let the migration framework think that the project is in an initial state
state = ProjectState()
# Get the SQL through the schema_editor bound to the connection
connection = connections['default']
with connection.schema_editor(collect_sql=True, atomic=migration.atomic) as schema_editor:
state = migration.apply(state, schema_editor, collect_sql=True)
# return the CREATE TABLE statement
return "\n".join(schema_editor.collected_sql)
if __name__ == "__main__":
import importlib
import sys
if len(sys.argv) < 2:
print("Usage: {} <app.model>".format(sys.argv[0]))
sys.exit(100)
app, model_name = sys.argv[1].split('.')
models = importlib.import_module("{}.models".format(app))
model = getattr(models, model_name)
rv = get_create_sql_for_model(model)
print(rv)

For Django v4.1.3, the above get_create_sql_for_model soruce code changed like this:
from django.db.migrations.state import ModelState
from django.db.migrations import operations
from django.db.migrations.migration import Migration
from django.db import connections
from django.db.migrations.state import ProjectState
def get_create_sql_for_model(model):
model_state = ModelState.from_model(model)
table_name = model_state.options['db_table']
# Create a fake migration with the CreateModel operation
cm = operations.CreateModel(name=model_state.name, fields=model_state.fields.items())
migration = Migration("fake_migration", "app")
migration.operations.append(cm)
# Let the migration framework think that the project is in an initial state
state = ProjectState()
# Get the SQL through the schema_editor bound to the connection
connection = connections['default']
with connection.schema_editor(collect_sql=True, atomic=migration.atomic) as schema_editor:
state = migration.apply(state, schema_editor, collect_sql=True)
sqls = schema_editor.collected_sql
items = []
for sql in sqls:
if sql.startswith('--'):
continue
items.append(sql)
return table_name,items
#EOP
I used it to create all tables (like the command syncdb of old Django version):
for app in settings.INSTALLED_APPS:
app_name = app.split('.')[0]
app_models = apps.get_app_config(app_name).get_models()
for model in app_models:
table_name,sqls = get_create_sql_for_model(model)
if settings.DEBUG:
s = "SELECT COUNT(*) AS c FROM sqlite_master WHERE name = '%s'" % table_name
else:
s = "SELECT COUNT(*) AS c FROM information_schema.TABLES WHERE table_name='%s'" % table_name
rs = select_by_raw_sql(s)
if not rs[0]['c']:
for sql in sqls:
exec_by_raw_sql(sql)
print('CREATE TABLE DONE:%s' % table_name)
The full soure code can be found at Django syncdb command came back for v4.1.3 version

peewee - Define models separately from Database() initialization

I need to use some ORM engine, like peewee, for handling SQLite database within my python application. However, most of such libraries offer syntax like this to define models.py:
import peewee
db = peewee.Database('hello.sqlite')
class Person(peewee.Model):
name = peewee.CharField()
class Meta:
database = db
However, in my application, i cannot use such syntax since database file name is provided by outside code after import, from module, which imports my models.py.
How to initialize models from outside of their definition knowing dynamic database file name? Ideally, models.py should not contain "database" mentions at all, like normal ORM.

Maybe you are looking at proxy feature :
proxy - peewee
database_proxy = Proxy() # Create a proxy for our db.
class BaseModel(Model):
class Meta:
database = database_proxy # Use proxy for our DB.
class User(BaseModel):
username = CharField()
# Based on configuration, use a different database.
if app.config['DEBUG']:
database = SqliteDatabase('local.db')
elif app.config['TESTING']:
database = SqliteDatabase(':memory:')
else:
database = PostgresqlDatabase('mega_production_db')
# Configure our proxy to use the db we specified in config.
database_proxy.initialize(database)

Refresh mongodb collection structure through python mongoengine

I'm writing a simple Flask app, with the sole purpose to learn Python and MongoDB.
I've managed to reach to the point where all the collections are defined, and CRUD operations work in general. Now, one thing that I really want to understand, is how to refresh the collection, after updating its structure. For example, say that I have the following model:
user.py
class User(db.Document, UserMixin):
email = db.StringField(required=True, unique=True)
password = db.StringField(required=True)
active = db.BooleanField()
first_name = db.StringField(max_length=64, required=True)
last_name = db.StringField(max_length=64, required=True)
registered_at = db.DateTimeField(default=datetime.datetime.utcnow())
confirmed = db.BooleanField()
confirmed_at = db.DateTimeField()
last_login_at = db.DateTimeField()
current_login_at = db.DateTimeField()
last_login_ip = db.StringField(max_length=45)
current_login_ip = db.StringField(max_length=45)
login_count = db.IntField()
companies = db.ListField(db.ReferenceField('Company'), default=[])
roles = db.ListField(db.ReferenceField(Role), default=[])
meta = {
'indexes': [
{'fields': ['email'], 'unique': True}
]
}
Now, I already have entries in my user collection, but I want to change companies to:
company = db.ReferenceField('Company')
How can I refresh the collection's structure, without having to bring the whole database down?
I do have a manage.py script that helps me and also provides a shell:
#!/usr/bin/python
from flask.ext.script import Manager
from flask.ext.script.commands import Shell
from app import factory
app = factory.create_app()
manager = Manager(app)
manager.add_command("shell", Shell(use_ipython=True))
# manager.add_command('run_tests', RunTests())
if __name__ == "__main__":
manager.run()
and I have tried a couple of commands, from information that I could recompile and out of my basic knowledge:
>>> from app.models import db, User
>>> import mongoengine
>>> mongoengine.Document(User)
field = iter(self._fields_ordered)
AttributeError: 'Document' object has no attribute '_fields_ordered'
>>> mongoengine.Document(User).modify() # well, same result as above
Any pointers on how to achieve this?
Update
I am asking all of this, because I have updated my user.py to match my new requests, but anytime I interact with the db its self, since the table's structure was not refreshed, I get the following error:
FieldDoesNotExist: The field 'companies' does not exist on the
document 'User', referer: http://local.faqcolab.com/company

Solution is easier then I expected:
db.getCollection('user').update(
// query
{},
// update
{
$rename: {
'companies': 'company'
}
},
// options
{
"multi" : true, // update all documents
"upsert" : false // insert a new document, if no existing document match the query
}
);
Explanation for each of the {}:
First is empty because I want to update all documents in user collection.
Second contains $rename which is the invoking action to rename the fields I want.
Last contains aditional settings for the query to be executed.

I have updated my user.py to match my new requests, but anytime I interact with the db its self, since the table's structure was not refreshed, I get the following error
MongoDB does not have a "table structure" like relational databases do. After a document has been inserted, you can't change it's schema by changing the document model.
I don't want to sound like I'm telling you that the answer is to use different tools, but seeing things like db.ListField(db.ReferenceField('Company')) makes me think you'd be much better off with a relational database (Postgres is well supported in the Flask ecosystem).
Mongo works best for storing schema-less documents (you don't know before hand how your data is structured, or it varies significantly between documents). Unless you have data like that, it's worth looking at other options. Especially since you're just getting started with Python and Flask, there's no point in making things harder than they are.