How would you simulate the running of multiple test servers in a Django unittest?
I have a common code-base that powers two different sites with a separate database and settings.py file. I'm trying to unittest a tool that imports data from one site to another.
The default unittest setup supports multiple sqlite3 databases, so I can confirm the data on one site gets into the other's database, but I'm not sure how to confirm the data in the UI on the other site, since the default client only seems to support a single server/port.
You probably no longer need an answer to this question
I think for your specific case you don't need to have multiple servers, just multiple tests - first testing that the output from the exporting server meets your assumption, and second testing that this assumed output is successfully imported into the database. In Django unit tests the database will be empty at the start of each test
You can generate large datasets in Django using Django Fixtures. Worth noting as you should consider whether you can use their existing tools for the export/import
Related
I'm facing a problem which I have no more ideas how to resolve.
I have need to test data which is returned by direct query from database.
During execution of TestCase django.db.connection.cursor() is returning data from main database, not from test one, which contain fixtures prepared for this test purposes.
I've tried to use both TestCase and TransactionalTestCase.
I've tried debugging, checking variables values and found out that connection is pointed onto test database for sure.
Do you know why it's returning data from main database? Is there any case when Django is copying data from main database to this created for tests purposes?
I'm using: Python 3.6.5, Django 2.1, pytest 4.6.3, pytest-django 3.5
Thanks in advance for any support.
Follow-up
Dears,
That problem occurs only while trying to perform custom raw SQL Query directly inside Test Case. Retrieving objects by standard Django QuerySets works fine.
Do you have any idea why this specific way to retrieve data from db not working during test execution?
I found an answer in Django documentation:
If your code attempts to access the database when its modules are compiled, this will occur before the test database is set up, with potentially unexpected results.
Still - if you know any way to avoid affecting tests by production data while raw sql queries are performed I would love to know how to do it.
Basically I have a program which scraps some data from a website, I need to either print it out to a django template or to REST API without using a database. How do I do this without a database?
Your best bet is to
a.) Perform the scraping in views themselves, and pass the info in a context dict to the template
or
b.) Write to a file and have your view pull info from the file.
Django can be run without a database, but it depends on what applications you enable. Some of the default functionality (auth, sites, contenttypes) requires a database. So you'd need to disable those. If you need to use them, you're SOL.
Other functionality (like sessions) usually uses a database, but you can configure it to use a cache or file or something else.
I've taken two approaches in the past:
1) Disable the database completely and disable the applications that require the database:
DATABASES = {}
2) Use a dummy sqlite database just so it works out of box with the default apps without too much tweaking, but don't really use it for anything. I find this method faster and good for setting up quick testing/prototyping.
And to actually get the data from the scraper into your view, you can take a number of approaches. Store the data in a cache, or just write it directly to your context variables, etc.
I have a file with a bunch of data common between several projects. The data needs to be loaded into the Django database. The file doesn't change that much, so loading it once on server start is sufficient. Since the file is shared between multiple projects, I do not have full control over the format, so I cannot convert this into a fixture or something.
I tried loading it in ready(), but then I run into a problem when creating a new database or migrating an existing database, since apparently ready() is called before migrations are complete and I get errors from using models that do not have underlying tables. I tried to set it in class_prepared signal handler, but the loading process uses more than one model, so I cannot really be sure all required model classes are prepared. Also it seems that ready() is not called when running tests, so unit tests fail because the data is missing. What is the right place to do something like this?
It seems that what I am looking for doesn't exist. Django trusts the user to deal with migrations and such and doesn't check the database on load. So there is no place in the system where you can load some data on system start and be sure that you can actually load it. What I ended up doing is loading the data in ready(), but do a sanity check first by doing MyModel.objects.exist() in a try: except: block and returning if there was an exception. This is not ideal, but I haven't found any other way.
I have a setup where I will have having around 50 sites using the same code, configured slightly differently. Rather than deploy the same code over and over again, duplicated across different folders and repositories, is there any way in Flask to centralise the working code of the site, as some sort of library?
In Django they have something a bit like this:
https://docs.djangoproject.com/en/dev/ref/contrib/sites/
Some ideas
Deploy 50 instances of UWSGI, duplicating the same code and different config
Deploy 50 instances of UWSGI, with the python code added as a sort of module or extension so there is only one instance of the code: http://flask.pocoo.org/docs/extensiondev/
Deploy 1 instance of UWSGI which has only one instance of the code and handles different hostnames: http://flask.pocoo.org/docs/patterns/appdispatch/
The code that I am duplicating is designed to query an API and show the results. The differences between the sites are two fold:
Templating - Although the sites will look similar, they will not be identical. They will have slightly different CSS and images.
The API query. Most of the smaller sites are for towns and cities. This means that the API request from these sites will be slightly modified so as to return results only in that area.
sitelondon.com might query the API for items only in London by default
sitehtml.com might query the API for items which have the "html" keyword by default
The focus on my end is performance for the user. I will be running these initially on a server with 2GB RAM which should be plenty.
Normally for these types of scenarios the "behaves differently based on Host" logic is built in the application.
So the better solution is #3 but my suggestion is to not use app dispatch.
Build the logic to get the configuration for the hostname directly in the main application (for example you can load the specialized configurations in a #before_request handler and use a single DB instance).
If you plan to use only one small server, as you said, this solution is light on resource.
50 different uWSGI instances with their own processes would fill your memory and start swapping easily.
I agree with #Paolo that #3 is your best option.
You could simplify it even more with URL rewriting in your web server. If you rewrite the URLs such that a query for http://sitelondon.com/example becomes http://sitelondon.com/london/example, and a query for http://sitehtml.com/example becomes http://sitehtml.com/html/example then you can easily get the site through the routes:
#app.route('/<site>/example')
def example(site):
return render_template(site + '/example.html')
With this setup you can organize your templates in sub-folders based on the site name, and then selecting the proper template becomes a matter of building the template path.
I hope this helps!
I need to populate my database with a bunch of dummy entries (around 200+) so that I can test the admin interface I've made and I was wondering if there was a better way to do it. I spent the better part of my day yesterday trying to fill it in by hand (i.e by wrapping stuff like this my_model(title="asdfasdf", field2="laksdj"...) in a bunch of "for x in range(0,200):" loops) and gave up because it didn't work the way I expected it to. I think this is what I need to use, but don't you need to have (existing) data in the database for this to work?
Check this app
https://github.com/aerosol/django-dilla/
Let's say you wrote your blog application (oh yeah, your favorite!) in Django. Unit tests went fine, and everything runs extremely fast, even those ORM-generated ultra-long queries. You've added several categorized posts and it's still stable as a rock. You're quite sure the app is efficient and ready to for live deployment. Right? Wrong.
You can use fixtures for this purpose, and the loaddata management command.
One approach is to do it like this.
Prepare your test database.
Use dumpdata to create JSON export of the database.
Put this in the fixtures directory of your application.
Write your unit tests to load this "fixture": https://docs.djangoproject.com/en/2.2/topics/testing/tools/#django.test.TransactionTestCase.fixtures
Django fixtures provide a mechanism for importing data on syncdb. However, doing this initial data propagation is often easier via Python code. The technique you outline should work, either via syncdb or a management command. For instance, via syncdb, in my_app/management.py:
def init_data(sender, **kwargs):
for i in range(1000):
MyModel(number=i).save()
signals.post_syncdb.connect(init_data)
Or, in a management command in myapp/management/commands/my_command.py:
from django.core.management.base import BaseCommand, CommandError
from models import MyModel
class MyCommand(BaseCommand):
def handle(self, *args, **options):
if len(args) > 0:
raise CommandError('need exactly zero arguments')
for i in range(1000):
MyModel(number=i).save()
You can then export this data to a fixture, or continue importing using the management command. If you choose to continue to use the syncdb signal, you'll want to conditionally run the init_data function to prevent the data getting imported on subsequent syncdb calls. When a fixture isn't sufficient, I personally like to do both: create a management command to import data, but have the first syncdb invocation do the import automatically. That way, deployment is more automated but I can still easily make modifications to the initial data and re-run the import.
I'm not sure why you require any serialization. As long as you have setup your Django settings.py file to point to your test database, populating a test database should be nothing more than saving models.
for x in range(0, 200):
m = my_model(title=random_title(), field2=random_string(), ...)
m.save()
There are better ways to do this, but if you want a quick test set, this is the way to go.
The app recommended by the accepted answer is no longer being maintained however django-seed can be used as a replacement:
https://github.com/brobin/django-seed
I would recommend django-autofixtures to you. I tried both django_seed and django-autofixtures, but django_seed has a lot of issues with unique keys.
django-autofixtures takes care of unique, primary and other db constraints while filling up the database