So, I am running a Flask web app as the frontend to a small SQLite inventory database. I am using Flask-SQLAlchemy for all of the database interaction. I have a search feature built in, so that you could (for example) find all pieces of hardware assigned to Person X. Or, find all pieces of hardware from Project ABC that have the status Available. The search feature supports optional criteria.
For security reasons, I can't post executable code here, but this is my search function:
for system in db.session.query(System).filter(and_(System.assignee_id.like("%{}%".format(assignee_id)),
System.serial.like("%{}%".format(serial)),
System.user_id.like("%{}%".format(user_id)),
System.project_id.like("%{}%".format(project_id)),
System.status_id.like("%{}%".format(status_id)))):
results.append(system)
The weird thing is, this works perfectly when I run the app in the Development config. But, in my Production config, if I try to use the 'Assignee' or 'User' criteria (both of which are strings of an email address), I get bad results. It shows me a bunch of systems, some of which have the correct assignee, but most of which don't. The only differences between the Production and Development configs are the file path to the SQLite database and the authentication mechanism to sign into the application.
Why would this return different results between the two? It works EXACTLY like it is supposed to in the Dev config.
UPDATE
After some additional debugging as suggested by the comments, it looks like the problem is in the actual data of the ProductionConfig. However, when I browse the .sqlite file in DB Browser, initially everything looks correct. Where is it getting these random assignees?
Related
I am wondering if there is a way to obtain the hostname of a Django application when running tests. That is, I would like the tests to pass both locally and when run at the staging server. Hence a need to know http://localhost:<port> vs. http://staging.example.com is needed because some tests query particular URLs.
I found answers on how to do it inside templates, but that does not help since there is no response object to check the hostname.
How can one find out the hostname outside the views/templates? Is it stored in Django settings somewhere?
Why do you need to know the hostname? Tests can run just fine without it, if you use the test client. You do not need to know anything about the system they're running on.
You can also mark tests with a tag and then have the CI system run the tests including that tag.
And finally there is the LiveServerTestCase:
LiveServerTestCase does basically the same as TransactionTestCase with one extra feature: it launches a live Django server in the background on setup, and shuts it down on teardown. This allows the use of automated test clients other than the Django dummy client such as, for example, the Selenium client, to execute a series of functional tests inside a browser and simulate a real user’s actions.
The live server listens on localhost and binds to port 0 which uses a free port assigned by the operating system. The server’s URL can be accessed with self.live_server_url during the tests.
Additional information from comments:
You can test if the URL of an image file is present in your response by testing for the MEDIA_URL:
self.assertContains(response, f'{settings.MEDIA_URL}/default-avatar.svg')
You can test for the existence of an upload in various ways, but the easiest one is to check if there's a file object associated with the FileField. It will throw ValueError if there is not.
I am working on scaling out a webapp and providing some database redundancy for protection against failures and to keep the servers up when updates are needed. The app is still in development, so I have chosen a simple multi-master redundancy with two separate database servers to try and achieve this. Each server will have the Django code and host its own database, and the databases should be as closely mirrored as possible (updated within a few seconds).
I am trying to figure out how to set up the multi-master (master-master) replication between databases with Django and MySQL. There is a lot of documentation about setting it up with MySQL only (using various configurations), but I cannot find any for making this work from the Django side of things.
From what I understand, I need to approach this by adding two database entries in the Django settings (one for each master) and then write a database router that will specify which database to read from and which to write from. In this scenario, both databases should accept both reads and writes, and writes/updates should be mirrored over to the other database. The logic in the router could simply use a round-robin technique to decide which database to use. From there on, further configuration to set up the actual replication should be done through MySQL configuration.
Does this approach sound correct, and does anyone have any experience with getting this to work?
Your idea of the router is great! I would add that you need automatically detect whether a databases is [slow] down. You can detect that by the response time and by connection/read/write errors. And if this happens then you exclude this database from your round-robin list for a while, trying to connect back to it every now and then to detect if the databases is alive.
In other words the round-robin list grows and shrinks dynamically depending on the health status of your database machines.
The another important notice is that luckily you don't need to maintain this round-robin list common to all the web servers. Each web server can store its own copy of the round-robin list and its own state of inclusion and exclusion of databases into this list. This is just because a database server can be seen from one web server and can be not seen from another one due to local network problems.
I have a existing Website deployed in Google App Engine for Python. Now I have setup the local development server in my System. But I don't know how to get the updated DataBase from live server. There is no Export option in Google's developer console.
And, I don't want to read the data for each request from Production Datastore, I want to set it up locally for once. The google manual says that it stores the local datastore in sqlite file.
Any hint would be appreciated.
First, make sure your app.yaml enables the "remote" built-in, with a stanza such as:
builtins:
- remote_api: on
This app.yaml of course must be the one deployed to your appspot.com (or whatever) "production" GAE app.
Then, it's a job for /usr/local/google_appengine/bulkloader.py or wherever you may have installed the bulkloader component. Run it with -h to get a list of the many, many options you can pass.
You may need to generate an application-specific password for this use on your google accounts page. Then, the general use will be something like:
/usr/local/google_appengine/bulkloader.py --dump --url=http://your_app.appspot.com/_ah/remote_api --filename=allkinds.sq3
You may not (yet) be able to use this "all kinds" query -- the server only generates the needed statistics for the all-kinds query "periodically", so you may get an error message including info such as:
[ERROR ] Unable to download kind stats for all-kinds download.
[ERROR ] Kind stats are generated periodically by the appserver
[ERROR ] Kind stats are not available on dev_appserver.
If that's the case, then you can still get things "one kind at a time" by adding the option --kind=EntityKind and running the bulkloader repeatedly (with separate sqlite3 result files) for each kind of entity.
Once you've dumped (kind by kind if you have to, all at once if you can) the production datastore, you can use the bulkloader again, this time with --restore and addressing your localhost dev_appserver instance, to rebuild the latter's datastore.
It should be possible to explicitly list kinds in the --kind flag (by separating them with commas and putting them all in parentheses) but unfortunately I think I've found a bug stopping that from working -- I'll try to get it fixed but don't hold your breath. In any case, this feature is not documented (I just found it by studying the open-source release of bulkloader.py) so it may be best not to rely on it!-)
More info about the then-new bulkloader can be found in a blog post by Nick Johnson at http://blog.notdot.net/2010/04/Using-the-new-bulkloader (though it doesn't cover newer functionalities such as the sqlite3 format of results in the "zero configuration" approach I outlined above). There's also a demo, with plenty of links, at http://bulkloadersample.appspot.com/ (also a bit outdated, alas).
Check out the remote API. This will tunnel your database calls over HTTP to the production database.
This used to be done in the adminstrator console. Now, according to the docs, it's controlled by a setting in the application's configuration files.
I updated my app.yaml file to include these lines and redeployed it:
#
# Module Settings
# https://cloud.google.com/appengine/docs/python/modules/#Python_Configuration
#
module: default
instance_class: F2
However, I haven't noticed any improvement in my application's performance. Specifically, I have a (cpu-bound) script that was taking 4-5 secs to run and there has been no difference since the change.
So my question is: am I doing this correctly? And is there a way to confirm (for example, in the logs or elsewhere in the admin console) the level at which my application's servers are running?
I should note that I am testing this on an unbilled application. Although I couldn't find any information in the docs that indicated this feature was limited to billed applications, I know that some features are unavailable on unbilled apps.
The settings you have there look correct.
If you are using modules, and it looks like you are, you can confirm the frontend instance class is what you set it to by viewing the "Versions" page on the old app engine console at http://appengine.google.com/
If you aren't using modules you can view the instance type on the "Application Settings" page.
Unfortunately, there doesn't seem to be a way to check the frontend instance class using the new cloud console.
If you look under Instances in the application dashboard you can see which ones you currently have running.
I am looking for a template engine for pushing and pulling data from configuration files. To be more specific, Cisco router configuration files. My goal has two parts
1) To be able to template my router config and insert unique data (hostname, interface IP's, ...etc) from an authoritative source (Mysql). Afterwards, I have a mechanism for loading the configs.
2) Once a device is configured and placed into production, I need a way of auditing against the latest version of my template. This would allow us to discover when operators change the running configuration.
Thoughts?
Let's take the simplest approach.
Use whatever language and templating engine you want, write a script that generates a config by e.g. a device name.
To check, generate a config for a device, download the actual config from that device, run diff. Mail the differences, if any, to people in charge of auditing.
The templating engine makes no difference in your case: you have no performance constraints, it seems. I'd take Python + Mako / Jinja / Cheetah, or Ruby + Rails, but even a bash + sed script could work.