I'm completely new to Django. I have developed a simple web application in Django and hosted it on external server. And that web application uses default mysql database. Now, I want to switch to a secondary server if my primary server goes down.
Copying and running the same code is not the option.
Can anyone explain how do I do it along with an example ?
not a beginner topic or answer but, in a prototypical production-esque deployment with nginx/apache/django, you can use the nginx upstream module
it'll detect when a node is down and won't route requests to the down node, so you'll get the failover behavior you desire out of the box
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream
I googled "nginx upstream module failover" and the second result had some examples: https://serverfault.com/questions/140990/nginx-automatic-failover-load-balancing
Related
In short:
I have a Django application being served up by Apache on a Google Compute Engine VM.
I want to access a secret from Google Secret Manager in my Python code (when the Django app is initialising).
When I do 'python manage.py runserver', the secret is successfully retrieved. However, when I get Apache to run my application, it hangs when it sends a request to the secret manager.
Too much detail:
I followed the answer to this question GCP VM Instance is not able to access secrets from Secret Manager despite of appropriate Roles. I have created a service account (not the default), and have given it the 'cloud-platform' scope. I also gave it the 'Secret Manager Admin' role in the web console.
After initially running into trouble, I downloaded the a json key for the service account from the web console, and set the GOOGLE_APPLICATION_CREDENTIALS env-var to point to it.
When I run the django server directly on the VM, everything works fine. When I let Apache run the application, I can see from the logs that the service account credential json is loaded successfully.
However, when I make my first API call, via google.cloud.secretmanager.SecretManagerServiceClient.list_secret_versions , the application hangs. I don't even get a 500 error in my browser, just an eternal loading icon. I traced the execution as far as:
grpc._channel._UnaryUnaryMultiCallable._blocking, line 926 : 'call = self._channel.segregated_call(...'
It never gets past that line. I couldn't figure out where that call goes so I couldnt inspect it any further than that.
Thoughts
I don't understand GCP service accounts / API access very well. I can't understand why this difference is occurring between the django dev server and apache, given that they're both using the same service account credentials from json. I'm also surprised that the application just hangs in the google library rather than throwing an exception. There's even a timeout option when sending a request, but changing this doesn't make any difference.
I wonder if it's somehow related to the fact that I'm running the django server under my own account, but apache is using whatever user account it uses?
Update
I tried changing the user/group that apache runs as to match my own. No change.
I enabled logging for gRPC itself. There is a clear difference between when I run with apache vs the django dev server.
On Django:
secure_channel_create.cc:178] grpc_secure_channel_create(creds=0x17cfda0, target=secretmanager.googleapis.com:443, args=0x7fe254620f20, reserved=(nil))
init.cc:167] grpc_init(void)
client_channel.cc:1099] chand=0x2299b88: creating client_channel for channel stack 0x2299b18
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
client_channel.cc:1879] chand=0x2299b88 calld=0x229e440: created call
...
call.cc:1980] grpc_call_start_batch(call=0x229daa0, ops=0x20cfe70, nops=6, tag=0x7fe25463c680, reserved=(nil))
call.cc:1573] ops[0]: SEND_INITIAL_METADATA...
call.cc:1573] ops[1]: SEND_MESSAGE ptr=0x21f7a20
...
So, a channel is created, then a call is created, and then we see gRPC start to execute the operations for that call (as far as I read it).
On Apache:
secure_channel_create.cc:178] grpc_secure_channel_create(creds=0x7fd5bc850f70, target=secretmanager.googleapis.com:443, args=0x7fd583065c50, reserved=(nil))
init.cc:167] grpc_init(void)
client_channel.cc:1099] chand=0x7fd5bca91bb8: creating client_channel for channel stack 0x7fd5bca91b48
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
So, we a channel is created... and then nothing. No call, no operations. So the python code is sitting there waiting for gRPC to make this call, which it never does.
The problem appears to be that the forking behaviour of Apache breaks gRPC somehow. I couldn't nail down the precise cause, but after I began to suspect that forking was the issue, I found this old gRPC issue that indicates that forking is a bit of a tricky area.
I tried to reconfigure Apache to use a different 'Multi-processing Module', but as my experience in this is limited, I couldn't get gRPC to work under any of them.
In the end, I switched to using nginx/uwsgi instead of Apache/mod_wsgi, and I did not have the same issue. If you're trying to solve a problem like this and you have to use Apache, I'd advice further investigating Apache forking, how gRPC handles forking, and the different MPMs available for Apache.
I'm facing a similar issue. When running my Flask Application with eventlet==0.33.0 and gunicorn https://github.com/benoitc/gunicorn/archive/ff58e0c6da83d5520916bc4cc109a529258d76e1.zip#egg=gunicorn==20.1.0. When calling secret_client.access_secret_version it hangs forever.
It used to work fine with an older eventlet version, but we needed to upgrade to the latest version of eventlet due to security reasons.
I experienced a similar issue and I was able to solve with the following:
import grpc.experimental.gevent as grpc_gevent
from gevent import monkey
from google.cloud import secretmanager
monkey.patch_all()
grpc_gevent.init_gevent()
client = secretmanager.SecretManagerServiceClient()
I wrote a Flask web application for a system that our company uses. However, we have another web application, which is running on Node.JS. The "problem" is that my colleague writes everything on node, while I write everything in Python.
We want to implement both applications on one webpage - for example:
My application will run on example.com/assistant
His application will run on example.com/app1 and example.com/app2
How can we do this? Can we somehow implement the templates that I use with his templates and vice-versa?
Thank you in advance!
V
Serving different apps from the same domain
You can use haproxy for directing requests to specific service based on ACL rules.
You could use path_beg rule, to direct any request beginning with specific path to be directed to corresponding server. See example below.
/etc/haproxy/haproxy.cfg
# only relevant part of the config file
# assumes all apps are on one machine
frontend http-in
bind *:80
acl py_app1 path_beg /assistant
acl node_app1 path_beg /app1
acl node_app2 path_beg /app2
default_backend main_servers
backend py_app1
server flask_app 127.0.0.1:5000
backend node_app1
server nodejs1 127.0.0.1:4001
backend node_app2
server nodejs2 127.0.0.1:4002
backend main_servers
server other1 127.0.0.1:3000 # nginx, apache, or whatever
Sharing template code between apps
This would be harder, as you would need to both agree on some kind of format, which needs to be language and framework-agnostic, and probably logic-less.
Mustache claims to be "framework-agnostic way to render logic-free views". I used it sparringly a few years ago so this one is first that came to mind, however you should do more research on this, maybe there is some better fit.
Python implementation
JS implementation
The problem would be to actually keep the templates always in sync with apps, and not break functionality of the views. If a template changes then you would need to test all apps that use this template file. Also, you probably will block one another from updating your apps at different times, because if one of you change the template files, then you must come to a consensus, update all relevant apps, and deploy them at one time.
i'ev developed a chatbot web application using Django and python3 on web faction server. Basically the chatbot interacts with a store's users as a customer server.It uses the REST API to POST the user input and GET to display the chatbot output and a python file to process the input and find the output.
How the chatting works:
1. chatboy.js: it first POST the user input in the API then run the python file chabot.py.
2. chabot.py: a python file that connects to the Django backend db.sqlite3 and the conversation.sqlite3. So it select the user input from db.sqlite3 then select the matching output in the conversation.sqlite3. Finally the file will update the chatbot output in the db.sqlite3.
3. chatboy.js: will GET the last chatbot output and display it.
In development stage when i was testing the application in my local server everything works fine but problems came we i deployed the Django project in the internet. Mostly and the main ERORR that stops the application form working is :
The database is locked
I did many research and found out that:
sqlite3 is not for production and only for small or stand alone application.
sqlite3 have a multiple threads problem.
not good for realtime chatting application.
Some suggestion said either
switch all the sqlite3 to Mysql ( but how? and how i can change the sql queries in the python file to fit Mysql)
use a fast key value store like Reddis ( does it mean to make it the backend db form my project or juts the conversation.sqlite3?)
please help me because it is very frustrating and i had this error for a long time and i couldn't found and solution for the problem.
thank you,
PS: i'm using Django 10 , python3 , sqlite3, web faction server
You only need to update your settings file to change the DATABASES setting from sqlite3 to mysql or postgresql. But before you do that you will need to create a database in your Webfaction Control Panel (more information here).
I'm hoping to be pointed in the right direction as far as what tools to use while in the process of developing an application that runs on two servers per client.
[Main Server][Client db Server]
Each client has their own server which has a django application managing their respective data, in addition to serving as a simple front end.
The main application server has a more feature-rich front end, using the same models/db schemas. It should have full read/write access to the client's database server.
The final desired effect would be a typical SaaS type deal:
client1.djangoapp.com => Connects to mysql database # client1_IP
client2.djangoapp.com => Connects to mysql database # client2_IP...
Thanks in advance!
You could use different settings files, let's say settings_client_1.py and settings_client_2.py, import common settings from a common settings.py file to keep it DRY. Then add respective database settings.
Do the same with wsgi files, create one for each settings. Say, wsgi_c1.py and wsgi_c2.py
Then, in your web server direct the requests for client1.djangoapp.com to wsgi_c1.py and client2.djangoapp.com to wsgi_c2.py
After fighting with different things here and there, I was finally able to get BottlePY running on Apache and run a MongoDB powered site. I am used to running Django apps, so I will be relating to that a bit in my question.
The Problem
Every time a page is loaded via BottlePY, the connection to the MongoDB database located on MongoHQ.com needs to be re-authenticated (meaning it probably had to reconnect).
What I Found
I attached a db.keep_alive() function to the top of each model function, so that before any mongodb query is run, it trys to run a simple query. If it fails, it catches the OperationFailure or AutoReconnect errors and then calls the db.authenticate() function. After it reauthenticates, I have it add a log to a logs db to monitor how often it needs to reauthenticate. Currently, it needs to reauthenticate on every page load (that requires running a query). This isn't right.
Difference from Django
I use this same concept in django, and have found that the db connection only needs to be authenticated after 10-15 minutes of no queries being run.
I don't understand why creating a pymongo connection in django would be different from creating one in bottle, since I am using the same driver, functions and methods. I am not using any ORMS or anything like that either.
Versions
Bottle: 0.9.dev
Django: 1.2.1 final
PyMongo: 1.8
I appreciate the help!
Update: A friend was able to take a quick look and noticed the following that may help with answering my question.
It appears that each request is
launching a new Python process, as
opposed to Django, in which a single
process remains running for a long
period of time.
This just ended up to be a weird thing between Bottle and MongoHQ. No real solution was found, but I couldn't recreate it with other frameworks. Any other ideas are appreciated.
does your apache xxx.conf contain something like:
WSGIDaemonProcess project user=mysite group=www-data processes=5 threads=1
WSGIProcessGroup project
I think most important should be threads=1