I need to create a copy of the database in my MySQL Server using Django Application
After a little research, i found mysqldump as the better approach
backup_file_path = f"/tmp/{src_database_name}_backup.sql"
backup_db_command = f"mysqldump -h {SQL_DB_HOST} -P 3306 -u {SQL_DB_USER} -p{SQL_DB_PASSWORD} {src_database_name} > {backup_file_path}"
print(backup_db_command) # TODO: remove
with os.popen(backup_db_command, "r") as p:
r = p.read()
print(f"Backup Output: {r}")
restore_command = f"mysql -u root -p{SQL_DB_PASSWORD} {dest_database_name} < {backup_file_path}"
with os.popen(restore_command, "r") as p:
r = p.read()
print(f"Restore Output: {r}")
My Queries:
Any issues with this approach
Any better approaches to do a copy of DB using Either python or Django ORM
You can try Using django-admin, Django’s command-line utility for dumpdata and loaddata.
Use the following command to dump data
django-admin dumpdata [app_label[.ModelName] [app_label[.ModelName]
...]] -o database_backup.sql
Use the following command to load data
django-admin loaddata database_backup.sql
docs-
dump data
load data
Related
I am using Open Semantic Search (OSS) and I would like to monitor its processes using the Flower tool. The workers that Celery needs should be given as OSS states on its website
The workers will do tasks like analysis and indexing of the queued files. The workers are implemented by etl/tasks.py and will be started automatically on boot by the service opensemanticsearch.
This tasks.py file looks as follows:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
#
# Queue tasks for batch processing and parallel processing
#
# Queue handler
from celery import Celery
# ETL connectors
from etl import ETL
from etl_delete import Delete
from etl_file import Connector_File
from etl_web import Connector_Web
from etl_rss import Connector_RSS
verbose = True
quiet = False
app = Celery('etl.tasks')
app.conf.CELERYD_MAX_TASKS_PER_CHILD = 1
etl_delete = Delete()
etl_web = Connector_Web()
etl_rss = Connector_RSS()
#
# Delete document with URI from index
#
#app.task(name='etl.delete')
def delete(uri):
etl_delete.delete(uri=uri)
#
# Index a file
#
#app.task(name='etl.index_file')
def index_file(filename, wait=0, config=None):
if wait:
time.sleep(wait)
etl_file = Connector_File()
if config:
etl_file.config = config
etl_file.index(filename=filename)
#
# Index file directory
#
#app.task(name='etl.index_filedirectory')
def index_filedirectory(filename):
from etl_filedirectory import Connector_Filedirectory
connector_filedirectory = Connector_Filedirectory()
result = connector_filedirectory.index(filename)
return result
#
# Index a webpage
#
#app.task(name='etl.index_web')
def index_web(uri, wait=0, downloaded_file=False, downloaded_headers=[]):
if wait:
time.sleep(wait)
result = etl_web.index(uri, downloaded_file=downloaded_file, downloaded_headers=downloaded_headers)
return result
#
# Index full website
#
#app.task(name='etl.index_web_crawl')
def index_web_crawl(uri, crawler_type="PATH"):
import etl_web_crawl
result = etl_web_crawl.index(uri, crawler_type)
return result
#
# Index webpages from sitemap
#
#app.task(name='etl.index_sitemap')
def index_sitemap(uri):
from etl_sitemap import Connector_Sitemap
connector_sitemap = Connector_Sitemap()
result = connector_sitemap.index(uri)
return result
#
# Index RSS Feed
#
#app.task(name='etl.index_rss')
def index_rss(uri):
result = etl_rss.index(uri)
return result
#
# Enrich with / run plugins
#
#app.task(name='etl.enrich')
def enrich(plugins, uri, wait=0):
if wait:
time.sleep(wait)
etl = ETL()
etl.read_configfile('/etc/opensemanticsearch/etl')
etl.read_configfile('/etc/opensemanticsearch/enhancer-rdf')
etl.config['plugins'] = plugins.split(',')
filename = uri
# if exist delete protocoll prefix file://
if filename.startswith("file://"):
filename = filename.replace("file://", '', 1)
parameters = etl.config.copy()
parameters['id'] = uri
parameters['filename'] = filename
parameters, data = etl.process (parameters=parameters, data={})
return data
#
# Read command line arguments and start
#
#if running (not imported to use its functions), run main function
if __name__ == "__main__":
from optparse import OptionParser
parser = OptionParser("etl-tasks [options]")
parser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False, help="Don\'t print status (filenames) while indexing")
parser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False, help="Print debug messages")
(options, args) = parser.parse_args()
if options.verbose == False or options.verbose==True:
verbose = options.verbose
etl_delete.verbose = options.verbose
etl_web.verbose = options.verbose
etl_rss.verbose = options.verbose
if options.quiet == False or options.quiet==True:
quiet = options.quiet
app.worker_main()
I read multiple tutorials about Celery and from my understanding, this line should do the job
celery -A etl.tasks flower
but it doesnt. The result is the statement
Error: Unable to load celery application. The module etl was not found.
Same for
celery -A etl.tasks worker --loglevel=debug
so Celery itself seems to be causing the trouble, not flower. I also tried e.g. celery -A etl.index_filedirectory worker --loglevel=debug but with the same result.
What am I missing? Do I have to somehow tell Celery where to find etl.tasks? Online research doesn't really show a similar case, most of the "Module not found" errors seem to occur while importing stuff. So possibly it's a silly question but I couldn't find a solution anywhere. I hope you guys can help me. Unfortunately, I won't be able to respond until Monday though, sorry in advance.
I got same issue, I installed and configured my queue as follows, and it works.
Install RabbitMQ
MacOS
brew install rabbitmq
sudo vim ~/.bash_profile
In bash_profile add the following line:
PATH=$PATH:/usr/local/sbin
Then update bash_profile:
sudo source ~/.bash_profile
Linux
sudo apt-get install rabbitmq-server
Configure RabbitMQ
Launch the queue:
sudo rabbitmq-server
In another Terminal, configure the queue:
sudo rabbitmqctl add_user myuser mypassword
sudo rabbitmqctl add_vhost myvhost
sudo rabbitmqctl set_user_tags myuser mytag
sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"
Launch Celery
I would suggest to go in the folder that contains task.py and use the following command:
celery -A task worker -l info -Q celery --concurrency 5
Beware that this error means two things:
The module is missing
The module exists but cannot be loaded. If it has errors in it, such as a SyntaxError for instance.
To check that it's not the latter, run:
python -c "import <myModuleContainingTasksDotPyFile>"
In the context of this question:
python -c "import etl"
If it crashes, fix this first (Unlike with celery, you'll get a detailed error message).
Solutions above did not work for me.
I had the same issue and my problem was that in main celery.py (that was in SmartCalend folder) I had:
app = Celery('proj')
but instead I must type there:
app = Celery('SmartCalend')
where SmartCalend is the actual app name where celery.py belongs (!). not any random word, but precisely app name. Thats nowhere mentioned, only in official docs here:
Try export PYTHONPATH=<parent directory> where parent directory is the folder where the etl is. Run the Celery worker, and see it if fixes your problem. This is probably one of the most common Celery "issues" (not really Celery, but Python in general). Alternatively, run the Celery worker from that folder.
Answer for MacOS Catalina:
When you install celery with pip (pip install celery), python can import celery, but you are not able to launch celery from the terminal because the terminal does not know of the celery executable.
Add celery to the path to fix:
nano ~/.bash_profile
In the file add: export PATH="/Users/gavinbelson/Library/Python/2.7/bin:$PATH"
To save the file in the nano editor: ctrl+o, then enter, then ctrl+x
To update the terminal with your change type: source ~/.bash_profile
Now you should be able to type celery in the terminal window
---- Note this is for the default python terminal command which runs version 2.7. If you are using python3 to run python, you would need to change alter the path variable accordingly
I use manage.py dumpdata --format xml --some-more-parameters to export a full dump of the database to xml. The database is MS sql server and I'm using pyodbc as the driver. The dumpdata command is run using PowerShell and since Django 1.7 does not support a --output argument for the dumpdata command I redirect the output into a file using PowerShell.
Unfortunately the database contains unicode characters (e.g. country \xd6sterreich) and these characters are scrambled int the export file.
Here's what didn't work:
./manage.py dumpdata --format xml > export.xml
./manage.py dumpdata --format xml | out-file -encoding utf8 export.xml
./manage.py dumpdata -format xml | out-file -encoding ANY_OTHER_SUPPORTED_ENCODING export.xml
None of these commands work. Umlauts and accents are scrambled and additionally the > export.xml method adds an invalid BOM to the file which will result in ./manage.py loaddata export.xml aborting with an UnicodeDecode error message when I try to import this on another host.
Any suggestions on how I could export the data and preserve the special characters? The same problem exists when using the json or yaml serializers.
I was able to work around this problem using my own export script. The script below will dump the data and store it in a utf-8 encoded xml file called export_CURRENT-DATE-TIME.xml. call_command() calls the dumpdata command in Django. The script below should be equivalent to using dumpdata with the following arguments:
./manage.py dumpdata --natural --natural-foreign --natural-primary --format xml --indent 2
import sys
import codecs
import os
import django
from django.core.management import call_command
from StringIO import StringIO
from datetime import datetime
# setup access to django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "PROJECT_NAME.settings")
django.setup()
# the actual export command
def do_work():
#print(u"\xd6sterreich")
call_command('dumpdata', use_natural_keys=True, use_natural_foreign_keys=True, use_natural_primary_keys=True, format='xml', indent=2)
# nasty hack to workaround encoding issues on windows
_stdout = sys.stdout
sys.stdout = StringIO()
do_work()
value = sys.stdout.getvalue().decode('utf-8')
sys.stdout = _stdout
with codecs.open('export_{}.xml'.format(datetime.now().strftime("%Y-%m-%d_%H-%M")), 'w', 'utf-8-sig') as f:
f.write(value)
print("export completed")
I have a simple python script that puts data in a database. Both the script and
the database have owner www-data. When I run sudo python and write the
commands one by one it works, but if I run python monitor.py or sudo python monitor.py it doesn't work; it says, "attempt to write a read only database".
This is my script: (it receives data from arduino)
from serial import Serial
from time import sleep
import sqlite3
serial_port = '/dev/ttyACM0';
serial_bauds = 9600;
# store the temperature in the database
def log_light(value):
conn=sqlite3.connect('/var/db/arduino.db')
curs=conn.cursor()
curs.execute("UPDATE sensor1 set status = (?)", (value,))
# commit the changes
conn.commit()
conn.close()
def main():
s = Serial(serial_port, serial_bauds);
s.write('T');
sleep(0.05);
line = s.readline();
temperature = line;
s.write('H');
sleep(0.05);
line = s.readline();
humidity = line;
s.write('L');
sleep(0.05);
line = s.readline();
light = line;
log_light(light);
if __name__=="__main__":
main()
It sounds like a permission problem. Write access is granted only to the user, which is root. You need to change the user to be directly yourself, not root. You can do this using chmod on many *nix systems.
You could also gove write access to anyone in the group.
Following the instructions on https://github.com/linuxlewis/djorm-ext-pgfulltext I added a search_index to one of my existing PostgreSQL models, as follows:
search_index = VectorField()
objects = SearchManager(
fields = ('materials','origin','style'),
config = 'pg_catalog.english',
search_field = 'search_index',
auto_update_search_field = True
)
Using my virtual-env and having pgfulltext installed, with this version djorm-ext-pgfulltext==0.9.3 according to pip freeze, I cd to the directory where manage.pyis located. And tried to run:
./manage.py update_search_field app_name
Wich resulted in the following error:
Unknown command: 'update_search_field' Type 'manage.py help' for
usage.
Am I missing something on how this command is used? Should I add the index directly to my db instance with psql? And if so, how could I achieve this?
Any help will be much appreciated.
"Unknown command"
...this usually means you forgot to add the app to your INSTALLED_APPS in settings.py
That's not surprising since it seems they forgot to mention that on the readme for the app, but you need something like:
INSTALLED_APPS = (
# ... (your existing apps)
'djorm_pgfulltext',
)
Django needs to be told about the app in this way so that it knows to look for the management/commands directory in the app.
I want to run a python scripts which should do:
Create a django project: django-admin startproject foobar
Create a app in the project: python manage.py barfoo
Add an entry of newly created app barfoo in the setting's INSTALLED_APP.
How can I achieve this?
There seems to be a pythonic way to do #1 and #2
https://docs.djangoproject.com/en/dev/ref/django-admin/#running-management-commands-from-your-code
from django.core import management
management.call_command('flush', verbosity=0, interactive=False)
management.call_command('loaddata', 'test_data', verbosity=0)
6 years later I stumbled upon this question trying to figure out how to write some tests for an app which only add a custom template tag that interact with other apps in the project. Hope this can help someone.
Building on #groovehunter answer: the official documentation now (Django 1.10) inculdes this feature outside dev.
Note that you need to change current directory to the created project before call startapp. See this answer for more details
from django.core import management
import os
management.call_command('startproject', 'foobar')
os.chdir('foobar')
management.call_command('startapp', 'barfoo')
or you can use the additional argumento to startproject to create the project in the current directory, if you're sure there won't be problems:
from django.core import management
management.call_command('startproject', 'foobar', '.')
management.call_command('startapp', 'barfoo')
Read a little abour subprocess and Popen method. This might be what you're looking for.
Popen(["django-admin", "startproject", "%s" % your_name ], stdout=PIPE).communicate()
Popen(["python", "manage.py", "%s" % your_app_name ], stdout=PIPE).communicate()
3.
I know that's not a perfect code, but I'm just giving an idea.
with open("settings.py", 'r') as file:
settings = file.readlines()
new_settings = []
for line in settings:
if "INSTALLED APPS" in line:
new_settings.append(line.replace("INSTALLED_APPS = (", "INSTALLED_APPS = (\n'%s'," % your_app_name))
else:
new_settings.append(line)
with open("settings.py", 'w') as file:
file.write("".join(new_settings))