Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
TL DR: should you use graylog instead mariadb as database for a moderate amount of data? Would this be a good idea?
Longversion:
There is a python script which puts some nmap monitoring data (maybe 1000 to 10000 rows) into a maria database each day. It is using sql alchemy, a flask website is using the same datatypes to display the results and everything is working fine. The model is not too complicated, 3 Tables with a one to many relation.
ip range -> hosts -> ports (with results)
Now my boss wants to put everything into graylog with a python binding and to abadon sql alchemy and the mariadb completly.
My question is: would you recommend to keep the existing structure and just export each night everything from the mariadb into the graylog server (the data is collected just once a night).
Replace the database / mysql alchemy and use Graylog directly? I have never worked with Graylog and I am not sure if you can use it as a database replacement.
Graylog is a log agregator like Splunk its not a relational database itself. From what you describe you are using the MariaDB Database like a logging utility for this data.
Questions to ask yourself are, do you have other logs you should be aggregating and collecting or is it just this one niche set of data?
If you have multiple sources then Graylog is one answer but if its just this subset then its probably overkill.
For context I use Splunk to aggregate over 50GB of logs per day, they come from servers, networking gear, firewalls and more. I'd never think to pour that data into a SQL Database, its not the right tool for the job.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I'm currently writing a chat-application for my Website and storing all the Messages in one database, by referring to sender and receiver with a Foreign-key.
Is this really the smartest Idea? I think the site could become slow when everybody is trying to access the same database or when the number of sent messages gets big. Would it be smarter/faster, to use one database per user, storing his/her messages there? If yes, am i right that this is not really possible to achieve this with Django?
Thanks for your Answer!
Would it be smarter/faster, to use one database per user
Probably not, because you will still have the same quantity of data and just have the overhead of have multiple database instances
this is not really possible to achieve this with Django?
Django is not a database manager, so the question does not really make sense. For example, Oracle or SQLite allow a single query to access multiple databases. So if you used Oracle with Django, you could have a different database per user. Anyway, even if by default Django only uses one single database connection, it can be configured to use multiple databases (thanks to #brunodesthuilliers for the info).
Database design is a rather complex operation: you have to design the tables to structure the data, the indexes, to speed up some accesses and/or enforce uniqueness, and optionaly views to easy request writing. Once this is done (and the database is used), you will have to define recurrent tasks to cleanup the database (index maintenance, archivage and purge of archived data).
If you intend to use a large farm of servers to be able to handle thousands of simultaneous accesses, a SQL database can indeed become a bottleneck, and you should considere NoSQL databases to dispatch the load on independant servers. But in this use case, Django would just be a casting error. Anyway, I know no example of real world use case where multiple databases are used just for performance reasons.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I'm using Python (3.7.4) to make a text/password manager, where the user can store text under different tabs (using tkinter for the interface) , and use a "master login" to access all data.
The only experience I've got with saving/storing data is using CSV files, and looping through them to get values.
Can anyone recommend anyway I can store text, without it being able to be opened from Windows Explorer, and need some sort of key to be opened?
The natural alternative to using csv files is the use of a database. A solution like sqlite might be enough for your solution. Directly from the documentation:
SQLite is a C library that provides a lightweight disk-based database
that doesn’t require a separate server process and allows accessing
the database using a nonstandard variant of the SQL query language.
Some applications can use SQLite for internal data storage. It’s also
possible to prototype an application using SQLite and then port the
code to a larger database such as PostgreSQL or Oracle.
Once you have learned sqlite, you can think of encrypting your database: in this post you will find many ideas for encrypting your sqlite database.
Otherwise you can switch to other more complete and complex DBMSs. The important thing is that you consider moving from the csv to using a db.
Take a look at SQLLite as a lightweight local database and use something like SQLCipher for strong AES 256-bit encryption.
I havn't read it fully but take a look at this blog for a python implementation.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I wonder if a good habit is to have some logic in MySQL database (triggers etc.) instead of logic in Django backend. I'm aware of fact that some functionalities may be done both in backend and in database but I would like to do it in accordance with good practices. I'm not sure I should do some things manually or maybe whole database should be generated by Django (is it possible)? What are the best rules to do it as well as possible? I would like to know the opinion of experienced people.
It is true that if you used a database for your business logic you could get maximum possible performance and security optimizations. However, you would also risk many things such as
No separation of concerns
Being bound to the database vendor
etc.
Also, whatever logic you write in your database won't be version controlled with your app. Thus, whenever you change your database, you will have to create all those things once again.
Instead, use Django ORM. It will create and manage your database based on your models by itself. As a result, whenever you recreate your database, you will just have to run migrations with one single command and you are done.
This will cover most of the situations. And whenever you will need those speeds of stored procedures, Django ORM has you covered as well.
In short, I believe that business logic should be kept out of the database as much as possible.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
i am trying to connect to a database and insert data with python.I found too many connector drivers like : mysql-connector,pymysql, MySQLdb. I want to know which way is better to communicate with database in python3.4 .
In our project we have remote machine with MySQL default client, we create SSH object of it and run sql query using traditional client. It's the most safe way and supports mostly everything with best optimized support.
However, if you want to prepare sql handle in python I will suggest go for pymysql, as pymysql is updated regularly & its very much stable & easy to use compared to others.
I would check out sqlalchemy http://www.sqlalchemy.org/. Or if you want more of an object-relational-manager then set up a small Django project https://www.djangoproject.com/.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I have a small flask app I want to deploy on my server and I'd like to be able to monitor it via an HTTP web interface. I don't need something complicated and I definitely don't want something that's difficult to set up. Previously I've used the Google App engine and the functionality in the Logs tab is completely fine.
The app is served through nginx & gunicorn and uses redis (w/ py-redis) and sqlite (w/ peewee). Ideally I'd like to be able to check the logs for all parts of the system from one place. Is this possible? What's the easiest way?
There is no definitive answer to the predicament and it would be whatever way you are most comfortable with.
You could change all your logging to write to a central database then create a small program which would scrape this data for you. This method also includes configuring a central syslog server:
http://www.linuxjournal.com/content/creating-centralized-syslog-server
What ever way you want to read these files is fine and all depends how much control you want. You could simply name all the logs based on hostname and rsync them to a central server from where you could parse them.
There are also free tools out there which will aid you in choosing you method take a look at:
http://www.linuxjournal.com/content/creating-centralized-syslog-server
There are also some proprietary systems you could use, such as Splunk:
http://www.splunk.com/
This is by no means a definitive list but should aim you in the right direction.