Architectural pattern for multiple Python applications with shared database - python

The project consists of two applications in Python with a single shared database:
Application A: performs preprocessing of raw data and stores it in the database;
Application B: performs data analysis techniques on the already populated database.
What should be the right architectural pattern for such a context?
Also, so far I have been using MongoDB and Flask to populate the database but I don't know if it would be suitable as a choice.

If you are asking about a database access pattern, you can always use ORMs (Object relational mappers). An object-relational mapper (ORM) is a code library that automates the transfer of data stored in relational database tables into objects that are more commonly used in application code.
You can learn more about ORM in here. https://www.fullstackpython.com/object-relational-mappers-orms.html
I believe this would be better suited for python than the DAO pattern we use in java. https://www.tutorialspoint.com/design_pattern/data_access_object_pattern.htm

Related

Is it possible to use 2 different frameworks in 1 backend of a web app?

I am an 11th grade student and I'm learning how to build a web app, with my teammates. Currently, We're making a website showing the school schedule (also to show students' marks) and helping the users to create their to-do lists, of course this web serves students like me. In the backend of the web, we use Python as the main language, Flask as the framework and MySQL to manipulate our database. Now, everything is ok and we're trying to make something like an admin interface for people who host the web. Specifically, it is where teachers can insert their students' grade, and maybe adjust the school timetable. The problem is, we're learning how to use Flask Admin to code that function, and we've found out this tech is only compatible with SQL Server. However, we have a better understand in MySQL therefore we could create multiple tasks, in contrast, we don't know how to use SQL Server to create those funcs. Now I have 2 main questions:
Could we use 2 different SQL in the backend of our web? It is the quickest way we know, however we have to learn how to use SQL Server.
Could we use 2 different Python backend frameworks in the backend of our web? We haven't searched which framework to use yet because we don't know if it's possible to do this.
We don't know any other ways to solve this problem except getting rid of MySQL and use SQL Server instead. However this is not the way we prefer and we hope those 2 questions answered. If there is anything wrong in our knowledge please just straightly comment to let us know, and we greatly welcome any other solutions. Thanks for answering!!
Directly from the flask-admin docs https://flask-admin.readthedocs.io/en/latest/advanced/#using-different-database-backends is the following:
Using Different Database Backends Other than SQLAlchemy… There are
five different backends for you to choose from, depending on which
database you would like to use for your application. If, however, you
need to implement your own database backend, have a look at Adding A
Model Backend.
If you don’t know where to start, but you’re familiar with relational
databases, then you should probably look at using SQLAlchemy. It
is a full-featured toolkit, with support for SQLite, PostgreSQL,
MySQL, Oracle and MS-SQL amongst others. It really comes into its own once you have lots of data, and a fair amount of relations between
your data models. If you want to track spatial data like
latitude/longitude points, you should look into GeoAlchemy, as well.
Regarding the original question, it is possible to use two different frameworks in the backend of a web app. One way to do so would be to set up a reverse proxy server (see https://www.nginx.com/resources/glossary/reverse-proxy-server/#:~:text=A%20reverse%20proxy%20server%20is,traffic%20between%20clients%20and%20servers.), but I would recommend giving SQLAlchemy before doing so.
Why do you think that flask-admin is tied to SqlServer? Flask (and flask-admin) can handle different connections to various databases:
https://github.com/flask-admin/flask-admin#introduction
https://flask-admin.readthedocs.io/en/v1.0.9/db/
My guess is you are currently using SqlAlchemy. As explained here, you can use different backends:
The string form of the URL is dialect[+driver]://user:password#host/dbname[?key=value..], where dialect is a database name such as mysql, oracle, postgresql, etc., and driver the name of a DBAPI, such as psycopg2, pyodbc, cx_oracle, etc.
Alternatively, the URL can be an instance of URL.
(https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine)
More on Engine here
Flask-admin is a admin view on your database tables - you cannot run it on a "different" db then the tables it should modify. It needs to have access to the database tables you want it to modify, so you cannot run "admin" on MS and "your data backend" on another database(-connection).
Some other things to think about:
MS-(T-)Sql and MySql are infrastructural choices, language wise they are very closely related so adapting MySql knowledge to T-SQL Syntax should be possible. Choosing SqlServer may force you to license it - and afaik that comes with fees (eiter on premise or as a azure subscription wich might or might not be free for schools (no idea - but you should check that)).
This sounds like an ambitious project for school - depending on where you live, privat data protection laws come into play especially considering you connect names with schedules with grades which would need you to implement a lot more to comply.
For first question, I checked flask-admin documentation and found that the framework already included serval built-in ORM library, i.e. SQLAlchemy, MongoEngine, pymongo and Peewee. This means you can just directly import the ORM library from the flask-admin package and use it to access your database. For your case, you should look for SQLAlchemy as you are using SQL Database. Both SQL Server and MySQL are supported by SQLAlchemy.
The Adding Model Views section in their official doc also mentioned it as well:
https://flask-admin.readthedocs.io/en/latest/introduction/#getting-started
For the second question, it is technically not possible to apply two different frameworks in one single backend application.

How to query MySQL database using API in python?

I have my data in database, and I want to build an API using python to query the database. What do I use or where do I even start ?
I can connect to my database and pull the data. But I do not know how to build an API for other users to query the database
NOT a trivial solution, but worth looking into, depending on how big a project this is and what else you may want to do with Python and MySQL.
Django is a framework for Python that can handle a lot of things but, most importantly, provides a very powerful interface to MySQL and other databases.
One of the many add-ons for Django is Django REST Framework which is specifically designed for setting up Restful APIs using Django & Python.
I have used Django & Django REST Framework extensively. It can take a while to get up to speed, but I think still a lot less work than if you were to take a basic connector between Python & MySQL (having to code SQL statements manually) and code the API interface at a lower level (Django REST Framework isn't perfect but it takes care of a lot of the details).

Python ORM - save or read sql data from/to files

I'm completely new to managing data using databases so I hope my question is not too stupid but I did not find anything related using the title keywords...
I want to setup a SQL database to store computation results; these are performed using a python library. My idea was to use a python ORM like SQLAlchemy or peewee to store the results to a database.
However, the computations are done by several people on many different machines, including some that are not directly connected to internet: it is therefore impossible to simply use one common database.
What would be useful to me would be a way of saving the data in the ORM's format to be able to read it again directly once I transfer the data to a machine where the main database can be accessed.
To summarize, I want to do:
On the 1st machine: Python data -> ORM object -> ORM.fileformat
After transfer on a connected machine: ORM.fileformat -> ORM object -> SQL database
Would anyone know if existing ORMs offer that kind of feature?
Is there a reason why some of the machine cannot be connected to the internet?
If you really can't, what I would do is setup a database and the Python app on each machine where data is collected/generated. Have each machine use the app to store into its own local database and then later you can create a dump of each database from each machine and import those results into one database.
Not the ideal solution but it will work.
Ok,
thanks to MAhsan's and Padraic's answers I was able to find the how this can be done: the CSV format is indeed easy to use for import/export from a database.
Here are examples for SQLAlchemy (import 1, import 2, and export) and peewee

Is there an equivalent of a SQL View in Google App Engine Python?

I've been learning python by building a webapp on google app engine over the past five or six months. I also just finished taking a databases class this semester where I learned about views, and their benefits.
Is there an equivalent with the GAE datastore using python?
Read-only views (the most common type) are basically queries against one or more tables to present the illusion of new tables. If you took a college-level database course, you probably learned about relational databases, and I'm guessing you're looking for something like relational views.
The short answer is No.
The GAE datastore is non-relational. It doesn't have tables. It's essentially a very large distributed hash table that uses composite keys to present the (very useful) illusion of Entities, which are easy at first glance to mistake for rows in a relational database.
The longer answer depends on what you'd do if you had a view.
First of all answer to your question: With normal GAE, i.e. non relational DB GAE, you won't have such things as views
Since you are probably starting with Relational SQL in school, I would suggest switch to Relational SQL based GAE at http://code.google.com/apis/sql/ and http://code.google.com/apis/sql/docs/before_you_begin.html#enroll ( I am not sure if it's available right away, or you need to wait for approval to use an instance, but register right away)
Web based applications are using emerging NON relational DBs and you would be benefited by studying them as well. That way you also could understand GAE non relational better. As a basic level start at http://en.wikipedia.org/wiki/NoSQL and then you have many more to explore, specially famous once being Mongo DB, Amazon Simple DB etc.

Are there database testing tools for python (like sqlunit)?

Are there database testing tools for python (like sqlunit)? I want to test the DAL that is built using sqlalchemy
Follow the design pattern that Django uses.
Create a disposable copy of the database. Use SQLite3 in-memory, for example.
Create the database using the SQLAlchemy table and index definitions. This should be a fairly trivial exercise.
Load the test data fixture into the database.
Run your unit test case in a database with a known, defined state.
Dispose of the database.
If you use SQLite3 in-memory, this procedure can be reasonably fast.

Categories