Full text mysql database search in Django - python

We have been using a MYSQL database for our project and Django as the backend framework. We want to support a full text search on a particular table and return the Queryset in Django. We know that Django supports full text search on a Postgres database but we can't move to another database now.
From what we have gathered till now -
Using inbuilt search functionality - Here we check on every field if the value exists and then take an OR to combine the results. Similar to the link (Django Search query within multiple fields in same data table).
This approach however straight forward may be inefficient for us because we have huge amounts of data.
Using a library or package - From what we have read Django haystack is something a lot of people are talking about when it comes to full text search.
Django Haystack - https://django-haystack.readthedocs.io/en/master/tutorial.html#installation
We haven't checked the library completely yet because we are trying to avoid using any library for this purpose. Let us know if you people have worked with this and have any views.
Any help is appreciated. Thanks.

Related

Is it possible to use 2 different frameworks in 1 backend of a web app?

I am an 11th grade student and I'm learning how to build a web app, with my teammates. Currently, We're making a website showing the school schedule (also to show students' marks) and helping the users to create their to-do lists, of course this web serves students like me. In the backend of the web, we use Python as the main language, Flask as the framework and MySQL to manipulate our database. Now, everything is ok and we're trying to make something like an admin interface for people who host the web. Specifically, it is where teachers can insert their students' grade, and maybe adjust the school timetable. The problem is, we're learning how to use Flask Admin to code that function, and we've found out this tech is only compatible with SQL Server. However, we have a better understand in MySQL therefore we could create multiple tasks, in contrast, we don't know how to use SQL Server to create those funcs. Now I have 2 main questions:
Could we use 2 different SQL in the backend of our web? It is the quickest way we know, however we have to learn how to use SQL Server.
Could we use 2 different Python backend frameworks in the backend of our web? We haven't searched which framework to use yet because we don't know if it's possible to do this.
We don't know any other ways to solve this problem except getting rid of MySQL and use SQL Server instead. However this is not the way we prefer and we hope those 2 questions answered. If there is anything wrong in our knowledge please just straightly comment to let us know, and we greatly welcome any other solutions. Thanks for answering!!
Directly from the flask-admin docs https://flask-admin.readthedocs.io/en/latest/advanced/#using-different-database-backends is the following:
Using Different Database Backends Other than SQLAlchemy… There are
five different backends for you to choose from, depending on which
database you would like to use for your application. If, however, you
need to implement your own database backend, have a look at Adding A
Model Backend.
If you don’t know where to start, but you’re familiar with relational
databases, then you should probably look at using SQLAlchemy. It
is a full-featured toolkit, with support for SQLite, PostgreSQL,
MySQL, Oracle and MS-SQL amongst others. It really comes into its own once you have lots of data, and a fair amount of relations between
your data models. If you want to track spatial data like
latitude/longitude points, you should look into GeoAlchemy, as well.
Regarding the original question, it is possible to use two different frameworks in the backend of a web app. One way to do so would be to set up a reverse proxy server (see https://www.nginx.com/resources/glossary/reverse-proxy-server/#:~:text=A%20reverse%20proxy%20server%20is,traffic%20between%20clients%20and%20servers.), but I would recommend giving SQLAlchemy before doing so.
Why do you think that flask-admin is tied to SqlServer? Flask (and flask-admin) can handle different connections to various databases:
https://github.com/flask-admin/flask-admin#introduction
https://flask-admin.readthedocs.io/en/v1.0.9/db/
My guess is you are currently using SqlAlchemy. As explained here, you can use different backends:
The string form of the URL is dialect[+driver]://user:password#host/dbname[?key=value..], where dialect is a database name such as mysql, oracle, postgresql, etc., and driver the name of a DBAPI, such as psycopg2, pyodbc, cx_oracle, etc.
Alternatively, the URL can be an instance of URL.
(https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine)
More on Engine here
Flask-admin is a admin view on your database tables - you cannot run it on a "different" db then the tables it should modify. It needs to have access to the database tables you want it to modify, so you cannot run "admin" on MS and "your data backend" on another database(-connection).
Some other things to think about:
MS-(T-)Sql and MySql are infrastructural choices, language wise they are very closely related so adapting MySql knowledge to T-SQL Syntax should be possible. Choosing SqlServer may force you to license it - and afaik that comes with fees (eiter on premise or as a azure subscription wich might or might not be free for schools (no idea - but you should check that)).
This sounds like an ambitious project for school - depending on where you live, privat data protection laws come into play especially considering you connect names with schedules with grades which would need you to implement a lot more to comply.
For first question, I checked flask-admin documentation and found that the framework already included serval built-in ORM library, i.e. SQLAlchemy, MongoEngine, pymongo and Peewee. This means you can just directly import the ORM library from the flask-admin package and use it to access your database. For your case, you should look for SQLAlchemy as you are using SQL Database. Both SQL Server and MySQL are supported by SQLAlchemy.
The Adding Model Views section in their official doc also mentioned it as well:
https://flask-admin.readthedocs.io/en/latest/introduction/#getting-started
For the second question, it is technically not possible to apply two different frameworks in one single backend application.

Querying a model based on user input in Django

I am new to Django and I am trying to build a basic search/filter feature; for example, a basic version of the refine/filter part on amazon while searching for products. (I am using Sqlite3 in development)
I think I could implement a filter in which you could click part of a form and it would return a page with the database items that match the query, however, I am not sure on how I could do this if the search contained more than one part to the query, for example if the search was to find a book that was published before 2009 and costs more than £4.99, I am unsure on how to do this.
I am looking to build a checkbox type of filter rather than a search like google.
This sort of filter/search
All help is appreciated, Thank You.
https://django-filter.readthedocs.io
This is what you're looking for.

Database design for getting best matching documents in Nosql database

In our project we have to develop a module which search for most related document available in our database. The search criteria is by using minimum three keyword (three keyword for matching accuracy).We are extract keyword manually and automatically by using keyword extraction modules. So we are not completely sure how we have to design the database and we are are confused about database selection also, most probably it will be Mongodb. So any have any suggestion or know about any article or document available in internet about these type of database design or application, sharing those will be very helpful....
You can use more-like-this query in Elasticsearch.

Python: RE vs. Query

I am building a website using Django, and this website uses blocks which are enabled for a certain page.
Right now I use a textfield containing paths were a block is enabled. When a page is requested, Django retrieves all blocks from database and does re.search on the TextField.
However, I was wondering if it is not a better idea to use a separate DB table for block/paths, were each row contains a single path and reference to a block, in terms of overhead.
A seperate DB table is definitely the "right" way to do it, because mysql has to send all the data from your TEXT fields every time you query. As you add more rows and the TEXT fields get bigger, you'll start to notice performance issues and eventually crash the server. Also, you'll be able to use VARCHAR and add a unique index to the paths, making lookups lightning fast.
I am not exactly familiar with Django, but if I am understanding the situation correctly, you should use a table.
In fact this is exactly the kind of use that DB software is designed and optimized for.
No worries. It will actually be faster.
By doing the search yourself, you are trying to implement part of the DB logic on your own. Fun, certainly, but not so fast. :)
Here are some nice links on designing a database:
http://dev.mysql.com/tech-resources/articles/intro-to-normalization.html
http://en.wikipedia.org/wiki/Third_normal_form
Hope this helps. Good luck. :-)

How to build an interactive search engine web interface using python

I have build a static web interface for searching data from some tables in my PostgreSQL database. The query website consists of a simple textfield for entering the search term, the result website presents the results as a simple html table. The server side code for searching the PostgreSQL database and returning the results is written in python using psycopg2.
Now I would like to add some interactive "Ajax features" to my search engine. When entering the search term I would like to be able to see a list of possible search terms like Google does it. On the results page, I would like to be able to sort the table showing the results.
What would be the easiest/recommended way to implement these features for my search engine web site?
I have not had to build a search outside of Django, but Haystack http://haystacksearch.org/ makes things very easy.
If you don't want to get into Django you could look at Whoosh. http://bitbucket.org/mchaput/whoosh/wiki/Home
what you call "Ajax features" are technically known as auto-suggest. Unless you want to reinvent the wheel. I would highly recommend indexing your db tables using Apache Solr. It comes with autosuggest, faceted filtering (like on most ecommerce sites) and spell-check. and since it is HTTP based you can hook into Python very easily using its RESTful API.

Categories