I have a setup where I will have having around 50 sites using the same code, configured slightly differently. Rather than deploy the same code over and over again, duplicated across different folders and repositories, is there any way in Flask to centralise the working code of the site, as some sort of library?
In Django they have something a bit like this:
https://docs.djangoproject.com/en/dev/ref/contrib/sites/
Some ideas
Deploy 50 instances of UWSGI, duplicating the same code and different config
Deploy 50 instances of UWSGI, with the python code added as a sort of module or extension so there is only one instance of the code: http://flask.pocoo.org/docs/extensiondev/
Deploy 1 instance of UWSGI which has only one instance of the code and handles different hostnames: http://flask.pocoo.org/docs/patterns/appdispatch/
The code that I am duplicating is designed to query an API and show the results. The differences between the sites are two fold:
Templating - Although the sites will look similar, they will not be identical. They will have slightly different CSS and images.
The API query. Most of the smaller sites are for towns and cities. This means that the API request from these sites will be slightly modified so as to return results only in that area.
sitelondon.com might query the API for items only in London by default
sitehtml.com might query the API for items which have the "html" keyword by default
The focus on my end is performance for the user. I will be running these initially on a server with 2GB RAM which should be plenty.
Normally for these types of scenarios the "behaves differently based on Host" logic is built in the application.
So the better solution is #3 but my suggestion is to not use app dispatch.
Build the logic to get the configuration for the hostname directly in the main application (for example you can load the specialized configurations in a #before_request handler and use a single DB instance).
If you plan to use only one small server, as you said, this solution is light on resource.
50 different uWSGI instances with their own processes would fill your memory and start swapping easily.
I agree with #Paolo that #3 is your best option.
You could simplify it even more with URL rewriting in your web server. If you rewrite the URLs such that a query for http://sitelondon.com/example becomes http://sitelondon.com/london/example, and a query for http://sitehtml.com/example becomes http://sitehtml.com/html/example then you can easily get the site through the routes:
#app.route('/<site>/example')
def example(site):
return render_template(site + '/example.html')
With this setup you can organize your templates in sub-folders based on the site name, and then selecting the proper template becomes a matter of building the template path.
I hope this helps!
Related
I have a generic flask application. The application is instrumented using opentelemetry-instrumentation-flask, I am shipping this data with the opentelemetry-exporter-otlp to an Elastic APM server. This is all working fine and it is done as the documentation shows.
There are some endpoints of the application that I would like not to track with the instrumentation, as they are noisy and add little-to-no value for me (For example, health endpoints). I want the instrumentation to ignore them, but I cannot find how.
How can this be done? I have been checking the documentation for Python and after searching over the internet, I could not find any clear answer about how this could be done... because I believe this must be doable.
You can do that using environment variable OTEL_PYTHON_FLASK_EXCLUDED_URLS. It takes a comma separated regular expressions for urls you want to exclude. For more detailed info please check here https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-flask#exclude-lists
So I recently became interested into knowing how to create a URL shortener without using bittly or other things, but I am not very good at using python to connect with other stuff. All I know is:
Checking to see if the URL is available ( Only to see if it has HTTP:// and unavailable characters, nothing to see if the domain is occupied or not. )
All of the other things... I need help with.
By the way, I COMPLETLY do not understand how to do that, so it would be great if you add comments to show me what is going on.
I suggest you take a look at Flask, it is a framework for building web applications (APIs, web apps, etc.).
DigitalOcean has a nice tutorial on this.
You can either use hashing algorithms for the custom shortened urls, or even let the user pick more readable names (like bit.ly/my-url). In this case you would be storing in a database the shortened url and the long url.
I have a server which generates some data on a daily basis. I am using D3 to visualise the data (d3.csv("path")).
The problem is I can only access the files if they are under my static_dir in the project.
However, if I put them there, they do eventually get cached and I stop seeing the updates, which is fine for css and js files but not for the underlying data.
Is there a way to put these files maybe in a different folder and prevent caching on them? Under what path will I be able to access them?
Or alternatively how would it be advisable to structure my project differently in order to maybe avoid this operation in the first place. Atm, I have a seperate process that generates the data and stores it the given folder which is independent from the server.
Many thanks,
Tony
When accessing the files you can always add ?t=RANDOM to the request in order to get a "new" data all the time.
Because the request (on the server-side) is "new" - there will be no cache, and from the client side it doesn't really matter.
To get a new random you can use Date.now():
url = "myfile.csv?t="+Date.now()
This is a bit of a strange question, I know, but bear with me. We've developed a RESTful platform using Python for one of our iPhone apps. The webapp version has been built using Django, which makes use of this API as well. We were thinking it would be a great idea to use Django's built-in control panel capabilities to help manage the data.
This itself isn't the issue. The problem is that everyone has decided it would be best of the admin center was essentially a client that sits on top of the RESTful platform.
So, my question is, is there a way to manipulate the model layer of Django to access our API directly, rather than communicated directly with the database? The model layer would act as the client passing requests and responses to and from the admin center.
I'm sure this is possible, but I'm not so sure as to where I would start. Any input?
I remember I once thought about doing such thing. At the time, I created a custom Manager using a custom QuerySet. And I overrode some methods such as _filter_or_exclude(), count(), exists(), select_related(), ... and added some properties. It took less than a week to become a total mess that had probably no chance to work one day. So I immediately stopped everything and found a more suitable solution.
If I had to do it once again, I would take a long time to consider alternatives. And if it really sounds like the best thing to do, I'd probably create a custom database backend. This backend would, rather than converting Django ORM queries to SQL queries, convert them to HTTP requests.
To do so, I think the best starting point would be to get familiar with django source code concerning database backends.
I also think there are some important things to consider before starting such development:
Is the API able to handle any Django ORM request? Put another way: Will any Django ORM query be translatable to an API request?
If not, may "untranslatable" queries be safely ignored? For instance, an ORDER BY clause might be safe to ignore. While a GROUP BY clause is very unlikely to be safely dismissed.
If some queries can't be neither translated nor ignored, may them be reasonably emulated. For instance, if your API does not support a COUNT() operation, you could emulate it by getting the whole data and count it in python with len(), but is this reasonable?
If they are still some queries that you won't be able to handle (which is more than likely): Are all "common" queries (in this case, all queries potentially used by Django Admin) covered and will it be possible to upgrade the API if an uncovered case is discovered lately or is introduced in a future version of Django?
According to the use case, there are probably tons of other considerations to take, such as:
the integrity of the data
support of transactions
the timing of a query which will be probably much higher than just querying a local (or even remote) database.
I have a large amount of automatically generated html files that I would like to push to my Plone website with a script. I currently generate the files, log into Plone, click edit on each individual page and copy and paste the html into the editor. I'd like to automate this. It would be nice to retain the plone versioning, have a auto generated comment for the edit, and come from a specific user.
I've read and tried Webdav with little luck at getting it working consistently and know that there is a way to connect to plone via ftp, but haven't tried it. I'm not sure if these are the methods that I need.
My google searches aren't leading me to anything useful. Any ideas on where to start looking for a solution to this? Or any tips on implementing it?
You can script anything in Plone via the following methods:
Through-the-web via API calls (e.g. XML-RPC, wsapi, etc.)
The bin/instance run script provided by plone.recipe.zope2instance (See charm for an example of this).
You can also use a migration framework like:
collective.transmogrifier
which allows you to write migration code, and trigger it via GenericSetup or Browser view. Additionally, there are applications written on top of Transmogrifier aimed roughly at what you are describing, the most popular of which is:
funnelweb
I would recommend that you consider using or writing a Transmogrifier "blueprint(s)" to do your import, and execute the pipeline with a tool that makes that easy:
mr.migrator
You can find blueprints by searching PyPI for "transmogrify". One popular set of blueprints is:
quintagroup.transmogrifier
One of the main attractions to the Transmogrifier approach, aside from getting the job done, is the ability to share useful blueprints with others.
I think transmogrifier is the best tool for this job, but this will definitely be a programming task no matter how you do it. It's used for many such migration jobs such as migrating from drupal.
There's an add-on, wsapi4plone.core that pumazi at WebLion started that provides web services for portals which you can then hook into. You can create, modify, delete content via XML-RPC calls. The only caveat is that it doesn't yet work with Collections (criteria specifically).
project: http://pypi.python.org/pypi/wsapi4plone.core
docs: http://packages.python.org/wsapi4plone.core/
You can also do it programmatically by hooking into the ZODB via Python (zopepy or some other method).
These should get you started:
http://plone.org/documentation/kb/manipulating-plone-objects-programmatically/reading-and-writing-field-values - you should be able to get an understanding of accessors and mutators (setters and getters), in your case you are going to be more than likely working with obj.Text (getter) and obj.setText (setter).
https://weblion.psu.edu/trac/weblion/wiki/AutomatingObjectCreation - lots of examples (slightly outdated but still relevant)
http://plone.org/documentation/faq/upload-images-files
Try to enable Webdav or ftp in Plone, then you can access Plone via webdav or ftp clients, pushing the html files. Plone (Zope) will recognises the html files as Pages.