Just start using the firebase + react to build a website. One of the designed features of my website is to crawl and show users the data parsed from another website (e.g., the stock price changes). I already have a python crawler responsible to parse the data, but I have no idea how to execute this python crawler (in the background) of my server in firebase (or it is not even possible)?
Here is the example usage of my system
user login and subscribe the website/data they are interesting
my crawler will parse that website every 1 hour and update the data to database
user can see the summary of change of website from database
One option I have in mind is running the crawler in my local machine and use the REST api to update the parsed data to firebase database. However, it seems a very inefficient/naive approach because it is kind of losing the meaning of deploying my server with cloud service, like firebase.
Firebase does not have any service/feature that allows you to periodically run Python or any other code. The closest thing to that is Cloud Functions, which can be triggered through an external service like cron-job.org.
For more in this, see:
Firebase Hosting Flask Application (on running Python on Firebase Hosting)
Using google cloud function to spawn a python script (for an elaborate way where you might apparently run Python on Cloud Functions, but I never have, nor am likely to ever try this myself)
Cloud Functions for Firebase trigger on time? (for running Cloud Functions periodically either through AppEngine, or cron-job.org).
Related
This is what I want to achieve:
Ask the user to authorize the collection of their data on a Google Analytics 4 property (or Universal Analytics but I would rather not)
Programmatically retrieve and store the data every n-hours
I was able to do (1) client-side by asking for authorization with google's OAUTH2 and making a call to Reporting API v4 https://developers.google.com/analytics/devguides/reporting/core/v4 using gapi on the front-end.
However, I'm not sure how to do it on a schedule without user interaction. I've searched Google's API docs and I believe there's a way to do it in python https://developers.google.com/analytics/devguides/reporting/core/v4/quickstart/service-py but I am currently limited to Node and the browser. I guess I could make a server in python that does the data fetching and connects with the Node application, but that's yet another layer of complications that I'm trying to avoid. Is there a way to do everything in Node?
GCP APIs are all documented in a way which allows everyone to generate client libraries in a variety of languages, including node.js. The documentation for the node.js client for Analytics Reporting is here.
For the question of how to schedule this on GCP, I would recommend you to use Cloud Scheduler. This will hit an endpoint running on Cloud Run, which will do the actual work. Alternatively, if you already have a service running somewhere else, you can simply add the required endpoints there and point Cloud Scheduler to it.
The overall design which I would suggest you goes something like this:
Build a site which takes the user through the OAUTH2 login process,
requesting the relevant Google Analytics Reporting API scopes
required to make the request.
Store the obtained credentials in their user database.(preferably
Firestore in Datastore mode)
Set up a Cloud Run service (or anything else), with two endpoints
Iteration endpoint: Iterate through the list of users and add tasks
to Cloud Tasks to hit the download endpoint for each one.
Download endpoint: Takes a user ID (e.g. as a query parameter) and
performs the download for this user. You will need to load the
credentials for the user from the database and use this to access the
reporting API.
Store the downloaded data in the desired location, e.g. Cloud
Storage, Firestore, Cloud SQL, etc.
Set up Cloud Scheduler to hit the iteration endpoint at the desired
frequency.
For the GCP services mentioned above, basically everything other than Analytics, you may use the "cloud" clients for node.js, which are available here
Note : The question you have asked is a very broad question and this answer is just a suggestion. You may think about other designs whichever works best for you.
I found a python package that I want to use within my angular based firebase project (it does some complex analysation of a text file).
What is the best way to use it? I see the following options:
Own docker container with flask, running in Cloud run (e.g. like that) - pass file in AJAX request, return JSON result.
Downsides: own endpoint that has to be noted down somewhere in main project, own repository: not in the other Node.js cloud functions
Call python script in Node.js cloud function (like this)
a little bit hacky piping file and log outputs as strings, probably not easy to get all the python dependencies working (would this work at all?)
Totally independent microservice just taking the file, analysing it and sending back JSON. Maybe as AWS lambda?
again "cut out" of the main project
I'd love to have a "clean and easy" integration in my existing Node.js cloud functions that I use in firebase. The firebase CLI can then take over all the URL endpoint handling etc. But I don't see a way to do this.
A kind of better capsulated approach would be to go with 1. or 3. and have a Node.js cloud function calling the endpoint. With that I would also not have the client code call the endpoint and have better possibilites in configuring it without having to update the client code.
Am I missing an approach? What would be the best way to do this?
Use Case: User uploads file, the file and some other values are saved to his account. The content of the file should be analysed (can be done asynchronously) and the results should be available for the user to be shown.
I went with option 1 and it is working quite well so far. It's very easy to deploy the dockerized flask server on Cloud Run. Also it's possible to control access and authentication from the firebase function to the Cloud Run container with Google's IAM.
I have a running python application that needs to receive some data and process them. and I also have a PHP server that can get these data. I want to send JSON data from PHP to my python app.
anyway except running a python web server and send data to it, or insert into DB and get from DB with python?
thanks.
I tried using python cherryPy web server.
#Niklas D It would be easier to answer your question, if you can give some more context about the application or use case you want to solve.
Some further possibilities are:
Glue Code (I never did it with python and php only C++ with python, but you should be able to find examples on the internet e.g. https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages#PHP )
Messaging Systems like RabbitMQ, ActiveMQ, ZeroMQ, etc.
Redis (I know you said except writing to a database, but Redis provides some features for publish subscribe https://redis.io/commands/pubsub which allows you to write to Redis from the one side and get data on the other side without polling the db all the time, which is the issue you have with using a database I guess) It's a bit easier to setup and use, than a messaging system.
TCP connection between the python and php application. https://medium.com/swlh/lets-write-a-chat-app-in-python-f6783a9ac170
If you want to send data to a python application using web protocols, i.e send POST, GET requests etc then you need to create a python web app to receive and handle those requests. Which in turn needs to be running off a webserver or you could build serverless functions to handle this, see https://serverless.com/
If you want to get data using a python application, i.e the python app sends POST and GET requests etc to your php app to ask for the JSON payload you can build an app using python's standard requests library https://docs.python.org/3/library/urllib.request.html or better still us the Requests package http://docs.python-requests.org/en/master/
Or you could do something and save the JSON file to disk and then open it with your python app. You'd need to set up scheduling or make your php app execute python code on the server... This last suggestion is a bad idea please don't unless your app is isolated and not publicly accessible or you know how to lock down your security.
I have a Webapp that writes the status of Azure VMs in a DB. I want to be able to have the DB update automatically every time a VM is started/stopped/restarted. How do I set up a trigger that POST to my Webapp when the VM is started/stopped/restarted? I'm using the Azure Python SDK. Thank you.
You don't need to write code to achieve this you could use EventGrids in Azure linked to a logic app which would write straight into your database. Here is an example of how to set up an EventGrid to send emails when linked to a logic app. This is an example in how to connect a Logic App to a database. The combination of these two examples will allow you to use Azure native technologies to achieve your goal.
Alternatively instead of writing straight to the database you could use the logic app to POST to your website using one of the connectors.
Hope that helps.
I'm trying to build a django app that can monitor and interact with a remote database (to interact with the database in a basic way - just performing a look-up and only sometimes making a little change in the remote data), it also has to sometimes store the remote data to its own database.
The website which sits on the remote database is a community website, and anyone without an account is allowed to post on the website's forums. I want the app to be able to check the database every now and then to see for any updates in the discussions. The site gets at least 100 posts an hour and since anyone is allowed to post on the forums without an account, it occasionally gets spammed, but unfortunately the CMS that is being used does not have a good anti-spam system set up.
Only way that I can think of at the moment is to make a python file, and in that file I can import MySQLdb. I can connect to the remote database (mysql) server and select all the posts that have been made in the last X minutes. Using a function that calculates the probability of a post being a spam or not, I can again talk to the remote database and flag the candidates to be not displayed on the website. I can have this file run "every now and then" using cron.
The problem here is a lack of control. I want to have a user interface that can show all the spam candidates on a single webpage and have an "unflag" button to make accidentally flagged posts to be shown on that website again. This means that I'll probably be better off writing a django web app than to write a single python script that simply just flags spam candidates.
How would I have a django app or perhaps a function within that app (which can perform all actions that the stand-alone python script as described above can perform) to run automatically every now then (say every minute)?
Maybe you should try django-celery?