I am currently trying to branch out my python skills and want to learn more about how to use cloud platforms, where I have chosen Google as I am not too keen on Amazon and Microsoft's business practices.
To learn about this what I want to do is run a machine learning script on Google Cloud Platform (GCP) which any of my local scripts can query and get a response from, e.g. ping the ML script with news headlilne and then the ML scipt estimates who published it.
I thought this was a good exercose as it would require me to run the script on a VM on GCP, store my machine learning weights on Cloud Storage and then interact with this script from my local machine. I am currently trying out the GCP courses at Coursera (more specifically Google Cloud Platform Fundamentals), however it is very general and I haven't found any other good tutorials around that could help me get started with this.
I know this is a semi broad question, but if anyone have any tutorials, books, papers, tips and tricks etc., in mind to help me get started on this that would be really helpful. (I have googled around now for several days but the best I could find was the Coursera courses, which I am doing but it doesn't feel totally ideal).
Apologies in advance if this is not the right place to ask this question.
What you are trying to achieve can be easily done using Cloud Functions (GF) or at the most App Engine (AE) (serverless app hosting). If it is simple and completes one or two actions go with GF. On the other hand, AE will give you more flexibility and room for advanced API integrations. You can host your 'app' in one of the above options and run your ML script of BigQuery using the data from your Google Storage.
As for your practicing, I strongly recommend Qwiklabs or Codelabs
Related
I have a python script which run 24 hours on my local system and my script uses different third party libraries that are installed using pip in python
Libraries
BeautifulSoup
requests
m3u8
My python script is recording some live stream videos from a website and is storing on system. How google cloud will help me to run this script 24/hours daily and 7days a week.I am very new to clouds. Please help me i want to host my script on google cloud so i want to make sure that my script will work there same as it is working on local system so my money will not lost .
Yes it can. I would recommend familiarizing yourself with this Quickstart: Deploy a Python Service to Cloud Run and What is Cloud Run. When you use Cloud Run, you can provide your own Docker image that uses Python, or select from preexisting images.
Once you have a Cloud Run instance running, you can tie it into other Cloud Run instances or Cloud Functions which are scalable functions that use Cloud Run under-the-hood and allow you to easily scale your app. Additionally, these instances spin down to 0 if nobody is using the app which saves costs greatly. This can be modified of course so that the app is always spun-up.
In general I highly recommend looking at Cloud Run but also other services can handle the task such as a Compute Engine.
If you want to run 24/7 application on the cloud, whatever the cloud, you must not use solution with timeout (like Cloud Run or Cloud Functions).
You can imagine using App Engine flex, but it won't be my best advice.
The most efficient for me (low maintenance, cost efficient), is to use GKE autopilot. A Kubernetes cluster managed for you, you pay only the CPU/Memory that your workloads use.
You have to containerize your app to do that.
So, I am working on a project that has a backend optimization algorithm written in GAMS.
The ideal solution is to be able to use this GAMS code that is already written with the python API for GAMS. I want to be able to call this code through an HTTP request and run the algorithm, so I wanted to make a Flask server for this. Ideally, it could run in Google's App Engine, but GAMS software must be installed.
I am not sure if this is possible on App Engine, or if it can be done in a Google instance.
The data input would be from CSV's in google cloud storage and the output would be put there as well.
I was wondering if anyone has tried this before or if you know more about Google cloud and think this will or will not work. I could not find much about this online.
Your question would benefit from more detail and including references e.g. Is this GAMS?
App Engine standard is opinionated and it may not (don't know) be possible for you to bundle GAMS as part of a Python deployment to App Engine standard.
However, if you were willing to bundle a Python (Flask) server and GAMS together, it's possible you could containerize the solution and run it on App Engine flexible as a custom runtime.
Alternatives exist and may be better matches to your needs including detaching the (Flask) frontend server from the GAMS backend (?) and deploying these on App Engine (or possibly Cloud using Functions, Cloud Run, Compute Engine etc.).
Someone who has done this may be willing to provide additional pointers|guidance.
The best solution to this problem, I found, was to get rid of GAMS entirely.
There were many ways I wished to integrate this model into my web app, so I translated the model to Python using PyOMO. The performance was slightly affected, but the model can now be run in Google Cloud.
I set up the model in a Flask server and now built a REST API around it for use in the web app.
I have a Machine learning module which uses Google Colab's free GPU for NLP tasks, and I want to make a web app out of it. I've been thinking of using React js for frontend and spring boot for the back end and was wondering whether there is a way to connect the code at Google Colab with the backend.
Want to know other alternative suggestions to building a web app incorporating the ML module in Colab as well. Any sort of help is appriciated.
It's possible. But there's no good examples of it.
There's an example of serving local resources, using http-server. Flask examples used to work too. But not much else.
I have been using Google Apps Script for a while and quite like how it can easily be deployed on the web. My question is: is there something similar to Google Apps Script for Python? That is, some where we can create a Python application and run it completely online without having to package or containerize the app.
I am also looking at Docker and Google's App Engine right now but haven't been able to get a clear picture of pros and cons.
Note: the practical application of what I am asking includes
manipulating large databases and let different clients run
manipulation scripts on their own cloud database themselves.
This is a very broad question, and not really the kind that Stackoverflow can help with. Trying reading How to create a Minimal, Complete, and Verifiable example?.
But to attempt to answer your question I need to break it down.
1) I assume that you are using the script editor / IDE for developing / debugging GScript (aka JavaScript) code. And that is the simplest way to develop Google Apps Script (GAS) software. There is no alternative to this for python or any other language. However it is not obligatory to use the IDE to write software that uses the Google Apps Script APIs.
2) This link, for example, is the Google Apps Script API quickstart that demonstrates how to use the GAS APIs from python.
3) However, to look at the big picture... you can regard GAS as an example of Serverless Computing, where the Google App Engine, AWS Lambda, and others (see the link) provide a straightforward environment for developing and deploying software without having to worry about packaging, containers, etc.
Docker might offer some advantages over traditional server deployment, but I would not regard it as an equivalent of serverless, as there is still a container environment to implement and manage.
The Pros and Cons of each are highly subjective, depending upon the nature of the problem your software needs to solve, and your starting point - if your business is already heavily into Azure or AWS then that might be the way to go. But if you require access to Google APIs for Maps, etc. then that might be the way to go, otherwise it probably boils down the specific services on offer, and cost.
Let me explain what I'm trying to achieve. In the past while working on Java platform, I used to write Java codes(say, to push or pull data from MySQL database etc.) then create a war file which essentially bundles all the class files, supporting files etc and put it under a servlet container like Tomcat and this becomes a web service and can be invoked from any platform.
In my current scenario, I've majority of work being done in Java, however the Natural Language Processing(NLP)/Machine Learning(ML) part is being done in Python using the NLTK, Scipy, Numpy etc libraries. I'm trying to use the services of this Python engine in existing Java code. Integrating the Python code to Java through something like Jython is not that straight-forward(as Jython does not support calling any python module which has C based extensions, as far as I know), So I thought the next option would be to make it a web service, similar to what I had done with Java web services in the past. Now comes the actual crux of the question, how do I run the ML engine as a web service and call the same from any platform, in my current scenario this happens to be Java. I tried looking in the web, for various options to achieve this and found things like CherryPy, Werkzeug etc but not able to find the right approach or any sample code or anything that shows how to invoke a NLTK-Python script and serve the result through web, and eventually replicating the functionality Java web service provides. In the Python-NLTK code, the ML engine does a data-training on a large corpus(this takes 3-4 minutes) and we don't want the Python code to go through this step every time a method is invoked. If I make it a web service, the data-training will happen only once, when the service starts and then the service is ready to be invoked and use the already trained engine.
Now coming back to the problem, I'm pretty new to this web service things in Python and would appreciate any pointers on how to achieve this .Also, any pointers on achieving the goal of calling NLTK based python scripts from Java, without using web services approach and which can deployed on production servers to give good performance would also be helpful and appreciable. Thanks in advance.
Just for a note, I'm currently running all my code on a Linux machine with Python 2.6, JDK 1.6 installed on it.
One method is to build an XML-RPC server, but you may wish to fork a new process for each connection to prevent the server from seizing up. I have written a detailed tutorial on how to go about this: https://speakerdeck.com/timclicks/case-studies-of-python-in-parallel?slide=68.
NLTK based system tends to be slow at response per request, but good throughput can be achieved given enough RAM.