I know this is not a direct code related question but more a best practices. I have several azure HTTPfunctions running but they timeout due to long calculations. I have added Durable orchestrations but even they time out.
As certain processes are long and time consuming (aka training an AI model) I have switched to Azure VM. What I would like to add to this is the possibility to start an Python task from an HTTP request on my azure VM.
basically doing the exact same as the Azure HTTPFunctions. What would be the best way to do this, any great documentation or recommendations much appreciated. So running an API on my VM in Python.
Related
Usually things go other way - you use DataDog to monitor Airflow, but in my case I need to access DataDog metrics over some DataDog API from Airflow so I can send it to Snowflake table.
Idea it to use this table to build alerting system in ThoughtSpot Cloud when Kafka lag happens since ThoughtSpot Cloud doesn't support calling API, at least not from cloud version.
I'm googling over this options endlessly but not finding any more optimal and less complicated solution. Any advices is highly appreciated.
I have connection between Google Sheets and python script that I read cells from a column then read these cells's data and then run the code to do its job.
So basically, what I want to do is setting up a webhook or anything that Make the code runs always and catching a new data in the google sheet.
I made some searches I got 3 Apps can do this but I don't know which one is suitable
1- Cloud Run
2- Cloud Build API
3-Cloud Deploy
and how to setup the the webhook, pretty appreciate
Cloud Scheduler is a fully managed cron job scheduler. It allows you to schedule virtually any job. You can automate everything, including retries in case of failure to reduce manual toil and intervention. Cloud Scheduler even acts as a single pane of glass, allowing you to manage all your automation tasks from one place.
I am currently trying to branch out my python skills and want to learn more about how to use cloud platforms, where I have chosen Google as I am not too keen on Amazon and Microsoft's business practices.
To learn about this what I want to do is run a machine learning script on Google Cloud Platform (GCP) which any of my local scripts can query and get a response from, e.g. ping the ML script with news headlilne and then the ML scipt estimates who published it.
I thought this was a good exercose as it would require me to run the script on a VM on GCP, store my machine learning weights on Cloud Storage and then interact with this script from my local machine. I am currently trying out the GCP courses at Coursera (more specifically Google Cloud Platform Fundamentals), however it is very general and I haven't found any other good tutorials around that could help me get started with this.
I know this is a semi broad question, but if anyone have any tutorials, books, papers, tips and tricks etc., in mind to help me get started on this that would be really helpful. (I have googled around now for several days but the best I could find was the Coursera courses, which I am doing but it doesn't feel totally ideal).
Apologies in advance if this is not the right place to ask this question.
What you are trying to achieve can be easily done using Cloud Functions (GF) or at the most App Engine (AE) (serverless app hosting). If it is simple and completes one or two actions go with GF. On the other hand, AE will give you more flexibility and room for advanced API integrations. You can host your 'app' in one of the above options and run your ML script of BigQuery using the data from your Google Storage.
As for your practicing, I strongly recommend Qwiklabs or Codelabs
I have a Python script that pulls some data from an Azure Data Lake cluster, performs some simple compute, then stores it into a SQL Server DB on Azure. The whole shebang runs in about 20 seconds. It needs sqlalchemy, pandas, and some Azure data libraries. I need to run this script daily. We also have a Service Fabric cluster available to use.
What are my best options? I thought of containerizing it with Docker and making it into an http triggered API, but then how do I trigger it 1x per day? I'm not good with Azure or microservices design so this is where I need the help.
You can use Web Jobs in App Service. It has two types of Azure Web Jobs for you to choose: Continuous and Trigger. As I see you need the type Trigger
You could refer to the document here for more details.In addition, here shows how to run tasks in WebJobs.
Also, you can use Azure function timer-based on python which was made generally available in recent months.
I'm a complete novice in this area, so please excuse my ignorance.
I have three questions:
What's the best (fastest, easiest, headache-free) way of hosting a python program online?
I'm currently looking at Google App Engine and Web Frameworks for Python, but all the options are a bit overwhelming.
Which gui/viz libraries will transfer to a web app environment without problems?
I'm willing to sacrifice some performance for the sake of simplicity.
(Google App Engine can't do C libraries, so this is causing a dilemma.)
Where can I learn more about running a program locally vs. having a program continuously run on a server and taking requests from multiple users?
Currently I have a working Python program that only uses standard Python libraries. It currently uses around 2.7gb of ram, but as I increase my dataset, I'm predicting it will use closer to 6gb. I can run it on my personal machine, and everything is just peachy. I'd like to continue developing on the front end on my home machine and implement the web app later.
Here is a relevant, previous post of mine.
Depending on your knowledge with server administration, you should consider a dedicated server. I was doing running some custom Python modules with Numpy, Scipy, Pandas, etc. on some data on a shared server with Godaddy. One program I wrote took 120 seconds to complete. Recently we switched to a dedicated server and it now takes 2 seconds. The shared environment used CGI to run Python and I installed mod_python on the dedicated server.
Using a dedicated server allows COMPLETE control (including root access) to the server which allows the compilation and/or installation of anything. It is a bit pricy but if you're making money with your stuff it might be worth it.
Another option would be to use something like http://www.dyndns.com/ where you can host a domain on your own machine.
So with that said, perhaps some answers:
It depends on your requirements. ~4gb of RAM might require a dedicated server. What you are asking is not necessarily an easy task so don't be afraid to get your hands dirty.
Not sure what you mean here.
A server is just a computer that responds to requests. On the dedicated server (I keep mentioning) you are operating in a Unix (or Windows) environment just like you would locally. You use SOFTWARE (e.g. Apache web server) to serve client requests. My vote is mod_python.
It's a greater headache than a dedicated server, but it should be much closer to your needs to go with an Amazon EC2 instance.
http://aws.amazon.com/ec2/#instance
Their extra large instance should be more than large enough for what you need to do, and you only turn the instance on when you need it so you don't have the massive bill that you get with a dedicated server that's the same size.
There are some nice javascript based visualization toolkits out there, so you can model your application to return raw (json) data and render that on the client.
I can mention d3.js http://mbostock.github.com/d3/ and the JavaScript InfoVis Toolkit http://thejit.org/