I currently have a free Azure Notebook account on https://notebooks.azure.com/ and would like to execute a Python script (or Jupyter Notebook) hosted on Azure automatically once every 10 minutes.
Is there a way to do this within the free Azure notebook account?
I am aware of several approaches which are described on the web, such as using Azure WebJobs, Azure Functions, Azure IoT and so on. However, all these approaches require me to upgrade to the the "Free" account which actually is only free for the first 12 months, so I would like to avoid that if possible.
As I known, there is not any feature about job like WebJobs for Azure WebApp or Jobs for Azure Databricks in Microsoft Azure Notebooks. So I tried to trigger a Python script via crontab on Ubuntu of Azure Notebooks, but failed because the cron service default not started and Azure does not offer the nbuser password for using sudo to start cron service.
However, I also tried to write a Python script hello.py as below.
from datetime import datetime
import time
while(True):
print(f"{datetime.now()} => Hello, world! ")
time.sleep(10) // 10 seconds
And I ran it in Terminal of Azure Notebooks, as the figures below, then I closed the terminal page and ran !tail -f ~/hello.log, it seems not to be terminated by the closed event of terminal page.
You can try to this way. If it's not what you want, I think it's impossible on Azure Notebooks.
There are a number of 'always free' services that come with the free Azure account which includes Azure Functions.
As long as you stay within the free functions limit, which is currently 1,000,000 requests per month, you wouldn't have to pay anything.
https://azure.microsoft.com/en-us/free/free-account-faq/
Related
I have a script that uses requests library. It is a web scaper that runs for at least 2 days and I don't want to leave my laptop on for that long. So, I wanted to run it on the Cloud but after a lot of trying and reading the documentation, I could not figure out a way to do so.
I just want this: When I run python my_program.py it shows the output on my command line but runs it using Google Cloud services. I already have an account and a project with billing enabled. I already installed the GCP CLI tool and can run it successfully.
My free trial has not ended. I have read quickstart guides but as I am fully beginner regarding the cloud, I don't understand some of the terms.
Please, help me
I think you'll need to setup a Google Cloud Compute Engine instance for that. It's basically a reserved computer/machine where you can run your code. Here's some steps that you should do just to get your program running on the cloud.
Spin up a Compute Engine instance
Gain access to it (through ssh)
Throw your code up there.
Install any dependencies that you may have for you script.
Install tmux and start a tmux session.
Run the script inside that tmux session. Depends on your program, you should be able to see some output inside the session.
Detach it.
Your code is now executing inside that session.
Feel free to disconnect from the Compute Engine instance now and check back later by attaching to the session after connecting back into your instance.
I have a script that downloads larger amounts of data from an API. The script takes around two hours to run. I would like to run the script on GCP and schedule it to run once a week on Sundays, so that we have the newest data in our SQL database (also on GCP) by the next day.
I am aware of cronjobs, but would not like to run an entire server just for this single script. I have taken a look at cloud functions and cloud scheduler, but because the script takes so long to execute I cannot run it on cloud functions as the maximum execution time is 9 minutes (from here). Is there any other way how I could schedule the python script to run?
Thank you in advance!
For running a script more than 1h, you need to use a Compute Engine. (Cloud Run can live only 1h).
However, you can use Cloud Scheduler. Here how to do
Create a cloud scheduler with the frequency that you want
On this scheduler, use the Compute Engine Start API
In the advanced part, select a service account (create one or reuse one) which have the right to start a VM instance
Select OAuth token as authentication mode (not OIDC)
Create a compute engine (that you will start with the Cloud Scheduler)
Add a startup script that trigger your long job
At the end on the script, add a line to shutdown the VM (with Gcloud for example)
Note: the startup script is run as ROOT user. Take care of the default home directory and the permission of the created files.
I am trying to schedule a python script to run on my GCE machine running ubuntu 18.04. Some points I am considering:
The VM is running 24/7 as it is on the free tier
I went through the following docs https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule and it suggests a way to start and stop the VM using cloud functions and scheduler but i am looking to run my python script on the running machine
I have to run it on GCE as cannot use cloud functions/app engine /cloud runs as the script runs for around 1 hour 30 mins which would cause a timeout on all the other compute options.
I am new to ubuntu and GCP so excuse me if there is a trivial solution i am missing.
Any pointers to trigger specific scripts on GCE using cloud functions would also be helpful as i might need to turn on and off the VM if i upgrade to a higher tier
I have a Python script that pulls some data from an Azure Data Lake cluster, performs some simple compute, then stores it into a SQL Server DB on Azure. The whole shebang runs in about 20 seconds. It needs sqlalchemy, pandas, and some Azure data libraries. I need to run this script daily. We also have a Service Fabric cluster available to use.
What are my best options? I thought of containerizing it with Docker and making it into an http triggered API, but then how do I trigger it 1x per day? I'm not good with Azure or microservices design so this is where I need the help.
You can use Web Jobs in App Service. It has two types of Azure Web Jobs for you to choose: Continuous and Trigger. As I see you need the type Trigger
You could refer to the document here for more details.In addition, here shows how to run tasks in WebJobs.
Also, you can use Azure function timer-based on python which was made generally available in recent months.
Say I have a file "main.py" and I just want it to run at 10 minute intervals, but not on my computer. The only external libraries the file uses are mysql.connector and pip requests.
Things I've tried:
PythonAnywhere - free tier is too limiting (need to connect to external DB)
AWS Lambda - Only supports up to Python 2.7, converted my code but still had issues
Google Cloud Platform + Heroku - can only find tutorials covering deploying applications, I think these could do what I'm looking for but I can't figure out how.
Thanks!
I'd start by taking a look at this question/answer that I asked previously on unix.stackexchange - I went with an AWS redhat installation and it was free to use.
Once you've decided on your VM, you can add SSH onto your server using any SSH client and upload your Python script. A personal preference is this application.
If you need to update the Python version on the server, you can do this by installing the required Python RPMs. A quick google should return the yum [or whichever RPM management system you're using] repository for the required RPMs.
Once you've installed the version of Python that you need, I'd suggest looking into the 'crontab' which can be used to schedule jobs. You can set a cronjob to run every 10minutes which will call your script.
See this site for more information on how to use the crontab
This sounds like a perfect use case for AWS Lambda which supports Python. You can invoke your Lambda on a schedule using Scheduled Events.
I see that you tried Lambda and it didn't work out for you which is too bad as that seems like the easiest route. You could also launch an EC2 instance and use userdata to schedule a cron when the instance starts.
Another option would be an Elastic Beanstalk worker with a cron.yml that defines your schedule. Elastic Beanstalk supports Python 3.4.
Update: AWS does now support Python 3.6. Just select Python 3.6 from the runtime environments when configuring.