I have a code with Jupyter notebook and i would like to schedule daily running by Google Cloud.
I already created VM instance and running my code there,
but I couldn't find any guide or video how to implement daily running.
So, how can I do that?
Google is offering a product which is called AI Platform Notebooks. It is implementing lots of useful stuff like lots of open-source frameworks, CI etc. There is also a blog post by the Google Cloud that explains the product in depth and can be found here. I think you can use that to achieve what you want.
You can use cron to schedule the notebook in the VM machine. Please take a look at nbconvert or papermill for executing notebooks.
The other ways to schedule Jupyter Notebook can be to use a web-based application for notebook scheduling:
Mercury
Notebooker
Both of them can automatically execute the notebook in the background and share the resulting notebook as a website.
Related
I have python scripts for automated trading for currency and I want to deploy them by running on Jupter Lab on a cloud instance. I have no experience with cloud computing or linux, so I have been trying weeks to get into this cloud computing mania, but I found it very difficult to participate in it.
My goal is to set up a full-fledged Python infrastructure on a cloud instance from whichever provider so that I can run my trading bot on the cloud.
I want to set up a cloud instance on whichever provider that has the latest python
installation plus the typically needed scientific packages (such as NumPy and pandas and others) in combination with a password-protected and Secure Sockets Layer (SSL)-encrypted Jupyter
Lab server installation.
So far I have gotten no where. I am currently looking at the digital ocean website for setting jupter lab up but there are so many confusing terms.
What is Ubuntu or Debian? Is it like a sub-variant of Linux operating system? Why do I have only 2 options here? I use neither of the operating system, I use the windows operating system on my laptop and it is also where I developed my python script. Do I need a window server or something?
How can I do this? I tried a lot of tutorials but I just got more confused.
Your question raises several more about what you are trying to accomplish. Are you just trying to run your script on cloud services? Or do you want to schedule a server to spin up and execute your code? Are you running a bot that trades for you? These are just some initial questions after reading your post.
Regarding your specific question regarding Ubuntu and Debian, they are indeed Linux distributions which are popular option for servers. You can set up a Windows server on AWS or another cloud provider, but Linux distributions being much more popular are going to have lots of documentation, articles, stackoverflow posts around a Linux based server.
If you just want to run a script on a cloud on demand, you would probably have a lot of success following Wayne's comment around PythonAnywhere or Google Colab.
If you want your own cloud server, I would suggest starting small and slow with a small or free tier EC2 instance by following a tutorial such as this https://dataschool.com/data-modeling-101/running-jupyter-notebook-on-an-ec2-server/ Alternatively, you could splurge for an AWS AMI which will have much more compute power and be configured.
I have similar problem and the most suiteble solution to me is using docker container for jupyter notebooks. The instructions on how to install Docker can be found at https://docs.docker.com/engine/install/ubuntu/ There is ready to use Docker image docker pull jupyter/datascience-notebook for jupyter notebook python stack. The docker compose files und sone addional insruction you will fid at https://github.com/stefanproell/jupyter-notebook-docker-compose/blob/master/README.md.
I'm a total noob at Python, I just downloaded Anaconda and started to use Jupyter Notebook.
I was wondering: since Jupyter Notebook looks web based, should I have any privacy concerns using it? i.e. are the data on my pc exposed out in the web?
You probably shouldn't worry about running notebooks on your localhost. If you want more info (e.g. if you ever intend to run your notebook at a remote server), this link will give you some insight regarding security concerns.
TL;DR: no, your data are not exposed.
I have a problem and I need a hint how to approach the problem.
I have django application, in which I have sme jupyter notebooks stored in my database. At this point, users can download notebooks and run them on their compuers.
I would like to add functionality, where user could run notebook online. I was thinking of two solutions:
first one is to use some free to use online service, like google colab, but I haven't found any with api where I could send file from my database (maybe you know about some?),
second is to run jupyter hub on my server. I saw how to run jupyter hub remotely, but I don't know how to grant users the access, so they can run notebooks simultaneously, and they don't have access to server itself thorugh it, and do all of this in django.
Do you have any hints that could help me get this functionality?
JupyterHub is a good approach if you trust your users. However, if you want to run untrusted code (like Google Colab does), you need sandboxing. In that case, you can use a Docker image to run notebooks. For example, mikebirdgeneau/jupyterlab. And there is a docker-compose file example: https://github.com/mikebirdgeneau/jupyterlab-docker/blob/master/docker-compose.yml
I am trying to build a service that would allow users using notebook to set automation parameters in a cell like the starting time as to when the notebook should start executing. The service would then take this input time and execute the notebook at the desired time and store the executed notebook to S3. I have looked into papermill but I believe there is no way to add automation parameters like start execution time using that. Is there any ways to achieve this? Or is there a way papermill can achieve this?
Papermill handles just parameterizing and executing the notebooks, not scheduling. For that, you need to use another tool. You can build something yourself on top of Apache Airflow which seems to be the most widespread scheduler for such case. It has a native support for Papermill (see here). Or you can use a ready tool like Paperboy.
To read in-depth about scheduling notebooks, take a look at the article by Netflix.
Take a look at the code here and here for a wrapper that will schedule notebook execution
The shell scripts above create a VM, runs the notebook, saves the output and destroy the instance.
In Google Cloud AI Platform Notebooks we provide a scheduling service which is in Beta now.
I currently run a personal website using Wordpress (but hosted on siteground) that is a set of engineering study guides. I would like to move towards making these study guides interactive (i.e. refreshing graphics based on sliders, doing basic calculations to indicate if a design works or not, so I need numpy). A friend recommended that I utilize Jupyter notebooks for this purpose, as you can both render LaTeX (which I'm currently using Mathjax with Wordpress to do), as well as have the types of interactive graphics I want using either Bokeh or Plotly.
While I've seen tutorials for sharing notebooks on specific servers, what I'm after is being able for others to run my notebook in their browser (read-only), where the notebook is privately hosted.
I'm still not sure if Jupyter is the correct avenue to accomplish what I want, so I'm open to other suggestions (someone also recommended using Julia, but I've seen fewer examples of this).
I agree with your friend that Jupyter Notebooks is an excellent approach. And while it's by no means the only method to accomplish what you're after, I'm hard-pressed to come up with an immediate alternative that doesn't require significant work to set up.
I can think of three primary methods of using Jupyter Notebooks which suit your needs:
1. Azure Notebooks
Microsoft has a new service called Azure Notebooks, which is (currently) totally free.
Azure Notebooks boasts the complete functionality of Jupyter Notebooks, and in addition to Python, users can also program cells in R and F#. As for typical usage of the service, here's a snippet from their FAQ:
Jupyter (formerly IPython), is a multi-lingual REPL on steroids. This is a free service that provides Jupyter notebooks along with supporting packages for R, Python and F# as a service. This means you can just login and get going since no installation/setup is necessary. Typical usage includes schools/instruction, giving webinars, learning languages, sharing ideas, etc. The service is provided by the Python team # Microsoft, which is part of the Data Group.
2. nbviewer
The top banner of the main Jupyter site contains a link link to an application called nbviewer.
Evidently, you can create your markdown / Jupyter syntax as a discrete page somewhere else, feed the URL to your page into nbviewer, and it'll render it for you right there in the results. If I were going to use this, I would either;
Create a discrete WordPress page for my Jupyter syntax, then feed that into nbviewer; or, more likely
Use GitHub to host my Jupyter Notebook pages (mainly for posterity and version control, over the Gist option), and use the raw text link as the source to feed into nbviewer.
3. Hosting Your Own Solution
If you're technically savy enough, I'd recommend this approach over nbviewer.
When you launch Jupyter Notebooks on your own machine, you access it through your browser using the default URL of http://localhost:8888. That means there must exist some mechanism to expose that port to external users, and allow them to have access to your Notebook, using the exact same interface. Two methods of doing so:
Using Jupyter Notebooks public server
Remotely accessing your normal Jupyter Notebook
Hope that helps! I'm curious to know if any of these options works out for you.
The Iodide Project (and subsequently, Pyodide) are two projects that aim to allow this. They're still in development, but might be worth looking into.
You can try to use Mercury framework. It allows you to transform notebooks into web applications (with interactive widgets). You need to add YAML header to the beginning of the notebook. Based on YAML the widgets will be generated. Your users can change widgets values and click Run button to execute the notebook with new inputs. You can decide whether to show or hide code for your users. You can serve multiple notebooks with Mercury on single server. It is based on Django so can be easily deployed on any server/cloud.
The example notebook:
The generated application for the above notebook:
The screenshot of app/notebooks gallery in the Mercury: