I want to run local code using local data on a remote server and get back execution results back to my Jupyter notebook cells.
Not usual scheme "run Jupyter notebook remotely, connect to remote notebook via ssh tunneling" but more sophisticated via custom remote kernel which I may choose from the kernel list, and run local code on remote server seamlessly.
Some packages (like this -- https://pypi.org/project/remote-kernel) mention that it is possible, but look dated and come with limited usage instructions.
Anyone knows how to implement this? If so, be as more detailed as possible, thanks!
Related
Suppose I have a Google Colab Notebook in an address like below:
https://colab.research.google.com/drive/XYZ
I want to keep it running for 12 hours, however, then again I want to turn my computer off. As a solution, I can connect to our Lab's server via ssh. The server is running all the time. I would like to know if it's possible that I load and run the notebook there?
I found a solution to connect to a Google Colab Session via ssh (colab_ssh package), but it again needs a running Colab Session.
I also tried to browse the link with lynx, but it needs login and this isn't supported by this browser.
Yes, it is possible. You would first need to download your colab notebook as an .ipynb file, then copy it to your server. Then, you can follow one of the guides on how to connect to a remotely running jupyter notebook session, like this one. All you need is the jupyter notebook software on your server, and an ssh client on your local computer.
Edit: I forgot to mention this: To keep your session alive even after closing the ssh connection, you can use tools like screen. The link provides more detailed explanation, but the general idea is that after connecting to your server, first you need to create a session like this:
screen -S <session_name>
which will create a new session and attach you to it (which is the term used when you are inside a session). Then, you can fire up your jupyter notebook here, and it will keep running even after closing the ssh connection. (You just have to make sure you don't kill the screen session using Ctrl+a followed by k)
Now, you have an indefinitely running jupyter notebook session on your server. You can connect to it via
ssh -N -f -L localhost:YYYY:localhost:XXXX remoteuser#remotehost
as mentioned in the first linked guide, use the browser to run a code cell on your jupyter notebook, and then turn off your laptop without worrying about interrupting your notebook session.
I have been using PaperMill for executing my python notebook periodically. To execute compute intensive notebook, I need to connect to remote kernel running in my EMR cluster.
In case of Jupyter notebook I can do that by starting jupyter server with jupyter notebook --gateway-url=http://my-gateway-server:8888 and I am able to execute my code on remote kernel. But how do I let my local python code(through PaperMill) to use remote kernel? What changes do what to make in Kernel Manager to connect to remote kernel?
One related SO answer I could find is here. This suggests to do port forwarding to remote server and initialize KernelManager with the connection file from the server. I am not able to do this as blockingkernelmanager is no longer in Ipython.zmp and I would also prefer HTTP connection like how jupyter does.
Hacky approach - Set up a shell script to do the following :
Create a python environment on your EMR masternode using the hadoop user
Install sparkmagic in your environment and configure all kernels as described in the README.md file for sparkmagic
Copy your notebook to master node/use it directly from s3 location
Run with papermill :
papermill s3://path/to/notebook/input.ipynb s3://path/to/notebook/output.ipynb -p param=1
Step 1 and 2 are one time requirements if your cluster master node is the same every time.
A slightly better approach :
Set up a remote kernel in your Jupyter itself : REMOTE KERNEL
Execute with papermill as a normal notebook by selecting this remote kernel
I am using both approaches for different use cases and they seem to work fine for now.
I'm currently using GCP to run the Jupyter notebooks on the notebook server provided by Google. Every time I open the notebook server in commandline, it shuts down when there is a network interruption or power outage on my end. I'm very naive on GCP too.
Is there any way that I could run the Ipython notebooks on the server and later collect the results without having to bother about anything else?
Thanks in advance!
Have you tried using GCP's AI Platform Notebooks?
https://cloud.google.com/ai-platform-notebooks/
You can open these notebooks directly in your browser unlike the older Datalab notebooks (no need to SSH). That should solve your network interruption issues
I work in an environment such that I have python installed on my laptop (windows 10). I also have access to a couple of Linux servers. I work with some very large datasets, some which would be too large for my machine's memory to handle.
My question, which I so far haven't found an answer to, is if it's possible to connect to the linux server and run the code from my windows PC. I want to take advantage of the processing and memory on the linux server, but don't want to log on to server and do all of my development there and execute the scripts on the linux environment.
As an aside, another person in my department has been asking if it's possible to do this from Jupyter Notebook or an IDE like Spyder?
For jupyter you could just run the Jupyter notebook on the server and connect to the server IP and edit the notebook locally.
Of course you'd have to save your local notebooks and import them on the server but that should be pretty straight forward.
You can see how to set up a public Jupyter instance on a server here:
https://jupyter-notebook.readthedocs.io/en/stable/public_server.html
I would really appreciate some help here, basically I'm learning to use tensorflow, I've decided that the easiest way to go about this would be to install ubuntu on either VMware and/or Virtualbox and then access the ipython notebook (came with anaconda) through the browser on the host computer.
I have successfully installed both vmware and virtualbox, I downloaded a ubuntu image and also successfully installed anaconda on both, I get it to work without a problem on both VMs and even installed tensorflow.
Some research online on how to expose the ipython to the host machine suggested port forwarding or ssh tunneling, none of these have worked (very likely I'm doing it wrong). Can someone please help? think of me as a newbie.
Generally you must edit the jupyter configuration file to allow network access to the notebook server. See this link: http://jupyter-notebook.readthedocs.org/en/latest/public_server.html for details. (Even if it is not a "public" server, you still intend to access the notebook server living in the VM from the host machine via a network connection...)
Here is a quotation from the linked documentation that indicates by default, you can only access the notebook via the localhost.
By default, a notebook server runs locally at 127.0.0.1:8888 and is accessible only from localhost. You may access the notebook server from the browser using http://127.0.0.1:8888.