I set up a new Ubuntu instance on AWS EC2. I SSH to the instance using a private key pair. I installed python, jupyter, pyspark and all the necessary modules. I then start a Jupyter notebook using tmux.
My main aim is simply to run pyspark on an AWS instance (using Jupyter). Unfortunately, I keep running into problems with stability of the Jupyter notebook/connection to the instance. After running the Jupyter notebook for some time (sometimes 5 minutes, other times 2 hours+), it ends up "disconnecting". The kernel in the Jupyter disconnects and then does not process any further calls. At that point, I cannot SSH into the instance (just hangs -> blank screen).
I tried running the same setup on GCP but run into the same symptoms.
Is there something basic that I am missing?
Why would I not be able to SSH into the instance?
Is it possible that the Ubuntu server is crashing?
Related
I am running a Jupyter Notebook on my laptop. Is it possible to run one/two cells of the script on a remote server that I have access to?
The remote server is more powerful, however I have been allocated a limited amount of storage on the server so the bulk of work has to happen on my device.
I have used OS.system to run a python script on the server from Jupyter on my laptop, but it seems inefficient.
Both devices are running Ubuntu.
I want to run local code using local data on a remote server and get back execution results back to my Jupyter notebook cells.
Not usual scheme "run Jupyter notebook remotely, connect to remote notebook via ssh tunneling" but more sophisticated via custom remote kernel which I may choose from the kernel list, and run local code on remote server seamlessly.
Some packages (like this -- https://pypi.org/project/remote-kernel) mention that it is possible, but look dated and come with limited usage instructions.
Anyone knows how to implement this? If so, be as more detailed as possible, thanks!
Suppose I have a Google Colab Notebook in an address like below:
https://colab.research.google.com/drive/XYZ
I want to keep it running for 12 hours, however, then again I want to turn my computer off. As a solution, I can connect to our Lab's server via ssh. The server is running all the time. I would like to know if it's possible that I load and run the notebook there?
I found a solution to connect to a Google Colab Session via ssh (colab_ssh package), but it again needs a running Colab Session.
I also tried to browse the link with lynx, but it needs login and this isn't supported by this browser.
Yes, it is possible. You would first need to download your colab notebook as an .ipynb file, then copy it to your server. Then, you can follow one of the guides on how to connect to a remotely running jupyter notebook session, like this one. All you need is the jupyter notebook software on your server, and an ssh client on your local computer.
Edit: I forgot to mention this: To keep your session alive even after closing the ssh connection, you can use tools like screen. The link provides more detailed explanation, but the general idea is that after connecting to your server, first you need to create a session like this:
screen -S <session_name>
which will create a new session and attach you to it (which is the term used when you are inside a session). Then, you can fire up your jupyter notebook here, and it will keep running even after closing the ssh connection. (You just have to make sure you don't kill the screen session using Ctrl+a followed by k)
Now, you have an indefinitely running jupyter notebook session on your server. You can connect to it via
ssh -N -f -L localhost:YYYY:localhost:XXXX remoteuser#remotehost
as mentioned in the first linked guide, use the browser to run a code cell on your jupyter notebook, and then turn off your laptop without worrying about interrupting your notebook session.
I have been using PaperMill for executing my python notebook periodically. To execute compute intensive notebook, I need to connect to remote kernel running in my EMR cluster.
In case of Jupyter notebook I can do that by starting jupyter server with jupyter notebook --gateway-url=http://my-gateway-server:8888 and I am able to execute my code on remote kernel. But how do I let my local python code(through PaperMill) to use remote kernel? What changes do what to make in Kernel Manager to connect to remote kernel?
One related SO answer I could find is here. This suggests to do port forwarding to remote server and initialize KernelManager with the connection file from the server. I am not able to do this as blockingkernelmanager is no longer in Ipython.zmp and I would also prefer HTTP connection like how jupyter does.
Hacky approach - Set up a shell script to do the following :
Create a python environment on your EMR masternode using the hadoop user
Install sparkmagic in your environment and configure all kernels as described in the README.md file for sparkmagic
Copy your notebook to master node/use it directly from s3 location
Run with papermill :
papermill s3://path/to/notebook/input.ipynb s3://path/to/notebook/output.ipynb -p param=1
Step 1 and 2 are one time requirements if your cluster master node is the same every time.
A slightly better approach :
Set up a remote kernel in your Jupyter itself : REMOTE KERNEL
Execute with papermill as a normal notebook by selecting this remote kernel
I am using both approaches for different use cases and they seem to work fine for now.
I'm new to cloud computing in general, and I've started a free trial with Amazon's Web Services, in hopes of using their EC2 servers to run some code for Kaggle competitions. I'm currently working on running a test Python script for doing some image processing and testing a linear classifier (I don't suspect these details are relevant to my problem, but wanted to provide some context).
Here are the steps I've gone through to run my script on an EC2 instance:
Log in to AWS, and start EC2 instance where I've installed relevant Python modules for my tasks (e.g. Anaconda distribution). As a sidenote, all my data and the script I want to run are in the same directory on this server instance.
SSH to my EC2 instance from my laptop, and cd to the directory with my script.
Run screen to run program in background.
Run script via python program.py and detach from screen session (ctrl + A, D)
Keep EC2 instance running, but exit from SSH session connecting my laptop to the server.
I've followed these steps a number of times, which result in either (a) "Broken Pipe" errors, or (b) in an error where the connection appears to "hang". In the case of (b), I've attempted to disconnect from the SSH session and reconnect to the server, however I am unable to do so due to an error stating "connection has been reset by peer".
I'm not sure if I need to configure something differently on the EC2 instance, or if I need to specify different options when connecting to the server via SSH. Any help here would be appreciated. Thanks for reading.
EDIT: I've been successful in running some example scripts using scikit-learn by setting up an iPython notebook, launching it with nohup, and running the code in a notebook cell. However, when trying to do the same with my Kaggle competition code, the same "hanging" issue happens, and the connection appears to be dropped, causing the code to stop running. The image dataset I'm running the code on in the second case is quite a bit larger than the dataset processed by the example code in the first case. Not sure if dataset size along is causing the issue, or how to solve this.