We Keep Coding

Connect local Jupyter Hub to Azure Databricks Spark Cluster

I have one Azure Databricks cluster. Although it provides notebook, my team is more familiar with using Jupyter Lab which they can upload offline-csv, install python packages.
I want to setup a Jupyter lab which can connect to the Spark Cluster.
Although databricks allow using remote kernel to access it - https://databricks.com/blog/2019/12/03/jupyterlab-databricks-integration-bridge-local-and-remote-workflows.html, it can't read local files on Jupyter lab.
Is there any way to use spark cluster with a local jupyter lab like https://medium.com/ibm-data-ai/connect-to-remote-kerberized-hive-from-a-local-jupyter-notebook-to-run-sql-queries-83d5e548d82c?
Many thanks

If you prefix a magic command with a %%, it will take the rest of the cell as its argument, which means %%local is used to send data to Spark cluster from Local instance.
Install databrickslabs_jupyterlab locally:
(base)$ conda create -n dj python=3.8 # you might need to add "pywin32" if you are on Windows
(base)$ conda activate dj
(dj)$ pip install --upgrade databrickslabs-jupyterlab[cli]==2.2.1
(db-jlab)$ dj $PROFILE -k
Start JupyterLab:
(db-jlab)$ dj $PROFILE -l
Test the Spark access:
import socket
from databrickslabs_jupyterlab import is_remote
result = sc.range(10000).repartition(100).map(lambda x: x).sum()
print(socket.gethostname(), is_remote())
print(result)
For more details, you can refer to Install Jupyter Notebook on your computer and connect to Apache Spark on HDInsight, Kernels for Jupyter Notebook on Apache Spark clusters in Azure HDInsight and Sending data to Spark cluster from Local instance

Google Colab wont connect to local runtime after update

I recently updated jupter_over_ws to version 0.0.7 and now Google Colab refuses to connect my local runtime.
I have tried multiple iterations of launching the localhost in chrome, running with --no-browser etc and keep receiving a 'Forbidden' 403.
The last iteration I got to in an attempt to connect to the notebook was:
jupyter notebook --NotebookApp.allow_origin='https://colab.research.google.com' --NotebookApp.port_retries=0 --notebook-dir="" --no-browser --allow-root --NotebookApp.token='' --NotebookApp.disable_check_xsrf=True --port=8888
Any help much appreciated

Colab recently updated it's local runtime connection instructions.
In particular, you'll need to provide the URL printed in the console when starting the local runtime.
Clicking the "More Details" button in the local connection dialog will provide additional instructions.

After update, you need first upgrade jupyter_http_over_ws:
pip install --upgrade jupyter_http_over_ws
And second, everytime you start the server with this code:
jupyter notebook --NotebookApp.allow_origin='https://colab.research.google.com' --port=8888 --NotebookApp.port_retries=0
It will show a new token in Anaconda's prompt, you need to copy.
When you click in Connect to local runtime in Colab, a box will ask for the token, then paste there.

I solved this by removing the password from the jupyter .config file, then copy and pasting the token url as you have outlined in the answers above, and as outlined in the official colab documentation:
https://research.google.com/colaboratory/local-runtimes.html

If you use Anaconda, you can follow the instructions:
Open cmd.exe as Administrator
Run pip install --upgrade jupyter_http_over_ws>=0.0.7 && jupyter serverextension enable --py jupyter_http_over_ws
Run jupyter notebook --NotebookApp.allow_origin='https://colab.research.google.com' --port=8892 --NotebookApp.port_retries=0
Copy the URL from the command prompt printed console ex: http://localhost:8892/?token=fecaca2501d00f89971ab8fb4a5add3ff682ade7e63f7265
Paste the URL including token in Google Colab connection settings as Backend URL

Installing Anaconda on Server

I have a Unix server where I have Python3 installed. I ssh to the server from my mac.
I was wondering if it possible to install Anaconda and Jupyter (will come with Anaconda) on the server so that I can just pull up Jupyter on the server terminal and run codes on jupyter running on the server.
Is it possible? And if yes, could someone guide me to the right link?

in a terminal on your remote server:
#download anaconda (change version if you want)
wget https://repo.continuum.io/archive/Anaconda3-2018.12-Linux-x86_64.sh
# run the installer
bash Anaconda3-5.1.0-Linux-x86-64.sh
# so changes in your path take place in you current session:
source ~/.bashrc
#To run a remote notebook, replace XXXX with your choice of four numbers like 9191
jupyter notebook --no-browser --port=XXXX
#copy the url that you get as a result
Then in your local machine, open up a terminal and write:
#XXXX is the port you specified in the previous step, YYYY is a local port, for example 9999 to keep it simple
ssh -f [USER]#[SERVER] -L YYYY:localhost:XXXX -N
Then copy the url from the previous step, paste it in a browser, since you used the same port, you don't have to change anything on the url

you can download anaconda using:wget https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
and install using: bash Anaconda3-5.1.0-Linux-x86_64.sh
After that just source the path of Anaconda in .bashrc file, it should work.
To access jupyter notebook, you can use ssh and run notebook in your browser on your host. Steps are mentioned in this link

Yes you can install anaconda on your linux machine (Server) and manage the python environment. But if just need Jupyter hosted in a server, just install Jupyter only and start the service which will server Jupyter Notebook. Access Jupyter notebook using your browser on any other PC.
Make a google search that how to install anaconda on Linux machine (Centos/Ubuntu etc)
After installation run following command
conda info
and then configure the Jupyter and run.
Simple way (Install Jupyter on a server): Install, Run, and Connect to Jupyter Notebook on a Remote Server

Connect jupyter python by tunneled connection (via ssh)

I have a problem with connecting to jupyter that I run on remote computer. I have configured jupyter with ssl as in the following doc:
https://jupyter-notebook.readthedocs.io/en/latest/public_server.html#hashed-pw
Then I run jupyter on remote machine with command: jupyter notebook as in tutorial.
When its running I make a tunnel in another terminal by running a command:
ssh -N -f -L 8889:127.0.0.1:8881 user_name#hostname.com
Of course the jupyter is running on port 8881. When I open the browser on my local computer and try to open page:
localhost:8889
I got an error that connection was reset, and in command line the ssh returns the following error:
channel 2: open failed: connect failed: Connection refused
Can anyone help me? I was looking for an answer for whole day and still can't fix it.

The problem is, that in my case jupyter is running on localhost. At my server remote server I have no access to remote localhost, so the solution was quite easy - I run jupyter on specified ip = 0.0.0.0
jupyter notebook --no-browser --port=8881 --ip=0.0.0.0

Access Jupyter notebook running on Docker container

I created a docker image with python libraries and Jupyter.
I start the container with the option -p 8888:8888, to link ports between host and container.
When I launch a Jupyter kernel inside the container, it is running on localhost:8888 (and does not find a browser). I used the command jupyter notebook
But from my host, what is the IP address I have to use to work with Jupyter in host's browser ?
With the command ifconfig, I find eth0, docker, wlan0, lo ...
Thanks !

You need to run your notebook on 0.0.0.0: jupyter notebook -i 0.0.0.0. Running on localhost make it available only from inside the container.

Host machine: docker run -it -p 8888:8888 image:version
Inside the Container : jupyter notebook --ip 0.0.0.0 --no-browser --allow-root
Host machine access this url : localhost:8888/tree‌​
When you are logging in for the first time there will be a link displayed on the terminal to log on with a token.

The docker run command is mandatory to open a port for the container to allow the connection from a host browser, assigning the port to the docker container with -p, select your jupyter image from your docker images.
docker run -it -p 8888:8888 image:version
Inside the container launch the notebook assigning the port you opened:
jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root
Access the notebook through your desktops browser on http://localhost:8888
The notebook will prompt you for a token which was generated when you create the notebook.

To get the link to your Jupyter notebook server:
After your docker run command, a hyperlink should be automatically generated. It looks something like this: http://localhost:8888/?token=f3a8354eb82c92f5a12399fe1835bf8f31275f917928c8d2 :: /home/jovyan/work
If you want to get the link again later down the line, you can type docker exec -it <docker_container_name> jupyter notebook list.

The below is how I get it running on Windows 7 with docker toolbox.
If you are using docker toolbox, open up the Docker quickstart terminal, and note the IP here:
docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com
Once you run the docker commands from the tensorflow installation website:
docker pull tensorflow/tensorflow # Download latest image
docker run -it -p 8888:8888 tensorflow/tensorflow # Start a Jupyter notebook server
You will receive a message like this:
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://127.0.0.1:8888/?token=d6e80acaf08e09853dc72f6b0f022b8225f94f
In the host, replace 127.0.0.1 with 192.168.99.100 and use the rest of that URL

You can use the command jupyter notebook --allow-root --ip[of your container] or give access to all ip using option --ip0.0.0.0.

As an alternative to building your own Docker image, you can also use the ML Workspace image. The ML Workspace is an open-source web IDE that combines Jupyter, VS Code, a Desktop GUI, and many other tools & libraries into one convenient Docker image. Deploying a single workspace instance is as simple as:
docker run -p 8080:8080 mltooling/ml-workspace:latest
All tools are accessible from the same port and integrated into the Jupyter UI. You can find further documentation here.

In the container you can run the following to make it available on your local machine (using your docker machine's ip address).
jupyter notebook --ip 0.0.0.0 --allow-root
You may not need to provide the --allow-root flag depending on your container's setup.

docker run -i -t -p 8888:8888 continuumio/anaconda3 /bin/bash -c "/opt/conda/bin/conda install jupyter -y --quiet && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook --notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser --allow-root"
i had to add --allow-root to the command and now its running

The Makefile below encapsulates the previous answers and ensures that jupyter and docker agree on the port. And if I just click/copy the link provided by jupyter, that solves port mismatch problems.
To use, just make jupyter or make jupyter PORT=xxxx from the proper folder. Then click the link in the jupyter output.
Remote Containers
If your container is on a remote host (say an AWS EC2), then you'll also need to set up an ssh tunnel with the correct port. For example, on the local machine:
ssh -N -f -L localhost:8888:localhost:8888 username#remote-host
But at least that's only one place I can manually mismatch ports.
Makefile
# Set your default jupyter port here.
# Avoid 8888 if you run local notebooks on that!
PORT=8888
# Your app root in the container.
APP_DIR=/app
# The docker image to launch. *** SET TO YOUR image:version! ***
APP_TAG=image:version
jupyter: ##
## Launch jupyter notebook from our container, mapping two folders
## Local Container Notes
## -----------------------------------------------------
## ./data -> /data Put data here!
## ./notebooks -> /notebooks Find notebooks here!
## -----------------------------------------------------
## Arg: PORT - specify port [${PORT}]
docker run \
-p $(PORT):$(PORT) \
-v $(PWD)/notebooks/:$(APP_DIR)/notebooks/ \
-v $(PWD)/data:/data \
$(APP_TAG) \
jupyter notebook --ip 0.0.0.0 --port $(PORT) \
--no-browser --allow-root

Go in the Docker and check cat /etc/jupyter/jupyter_notebook_config.py :
You should see / add this line :
c.NotebookApp.allow_origin = 'https://colab.research.google.com'