Starting Spyder/Jupyter Kernel on Server headless - python

I want to connect my local spyder-ide to a remote spyder-kernel running on our ML-server. but as soon as i shut down the console, the kernel halts. is there a way to start the kernel in the background and keep it running ?
i can start a kernel with python -m spyder_kernel.console on the server and can connect to it. but this opens a ipython console on the server. as soon as I close it, the kernel shuts down. is there a possibilty to start this kernel like "nohup" or headless.
i can do it with jupyter kernel& , but then i'm missing the spyder functionalities like the variable explorer in the IDE.
thanks for your help !

python -m spyder_kernels.console& resolved the question !

Related

Jupyter notebook looses connection

I am fairly new to working with python, so I am sorry if this is a naive question.
I have set up a jupyter notebook that I start through the windows terminal. I run it with python 3.9.7 in an Anaconda virtual environment. I use microsoft edge as host browser because I had problems with google chrome blocking jupyter lab.
Since yesterday, I receive an error message after opening my notebook with - jupyter lab command (which has worked for me since some weeks now). The puzzling thing is, that I can open my notebook and work in it for some minutes usually, but then this message pops up:
Server Connection Error: A connection to the Jupyter server could not be established. JupyterLab will continue trying to reconnect. Check
your network connection or Jupyter server configuration.
Unfortunately, I have really no idea why this could be, therefore I have not tried much to fix this yet. I have a stable internet connection. Simply closing the notebook and reopening it worked for some times, but after a while of working in the notebook it looses the connection again.
Has anyone experienced similar problems?
Thank you for your help and ideas!
I had this issue with jupyter lab my error seems the same:
"A connection to the Jupyter server could not be established.
JupyterLab will continue trying to reconnect. Check your network
connection or Jupyter server configuration."
The problem for me was that I had started the program from the command line using the "jupyter-lab" command and had subsequently closed my terminal after the jupyter lab opened. Closing terminal shut down the required server. Left the terminal open and is working normally.

How to let the python script (abc.py) keep executing on AWS even after Connection lost or SSH connection is terminated?

I was using Jupyter notebook on AWS AMI and using the port forwarding on Windows using PuTTY. I got my connection terminated and all the work of 24 hours is lost now and I'm unable to get that. So I used a script instead of Notebook and the same thing happened. I used to think that the process would keep going on even if the Shell connection is lost. But next time I login, I don't see anything.
I used top, htop aux to find if my processes are still running but they don't show my process. Please help how can I stop this from happening.
I am using Windows on 10 with Putty on local and Ubuntu 18 on the AWS AMI.
You can try
nohup python3 myscript.py &
Try the tmux
apt install tmux
I often use it to keep my web runing on server(not professional but convenient)
You can use screen.
You can install it wit:
sudo apt install screen
You can find more information right here: https://linuxize.com/post/how-to-use-linux-screen/

Why IPython Console in Spyder4 updates constantly when connecting to remote kernel on server?

I followed the instructions on the official Spyder page and successfully connected to remote kernel via SSH. However I have a problem in my Ipython Console as it refreshes every 2-3 seconds adding In [1]: lines infinitely. I suppose it is refreshing of a state on server, but could someone explain what is this, why this happens and can I turn it off, so that the Console behaves the same as when I work on my local kernel(without infinite adding of empty lines)?
The solution could be found here in post of scott-8: https://github.com/spyder-ide/spyder/issues/10240#issuecomment-543913159
The copy of the answer:
Don't know about the issue above, but here is what solved my issue for anyone reading: instead of running python -m spyder_kernels.console and connecting to the kernel, quit the kernel after running it. Then restart the kernel with python -m spyder_kernels.console -f kernel-xxxxx.json, specifying the file that was just created in the runtime directory, and connect to it. This fixed my problem for some reason.

Detaching ipython notebook process without closing it?

I have an ipython noteboook where I am running a process that takes a very long time. I am using ipython %R magic in much of it, so I can't easily convert the notebook to a python script.
Is there a way I can open my notebook, run all, and then close my browser and disconnect from the terminal and still have the notebook running in the background that I can connect to later?
I see information on Stack Exchange about keeping the kernel alive, but I'm confused as to how this interfaces with the actual code running within a notebook.
I'm using tmux, to detach from running jupyter-notebook type ctrl-C then type tmux detach command : ctrl-B, followed by d
link for tmux setup : https://stackoverflow.com/a/42505000/7358899

How to close IPython Notebook properly?

How to close IPython Notebook properly?
Currently, I just close the browser tabs and then use Ctrl+C in the terminal.
Unfortunately, neither exit() nor ticking Kill kernel upon exit does help (they do kill the kernel they but don't exit the iPython).
There isn't currently a better way to do it than Ctrl+C in the terminal.
We're thinking about how to have an explicit shutdown, but there's some tension between the notebook as a single-user application, where the user is free to stop it, and as a multi-user server, where only an admin should be able to stop it. We haven't quite worked out how to handle the differences yet.
(For future readers, this is the situation with 0.12 released and 0.13 in development.)
Update December 2017
The IPython Notebook has become the Jupyter Notebook. A recent version has added a jupyter notebook stop shell command which will shut down a server running on that system. You can pass the port number at the command line if it's not the default port 8888.
You can also use nbmanager, a desktop application which can show running servers and shut them down.
Finally, we are working on adding:
A config option to automatically shut down the server if you don't use it for a specified time.
A button in the user interface to shut the server down. (We know it's a bit crazy that it has taken this long. Changing UI is controversial.)
I think accepted answer outdated and is not valid anymore.
You can terminate jupyter notebook from web interface on file menü item.
When you move Mouse cursor on "close and halt", you will see following explanation.
And when you click "close and halt", you will see following message on terminal screen.
If you run jupyter in the background like me:
jupyter notebook &> /dev/null &
Then to exit jupyter completely, instead of Ctl-C, make an alias command:
echo 'alias quitjupyter="kill $(pgrep jupyter)"' >> ~/.bashrc
Restart your terminal. Kill all jupyter instances:
quitjupyter
Note: use double quotes inside of single quotes as shown above. The other way around will evaluate the expression before writing it to your .bashrc (you want to write the command itself not 'kill 1430' or whatever process number may be associated with a current jupyter instance). Of course you can use any alias you wish. I actually use 'qjup':
echo 'alias qjup="kill $(pgrep jupyter)"' >> ~/.bashrc
Restart your terminal. Kill all jupyter instances:
qjup
First step is to save all open notebooks. And then think about shutting down your running Jupyter Notebook. You can use this simple command:
$ jupyter notebook stop
Shutting down server on port 8888 ...
Which also takes the port number as argument and you can shut down the jupyter notebook gracefully.
For eg:
jupyter notebook stop 8889
Shutting down server on port 8889 ...
Additionally to know your current jupyter instance running, check below command:
shell> jupyter notebook list
Currently running servers:
http://localhost:8888/?token=ef12021898c435f865ec706de98632 :: /Users/username/jupyter-notebooks [/code]
These commands worked for me:
jupyter notebook list # shows the running notebooks and their port-numbers
# (for instance: 8080)
lsof -n -i4TCP:[port-number] # shows PID.
kill -9 [PID] # kill the process.
This answer was adapted from here.
Try killing the pythonw process from the Task Manager (if Windows) if nothing else works.
Linux (Ubuntu 14.04)
As mentioned, try to kill ipython notebook processes properly by first going to the "running" tab in your ipynb/jupyter browser session, and then check open terminals on your console and shut down with ctrl-c. The latter should be avoided if possible.
If you run an ipython notebook list and continue to see running ipython servers at different ports, make note of which ports the existing notebooks are being served to. Then shut down your TCP ports:
fuser -k 'port#'/tcp
I'm not sure if there are other risks involved with doing this. If so, let me know.
Actually, I believe there's a cleaner way than killing the process(es) using kill or task manager.
In the Jupyter Notebook Dashboard (the browser interface you see when you first launch 'jupyter notebook'), browse to the location of notebook files you have closed in the browser, but whose kernels may still be running.
iPython Notebook files appear with a book icon, shown in green if it has a running kernel, or gray if the kernel is not running.
Just select the tick box next to the running file, then click on the Shutdown button that appears above it.
This will properly shut down the kernel associated with that specific notebook.
Environment
My OS is Ubuntu 16.04 and jupyter is 4.3.0.
Method
First, i logged out jupyter at its homepage on browser(the logout button is at top-right)
Second, type in Ctrl + C in your terminal and it shows:
[I 15:59:48.407 NotebookApp]interrupted Serving notebooks from local
directory: /home/Username 0 active kernels
The Jupyter Notebook is running at:
http://localhost:8888/?token=a572c743dfb73eee28538f9a181bf4d9ad412b19fbb96c82
Shutdown this notebook server (y/[n])?
Last step, type in y within 5 sec, and if it shows:
[C 15:59:50.407 NotebookApp] Shutdown confirmed
[I 15:59:50.408
NotebookApp] Shutting down kernels
Congrats! You close your jupyter successfully.
The best way now is to use the "Quit" button that is just to the left of the "Logout" button. I have to admit that I do not understand the utility of the Logout button. However, I am glad that they have added the exceedingly useful Quit button.
Option 1
Open a different console and run
jupyter notebook stop [PORT]
The default [PORT] is 8888, so, assuming that Jupyter Notebooks is running on port 8888, just run
jupyter notebook stop
If it is on port 9000, then
jupyter notebook stop 9000
Option 2 (Source)
Check runtime folder location
jupyter --paths
Remove all files in the runtime folder
rm -r [RUNTIME FOLDER PATH]/*
Use top to find any Jupyter Notebook running processes left and if so kill their PID.
top | grep jupyter &
kill [PID]
One can boilt it down to
TARGET_PORT=8888
kill -9 $(lsof -n -i4TCP:$TARGET_PORT | cut -f 2 -d " ")
Option 3
As a last resort, what Hary shares here can also be helpful
sudo pkill -1 -f python
Note: If one wants to launch one's Notebook on a specific IP/Port
jupyter notebook --ip=[ADD_IP] --port=[ADD_PORT] --allow-root &
If you are on mac and using Jupyter lab you can do the following in terminal
jupyter list
Above will give you all the running servers. Once you get to know the post just use jupyter lab stop. For example if the server was running on 8888 just execute
jupyter lab stop 8888
In the browser session you can also go to Kernel and then click Restart and Clear Output.
I am copy pasting from the Jupyter/IPython Notebook Quick Start
Guide Documentation, released on Feb 13, 2018.
http://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html
1.3.3 Close a notebook: kernel shut down
When a notebook is opened, its “computational engine” (called the kernel) is automatically started. Closing the
notebook browser tab, will not shut down the kernel, instead the kernel will keep running until is explicitly shut down.
To shut down a kernel, go to the associated notebook and click on menu File -> Close and Halt. Alternatively, the
Notebook Dashboard has a tab named Running that shows all the running notebooks (i.e. kernels) and allows shutting
them down (by clicking on a Shutdown button).
Summary: First close and halt the notebooks running.
1.3.2 Shut down the Jupyter Notebook App
Closing the browser (or the tab) will not close the Jupyter Notebook App. To completely shut it down you need to
close the associated terminal.
In more detail, the Jupyter Notebook App is a server that appears in your browser at a default address
(http://localhost:8888). Closing the browser will not shut down the server. You can reopen the previous address
and the Jupyter Notebook App will be redisplayed.
You can run many copies of the Jupyter Notebook App and they will show up at a similar address (only the number
after “:”, which is the port, will increment for each new copy). Since with a single Jupyter Notebook App you can
already open many notebooks, we do not recommend running multiple copies of Jupyter
Notebook App.
Summary: Second, quit the terminal from which you fired Jupyter.
Step 1 - On shell just do control+z (control+c)
Step 2 _ close the web browser
To kill all jupyter-notebook and jupyter-lab, open terminal and just kill all jupyter that running in background by giving
pkill jupyter

Categories