I am trying to get a Jupyter notebook up and running on Arch. I have tried installying the jupyter package. I also tried following the Arch wiki and installed jupyter-notebook and jupyter-nbconvert and python-ipywidgets. Lastly, I tried using the pip instructions. All three of which fail and give:
➜ ~ jupyter notebook
Traceback (most recent call last):
File "/usr/sbin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/usr/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/usr/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/notebook/notebookapp.py", line 1368, in initialize
self.init_webapp()
File "/usr/lib/python3.6/site-packages/notebook/notebookapp.py", line 1188, in init_webapp
self.http_server.listen(port, self.ip)
File "/usr/lib/python3.6/site-packages/tornado/tcpserver.py", line 142, in listen
sockets = bind_sockets(port, address=address)
File "/usr/lib/python3.6/site-packages/tornado/netutil.py", line 197, in bind_sockets
sock.bind(sockaddr)
OSError: [Errno 22] Invalid argument
It seems that the other reports related to this involve a different error with sock.bind. I am not sure if the issue is related or not. Any guidance on the matter would be appreciated.
When using socket.bind() usually you can pass it either a hostname or an ip. In particular as we oly listen on the local loopback address we can pass either localhost or 127.0.0.1 (for ip v4) or ::1 for ip v6.
In theory both would be identical, but in practice there are number of systems where one (or the other) is problematic. It can be firewall, and antivirus seeing this binding as suspicious, or strange network configurations. While you probably still should investigate why socket.bind() refuses localhost (in your case), you can configure the jupyter notebook server to bind directly to 127.0.0.1 by using either : jupyter notebook --ip=127.0.0.1, or change the configuration of jupyter notebook server with the equivalent long form option c.NotebookApp.ip='127.0.0.1'.
Also if his is widespread on Arch (and install via the arch repo) I would suggest contacting the arch package maintainer to have a custom patch that switch the default value to 127.0.0.1.
Related
I recently installed TensorFlow with GPU support using docker:
docker pull tensorflow/tensorflow:latest-gpu-jupyter
But sometimes when I start a jupyter notebook server using the command:
docker run --gpus all -it -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'
I see the following exception on the terminal:
[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 272, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 234, in wrapper
yielded = ctx_run(next, result)
File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 162, in _fake_ctx_run
return f(*args, **kw)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 580, in complete_request
matches = yield gen.maybe_future(self.do_complete(code, cursor_pos))
File "/usr/local/lib/python3.6/dist-packages/ipykernel/ipkernel.py", line 348, in do_complete
return self._experimental_do_complete(code, cursor_pos)
File "/usr/local/lib/python3.6/dist-packages/ipykernel/ipkernel.py", line 373, in _experimental_do_complete
completions = list(_rectify_completions(code, raw_completions))
File "/usr/local/lib/python3.6/dist-packages/IPython/core/completer.py", line 484, in rectify_completions
completions = list(completions)
File "/usr/local/lib/python3.6/dist-packages/IPython/core/completer.py", line 1818, in completions
for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
File "/usr/local/lib/python3.6/dist-packages/IPython/core/completer.py", line 1862, in _completions
full_text=full_text, cursor_line=cursor_line, cursor_pos=cursor_column)
File "/usr/local/lib/python3.6/dist-packages/IPython/core/completer.py", line 2030, in _complete
cursor_pos, cursor_line, full_text)
File "/usr/local/lib/python3.6/dist-packages/IPython/core/completer.py", line 1374, in _jedi_matches
text[:offset], namespaces, column=cursor_column, line=cursor_line + 1)
File "/usr/local/lib/python3.6/dist-packages/jedi/api/__init__.py", line 726, in __init__
project=Project(Path.cwd()), **kwds)
TypeError: __init__() got an unexpected keyword argument 'column'
After that, I have to restart the server or reconnect from google colab.
Any ideas where the error might come from and how to fix it?
This seems to be an incompatibility between jedi and ipython, see this issue.
The fix would be to pin jedi to 0.17.2, so either run:
pip install jedi==0.17.2
Or if you are using poetry add this to your pyproject.toml:
jedi = "<=0.17.2"
But since you are using a docker image that image will need to be updated. It seems to be gpu-jupyter.Dockerfile.
I would raise an issue on that project and see if they can pin jedi like they did for nbformat, or you could just fork it. They should probably upgrade python as well, 3.6 is getting a bit long in the tooth.
I will add more details to #daphtdazz's answer. I had to do the following steps to solve this issue:
1. Download TensorFlow from github:
git clone https://github.com/tensorflow/tensorflow.git
2. Edit the file gpu-jupyter.Dockerfile to add jedi==0.17.2 at the end of line 104:
vim tensorflow/tensorflow/tools/dockerfiles/dockerfiles/gpu-jupyter.Dockerfile
3. Placed myself inside the dockerfiles folder :
cd tensorflow/tensorflow/tools/dockerfiles/
4. Build the image:
docker build -f ./dockerfiles/gpu-jupyter.Dockerfile -t tf .
I have 2 servers. I am attempting to setup one for continuous integration for my main website server
Web server 1(cloud-hosting):
Python3.6
Django3.1
Ubuntu16.04
Webserver 2(VPS):
Python3.7
Django3.1
Ubuntu16.04
Jenkins
--ShiningPanda(plugin)
Im new to web development, so if it seems odd as far as my web server types, that is why. I have been following along in the book Test Driven Development with Python. My issue is that when running python manage.py test [app] My [app] inherits from the class StaticLiveSever to generate a testing environment. On webserver 1, this works fine. On webserver 2, i get an error that the request address cannot be assigned. I use jenkins to build the environment, but the error i get is OSerror[99]:cannot assign requested address. I dont understand why this is happening when i run the same commands in Web Sever 1. It runs fine. Although again, the commands are run by jenkins and jenkins is configured to run python3.7
Full Traceback(Main Issue)
Traceback (most recent call last):
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/test/testcases.py", line 1449, in setUpClass
raise cls.server_thread.error
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/test/testcases.py", line 1374, in run
self.httpd = self._create_server()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/test/testcases.py", line 1389, in _create_server
return ThreadedWSGIServer((self.host, self.port), QuietWSGIRequestHandler, allow_reuse_address=False)
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/core/servers/basehttp.py", line 67, in __init__
super().__init__(*args, **kwargs)
File "/usr/lib/python3.7/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib/python3.7/wsgiref/simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "/usr/lib/python3.7/http/server.py", line 137, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 99] Cannot assign requested address
After hardcoded host
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/Superlists/functional_tests/base.py", line 47, in setUp
self.browser = webdriver.Firefox()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 164, in __init__
self.service.start()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 100, in start
self.assert_process_still_running()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 113, in assert_process_still_running
% (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: 69
geckodriver.log w/ hardcoded host ip in LiveTestServer
eckodriver: error: Address not available (os error 99)
geckodriver 0.27.0 (7b8c4f32cdde 2020-07-28 18:16 +0000)
WebDriver implementation for Firefox
USAGE:
geckodriver [FLAGS] [OPTIONS]
[...]
Hopefully the tracebacks above are not too confusing. Ultmiately what i did notice was the when im in webserver 2, i access the Django testcases.py module that has LiverServerThread and hardcode host=0.0.0.0 instead of host=localhost (1st traceback). The connection is then made although then the problem lies with geckodriver and the same thing (2nd traceback). I need to hard code ip 0.0.0.0 to be able to establish a connection, but then geckodriver is just listening, which i am assuming, at a completely different location (no error.log shown here).
So 1st id like to at least be able to make a connection to run the LiveServerThread class properly. Then try and resolve the issue with geckodriver. I also was not sure if the type of servers im running on poses as the problem.
I'm trying to create a data science environment in Google Cloud Platform, with Python, Miniconda and Jupyter. But I'm encountering an error when I try to access jupyter with an external IP
I followed these steps:
1 - I've accessed the Resource Management page (https://console.cloud.google.com/cloud-resource-manager). Create a new project (test_01)
2 - I opened the Dashboard (https://console.cloud.google.com/home/dashboard). I selected the project I just created (test_01)
3 - In the top left menu - Select "Compute Engine" and click on "VM instances. Click on the "Create" button
4 - I filled in: Name the instance (the-who), Select the region (in my case - northamerica-northeast1-a), Select the memory, storage and CPU, Select the boot disk, Remote access SSH, Make sure you can access the VM from the internet by allowing Http and Https traffic and Click on "Create"
5 - I opened Google's Cloud Shell. I pressed the button >_
6 - With the Ubuntu terminal open, I typed in to create a password for root: sudo passwd
7 - Enter the command to enter SSH: gcloud compute ssh the-who
8 - There is always a question about the zone: Did you mean zone [us-east1-c] for instance: [the-who] (Y/n)? - Since mine zone is different (northamerica-northeast1-a) I press n
9 - I begin to install packages (bzip2 git libxml2-dev): "sudo apt-get update" and "sudo apt-get install bzip2 git libxml2-dev"
10 - Now install the miniconda, scikit-learn, pandas, jupyter and ipython: "wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh", "bash Miniconda3-latest-Linux-x86_64.sh", "rm Miniconda3-latest-Linux-x86_64.sh", "source .bashrc" and "conda install scikit-learn pandas jupyter ipython"
11 - I check installation: "python --version" and "which python"
12 - Now I want to set up my VM to allow Web access to my Jupyter notebooks. I returned to the instance panel and in the top left menu I selected "VPC Network > Firewall rules". I clicked on the "CREATE FIREWALL RULE" link and filled in with the following values
13 - Name: jupyter-rule * Source IP ranges: 0.0.0.0/0 * Specified protocols and ports: tcp:8888 * Target tags: the-who * and leave all the other variables to their default values. As a result, this firewall rule allows all incoming traffic (of all IPs) to reach port 8888.
14 - I returned to the VM page (upper left menu> Compute Engine> VM instances) and clicked on the VM name. I wrote down the VM's external IP address
15 - Back to the terminal I configure the jupyter. I type "jupyter notebook --generate-config" to generate the configuration file and to generate a password "jupyter notebook password"
16 - Now edit the jupyter configuration file: vim .jupyter/jupyter_notebook_config.py
17 - In the first line write and then save and close the file: c.NotebookApp.ip = '*'
18 - The above command will allow the notebook to be available for all IP addresses on your VM. And then I started the Jupyter Notebook with the command line: jupyter-notebook --no-browser --port=8888
But I found this error message:
(base) reichaves#the-who:~$ jupyter-notebook --no-browser --port=8888
[I 11:51:06.234 NotebookApp] Writing notebook server cookie secret to /run/user/1001/jupyter/notebook_cookie_secret
Traceback (most recent call last):
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'allow_remote_access'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 864, in _default_allow_remote
addr = ipaddress.ip_address(self.ip)
File "/home/reichaves/miniconda3/lib/python3.7/ipaddress.py", line 54, in ip_address
address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/reichaves/miniconda3/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "</home/reichaves/miniconda3/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1628, in initialize
self.init_webapp()
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1378, in init_webapp
self.jinja_environment_options,
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 159, in __init__
default_url, settings_overrides, jinja_env_options)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 252, in init_settings
allow_remote_access=jupyter_app.allow_remote_access,
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 867, in _default_allow_remote
for info in socket.getaddrinfo(self.ip, self.port, 0, socket.SOCK_STREAM):
File "/home/reichaves/miniconda3/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
Please, does anyone know if any setup details were missing?
Recent versions of Jupyter need to be told which IP addresses to listen to. You need in the config file:
c.NotebookApp.ip = '0.0.0.0'
Even with this change, and a password, I prefer to put Jupyter in a Singularity container, since people with web access to it can run any python code on the VM. For a recipe to do this, see:
https://bitbucket.org/account/user/stfcsciml/projects/JSV
I am running jupyter notebooks through a docker container. I have files, notebooks, etc within the container. I decide in class one day to attempt and install the jupyterthemes package because who doesn't like more colors. I opened a new ipynb and followed instructions per this site: https://github.com/dunovank/jupyter-themes
But it was basically just this:
!pip install jupyterthemes
!jt -t chesterish
The theme does not immediately appear and the directions suggest restarting the notebook or refreshing the browser. This is where the problems start, after trying to refresh or close and restart the notebook, it no longer works and just displays a large "500 : Internal Server Error" on the page. After trying to restart the home page of my notebook (this is locally hosted through docker and run on chrome btw), the jupyter window in chrome displays nothing at all.
Here I go back to terminal and docker and shut down the container. Then I try to restart the same container hoping it will work now. I try to start it as I usually would docker start -ai container_name but it is not successful. It displays these errors everytime
Executing the command: jupyter notebook
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py",
line 528, in get
value = obj._trait_values[self.name]
KeyError: 'allow_remote_access'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-
packages/notebook/notebookapp.py", line 869, in _default_allow_remote
addr = ipaddress.ip_address(self.ip)
File "/opt/conda/lib/python3.6/ipaddress.py", line 54, in ip_address
address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.6/site-
packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.6/site-
packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.6/site-
packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1629, in initialize
self.init_webapp()
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1379, in init_webapp
self.jinja_environment_options,
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 158, in __init__
default_url, settings_overrides, jinja_env_options)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 251, in init_settings
allow_remote_access=jupyter_app.allow_remote_access,
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 872, in _default_allow_remote
for info in socket.getaddrinfo(self.ip, self.port, 0, socket.SOCK_STREAM):
File "/opt/conda/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
So I can no longer access the entire docker container and my files and notebooks within. So I have two questions then:
Can I somehow restore my docker container or at least retrieve the materials within?
and
Why did this error occur during theme installation and how could I go about doing this without breaking my jupyter server or docker container? I have built new containers and attempted again with exactly the same results.
Any advice about how to get files from a not-running docker container, or about compatibility issues between docker, jupyter and the theme package and how to solve them would be much appreciated. For the time being I can work from a new container and keep up with schoolwork, but in the future would be nice to get back my stuff from that container and learn how to successfully change my theme if I want.
So I have an answer to half the question, we found a way to copy and export all the files from my broken, not-running docker container. The files kind of 'invisible' when the container isnt running so it took some trickery to find where they are located and what path to use to call them from terminal.
I'm running docker on a macbook and the location of all the files in a new container we made were container:./home/jovyan/.
Also made a folder called 'Dump' on my normal user desktop to transfer container contents to. After messing around with new 'fake' containers we found a successful way to pull files from a not-running one. I used
docker cp container_name:./home/jovyan/. ./Dump
Where container name is obviously your container and Dump is where you want the files to go. /jovyan/ was the largest one I could call and took everything I had out of the container, but if you knew more folder and filenames you could specify farther and extract specific things.
This is probably pretty simple for most experienced programmers, but as a newbie the hard part was finding where docker stored my container files and what path to use. /home/jovyan/. worked on my mac but could be different for you. If you have a broken container just make a new test one with a recognizable file in it and mess around until you figure out how to pull it out. Opening a new terminal window within the test jupyter notebook helped me find what docker was labeling my pathways.
Still wondering how to actually install those themes though.... dont think itll work work docker and jupyter, probably just too much incompatibility already.
If you stil have the issue, this is what fixed it for me:
In docker terminal, make sure you are using bash; so that the prompt starts with "(base)". I just typed bash and then the prompt looked good.
conda install -c conda-forge jupyterthemes (or pip if you're not using anaconda)
conda update jupyterthemes (it found some updates of other packages, apparently necessary)
jt -t monokai -f fira -fs 10 -nf ptsans -nfs 11 -N -kl -cursw 2 -cursc r -cellw 95% -T (Or another setting; this was copied from Ashraf Khan on Kaggle).
Hard refresh page (chrome on windows: ctrl-F5
I think step 3 was key here, but not sure. However it works now.
This is the error I get in VSCode when running docker-compose.
PS D:\Work\Imho\Api> docker-compose -f docker-compose.yml up
Creating network "api_default" with the default driver
Pulling ignite (apacheignite/ignite:latest)...
latest: Pulling from apacheignite/ignite
5040bd298390: Pulling fs layer
fce5728aad85: Pulling fs layer
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose\cli\main.py", line 68, in main
File "compose\cli\main.py", line 118, in perform_command
File "compose\cli\main.py", line 926, in up
File "compose\project.py", line 401, in up
File "compose\service.py", line 305, in ensure_image_exists
File "compose\service.py", line 1001, in pull
File "compose\progress_stream.py", line 37, in stream_output
File "codecs.py", line 370, in write
File "site-packages\colorama\ansitowin32.py", line 40, in write
File "site-packages\colorama\ansitowin32.py", line 141, in write
File "site-packages\colorama\ansitowin32.py", line 169, in write_and_convert
File "site-packages\colorama\ansitowin32.py", line 174, in write_plain_text
IOError: [Errno 0] Error
Failed to execute script docker-compose
docker-compose.yml:
version: '3'
services:
ignite:
image: apacheignite/ignite
This error only happens on one of my 3 machines, so I have no idea what causes it. Resetting to factory defaults or reinstalling docker and vscode doesn't help. Without docker compose the image is pulling and then working fine. All the machines are on windows 10 and latest stable docker installed.
Here is the issue I opened on github. It appeared that this behavior only occurs in VSCode powershell console, but not in regular powershell console. The solution with changing encoding suggested there didn't help.
Solution: Update to 1803 on Windows 10. If you can't update, use external command prompt.
The same thing happens to me with Python scripts. Apparently, this is a Windows Update issue:
https://github.com/Microsoft/vscode/issues/36630#issuecomment-359969098
Edit: My workaround due to my company restricting updates is to use external terminal in VS Code