We Keep Coding

After installing jupyter themes, notebooks and docker container no longer working

I am running jupyter notebooks through a docker container. I have files, notebooks, etc within the container. I decide in class one day to attempt and install the jupyterthemes package because who doesn't like more colors. I opened a new ipynb and followed instructions per this site: https://github.com/dunovank/jupyter-themes
But it was basically just this:
!pip install jupyterthemes
!jt -t chesterish
The theme does not immediately appear and the directions suggest restarting the notebook or refreshing the browser. This is where the problems start, after trying to refresh or close and restart the notebook, it no longer works and just displays a large "500 : Internal Server Error" on the page. After trying to restart the home page of my notebook (this is locally hosted through docker and run on chrome btw), the jupyter window in chrome displays nothing at all.
Here I go back to terminal and docker and shut down the container. Then I try to restart the same container hoping it will work now. I try to start it as I usually would docker start -ai container_name but it is not successful. It displays these errors everytime
Executing the command: jupyter notebook
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py",
line 528, in get
value = obj._trait_values[self.name]
KeyError: 'allow_remote_access'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-
packages/notebook/notebookapp.py", line 869, in _default_allow_remote
addr = ipaddress.ip_address(self.ip)
File "/opt/conda/lib/python3.6/ipaddress.py", line 54, in ip_address
address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.6/site-
packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.6/site-
packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.6/site-
packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1629, in initialize
self.init_webapp()
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1379, in init_webapp
self.jinja_environment_options,
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 158, in __init__
default_url, settings_overrides, jinja_env_options)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 251, in init_settings
allow_remote_access=jupyter_app.allow_remote_access,
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 872, in _default_allow_remote
for info in socket.getaddrinfo(self.ip, self.port, 0, socket.SOCK_STREAM):
File "/opt/conda/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
So I can no longer access the entire docker container and my files and notebooks within. So I have two questions then:
Can I somehow restore my docker container or at least retrieve the materials within?
and
Why did this error occur during theme installation and how could I go about doing this without breaking my jupyter server or docker container? I have built new containers and attempted again with exactly the same results.
Any advice about how to get files from a not-running docker container, or about compatibility issues between docker, jupyter and the theme package and how to solve them would be much appreciated. For the time being I can work from a new container and keep up with schoolwork, but in the future would be nice to get back my stuff from that container and learn how to successfully change my theme if I want.

So I have an answer to half the question, we found a way to copy and export all the files from my broken, not-running docker container. The files kind of 'invisible' when the container isnt running so it took some trickery to find where they are located and what path to use to call them from terminal.
I'm running docker on a macbook and the location of all the files in a new container we made were container:./home/jovyan/.
Also made a folder called 'Dump' on my normal user desktop to transfer container contents to. After messing around with new 'fake' containers we found a successful way to pull files from a not-running one. I used
docker cp container_name:./home/jovyan/. ./Dump
Where container name is obviously your container and Dump is where you want the files to go. /jovyan/ was the largest one I could call and took everything I had out of the container, but if you knew more folder and filenames you could specify farther and extract specific things.
This is probably pretty simple for most experienced programmers, but as a newbie the hard part was finding where docker stored my container files and what path to use. /home/jovyan/. worked on my mac but could be different for you. If you have a broken container just make a new test one with a recognizable file in it and mess around until you figure out how to pull it out. Opening a new terminal window within the test jupyter notebook helped me find what docker was labeling my pathways.
Still wondering how to actually install those themes though.... dont think itll work work docker and jupyter, probably just too much incompatibility already.

If you stil have the issue, this is what fixed it for me:
In docker terminal, make sure you are using bash; so that the prompt starts with "(base)". I just typed bash and then the prompt looked good.
conda install -c conda-forge jupyterthemes (or pip if you're not using anaconda)
conda update jupyterthemes (it found some updates of other packages, apparently necessary)
jt -t monokai -f fira -fs 10 -nf ptsans -nfs 11 -N -kl -cursw 2 -cursc r -cellw 95% -T (Or another setting; this was copied from Ashraf Khan on Kaggle).
Hard refresh page (chrome on windows: ctrl-F5
I think step 3 was key here, but not sure. However it works now.

Getting botocore.exceptions.NoCredentialsError: Unable to locate credentials especially when I using CronTab

I understand we have couple of answers for this error. However, my issue is different hence the question.
I am pushing audio files from a raspberry pi to AWS-S3 instance. The upload works without any problem when I run the script manually. However the same script when run through a crontab instance it throws up the above error.
My python code is as follows:
import boto3
import argparse
import os
ap=argparse.ArgumentParser()
ap.add_argument("-f","--filetoupload", required=True, help="file to upload")
args=vars(ap.parse_args())
file = os.path.basename(args['filetoupload'])
client=boto3.client('s3', region_name='ap-south-1')
print("[INFO:] Uploading file to cloud")
client.upload_file(args['filetoupload'],'MyS3Bucket',file)
print("[INFO:] File upload completed successfully")
I am calling this python script in a bash script.
python /home/pi/s3upload.py --filetoupload /home/pi/upload/${FILENAME}.mp3
The mp3 file gets created and as said above, when I run it manually it runs without any trouble.
I checked for the config and credentials files within .aws folder. As indicated in other responses in SO, they start with default and have the right credentials set up.
The complete error message is as follows:
[INFO:] Uploading file to cloud
Traceback (most recent call last):
File "/home/pi/s3upload.py", line 10, in <module>
client.upload_file(args['filetoupload'],'MyS3Bucket',file)
File "/usr/local/lib/python2.7/dist-packages/boto3/s3/inject.py", line 110, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 279, in upload_file
future.result()
File "/usr/local/lib/python2.7/dist-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/usr/local/lib/python2.7/dist-packages/s3transfer/futures.py", line 233, in result
raise self._exception
botocore.exceptions.NoCredentialsError: Unable to locate credentials
script finished

By default, when crontab runs a script, it sets the $HOME variable to be /.
AWS looks for its credentials in a location which resolves to $HOME/.aws.
So, you need to either move your .aws directory into the root dir, which is probably not a good idea, or redefine $HOME to point to the folder that contains your .aws directory.
The easiest way to do this, if you are running your python script from a shell script, is to add:
export /HOME=/home/pi/
To the start of your shell script.

antiSMASH download_databases.py error in subprocess call

I am trying to intall antiSMASH on my research group's server but hitting an issue with the final stage of downloading the relevant databases. The makers provide a script "download_databases.py" to do this for you (see code at https://bitbucket.org/antismash/antismash/src/718da23d059742048bf044a1ed663806051eb0b2/download_databases.py?at=master&fileviewer=file-view-default).
Sadly there seems to be some kind of access issue (I'm not root and sudo doesn't help).
Server is RedHat CentOS 7.2.1511
Command run is "python download_databases.py" in antismash directory. Output is below:
Creating checksum of Pfam-A.hmm.gz
Extraction of Pfam-A.hmm.gz finished successfully.
Traceback (most recent call last):
File "download_databases.py", line 221, in <module>
main()
File "download_databases.py", line 198, in main
compile_pfam(filename)
File "download_databases.py", line 161, in compile_pfam
execute(command)
File "download_databases.py", line 51, in execute
stderr=subprocess.PIPE)
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Does anyone with experience of this have an idea of what I need to do? I've tried a local install of Python (2.7) where I have read/write access but that doesn't seem to help as the script is still looking elsewhere for the subprocess script (my own install is at the top of the PATH). Do I possibly need to install something extra with the local install or customise the install locations to include the subprocess etc stuff?
Thanks in advance. I'm no expert and really appreciate the help.
EDIT: Well I feel quite stupid now. Thanks to Hannu for suggesting I check $PATH and what binary the script was trying to execute. It turned out to be one of antiSMASH's dependencies (hmmpress) which I hadn't yet added to the $PATH on this machine. It hadn't occurred to me that it would be required prior to actually running antiSMASH.

Add print commands or something like that to your execute function and see what it tries to execute. The only explanation to get the error in that command is that the script or binary it tries to execute cannot be found.
Try adding a full path to your binary. That alone might solve the issue. Check that there are no typos in the command. Remember commands needs to be a list if it contains parameters, for example ["/usr/bin/foo/myprogram", "-a", "42", "-b", "/tmp/outputfile"] or whatever it is you need to pass to the program.

Django source code won't update on server

I have a Django website running and any updates I make to the source code won't update.
(Reason I'm changing the file is because one line of code is generating an error. What's weird is I commented out this line of code that causes the error, but the code still runs and thus still causes the error. In the django.log it shows that line causing the error still, but it also shows it commented out now. So the error log shows my new source code, but the application itself isn't executing the new code)
I am very new to Django, so I don't really know what's going on here (not my website, I got thrown on this project for work.)
Researching around for this, I have already tried to restart apache:
$ sudo apachectl restart
$ sudo service apache2 restart
and I've also tried to touch the wsgi.py file:
$ touch wsgi.py
and I have even deleted the .pyc file. Nothing has worked and the old line of code is still executing, even though the logs show it commented out.
Not sure where else to check or what else I'm missing.

Whichever service you are using, do a full stop and a full start (i.e., not just restart).
sudo service apache2 stop
sudo service apache2 start
If you are using uwsgi or gunicorn, you will have to do the same for them. Some init scripts when issuing restart do not restart the master worker process which might cause a cached compiled version of your file to still reside in memory (with the incorrect code).

With the help of #2ps I was able to figure out my problem. When I tried to stop Apache, the website was still up.
I realized there's another IP address for the website, so I'm guessing the first one must redirect to the second one?
Either way, I reopened SSH in the other IP address, restarted Apache and the source code updated immediately!
UPDATE:
As per #VidyaSagar request, I'm providing more info as it seems to be a weird fluke with Django. My OP was that a certain line of code was causing an error. I commented out this line, deleted the .pyc file, and restarted Apache. Another error occurred (as expected due to the code). So then I un-commented that line back to how it was previously, again deleted the .pyc and restarted Apache, and the system worked like normal. It seems that Django just wanted me to have a fresh re-compile of the file?
Django version: 1.7.4
Traceback of django.log
ERROR Internal Server Error: /upload/
Traceback (most recent call last):
File "/home/company/app/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/company/app/app/geo_app/views.py", line 306, in upload
shutil.make_archive(kml_dir, 'zip', root_dir=kml_dir)
File "/usr/lib/python2.7/shutil.py", line 521, in make_archive
save_cwd = os.getcwd()
OSError: [Errno 2] No such file or directory
ERROR Internal Server Error: /upload/
Traceback (most recent call last):
File "/home/company/app/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/company/app/app/geo_app/views.py", line 306, in upload
# shutil.make_archive(kml_dir, 'zip', root_dir=kml_dir)
File "/usr/lib/python2.7/shutil.py", line 521, in make_archive
save_cwd = os.getcwd()
OSError: [Errno 2] No such file or directory

For those of you with cPanel, if you go under "Setup Python App" and click "Restart" it should update. Saved me about 5 times.

pip tries to use git when git is not installed

I have a project folder that is a git repository (has a .git folder). When I use the command pip freeze, pip tries to use git. However, I don't have git installed on my system so this causes an error:
(env) PS C:\Users\eclaird\work\myproject> pip freeze
Cannot find command 'git'
Storing complete log in C:\Users\eclaird\pip\pip.log
(env) PS C:\Users\eclaird\work\myproject>
pip.log:
------------------------------------------------------------
C:\Users\eclaird\work\env\Scripts\pip-script.py run on 01/09/14 11:54:42
Cannot find command 'git'
Exception information:
Traceback (most recent call last):
File "C:\Users\eclaird\work\env\lib\site-packages\pip\basecommand.py", line 134, in main
status = self.run(options, args)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\commands\freeze.py", line 73, in run
req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\__init__.py", line 180, in from_dist
req = get_src_requirement(dist, location, find_tags)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\__init__.py", line 249, in get_src_requirement
return version_control().get_src_requirement(dist, location, find_tags)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\git.py", line 151, in get_src_requirement
repo = self.get_url(location)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\git.py", line 122, in get_url
[self.cmd, 'config', 'remote.origin.url'],
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\__init__.py", line 110, in cmd
command = find_command(self.name)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\util.py", line 108, in find_command
raise BadCommand('Cannot find command %r' % cmd)
BadCommand: Cannot find command 'git'
Is there a way to disable the git integration in pip?
(pip 1.4.1,
Python 2.7.6)

Unfortunately, no. There is no config option to enable/disable backends.
Details, found by digging in the code:
Git module is always registered: In pip/install.py, the git module is imported. At the end of it, it registers itself, and will thus be queried whenever one of the schemes declared in ´git.Git´ matches the url of the dependency.
schemes = ('git', 'git+http', 'git+https', 'git+ssh', 'git+git', 'git+file')
Exception is not handled Funnily, freezing anticipates that an error might occur when determining the dependency url to be frozen. Excerpt from pip.FrozenRequirement:
try:
req = get_src_requirement(dist, location, find_tags)
except InstallationError:
logger.warn("Error when trying to get requirement for VCS system %s, falling back to uneditable format" % ex)
InstallationError inherits from PipError. Unfortunately, an exception of type ´BadCommand´ is raised, which inherits from ´PipError´.
So, aside from hacking the source: Nothing you can do. If you need this to work, you need to install git, hack the source, or simulate a git executable. If you go for the latter, for starters you need to fake ´git config remote.origin.url´, which is called (and fails) in pip.vcs.git.Git.get_url.
Hope that helps, even though it's not a yes. ;)