I am trying to work with jupyterlab on a remote server that I don't manage, and I want to add my custom libraries to the path so that I can import and use them. Normally, I would go into .bashrc and add to PYTHONPATH there using
export PYTHONPATH="/home/username/path/to/module:$PYTHONPATH"
but this hasn't worked. I have tried this in .bashrc and .bash_profile to no fortune. I have also tried
export JUPYTER_PATH="/home/username/path/to/module:$JUPYTER_PATH"
as I read that somewhere else, and tried it in both the files named above.
What else can I try?
Ideally I'd like to put in some line in jupyterlab that returns the file it is using to add to the path, is that possible?
Or perhaps there is some command I can type directly into a terminal that I can access through jupyterlab that would allow me to add things to my path perminantley. I know that I can use os.path.insert (or similar) at the start of a notebook but as there are certain things I will want to use in every notebook this is a less than ideal solution for me.
Thanks
In a Specific Notebook
Manually append the path to sys.path in the first cell of the notebook
import sys
extra_path = ... # whatever it is
if extra_path not in sys.path:
sys.path.append(extra_path)
As a System Configuration
Modify ~/.ipython/profile_default/ipython_config.py using the shell functionality so that the path gets modified for every notebook.
If that file does not exist, create it by using ipython profile create.
Then insert the modification to sys.path into it by modifying the c.InteractiveShellApp.exec_lines variable, e.g.
c.InteractiveShellApp.exec_lines = [
'import sys; sys.path.append(<path to append>)'
]
Partially stolen from this answer, which has a different enough context to warrant being a different question.
Related
I'm using WingIDE for development and Ipython for running my scripts. I'm facing some inconvenience on several points:
Whenever I update my code, I have to reload my module in order to update it in IPython. To solve that I followed Jomonsugi's answer in the following link: How to automatically reload modules in IPython?
and it works.
Each time I launch IPython, my path is set to my exe's location and I have to cd to my script's directory.
I tried to change directory automatically when launching IPython by setting the parameter c.InteractiveShell.ipython_dir = <mypath> in ipython_config.py but it doesn't work.
Is it possible to set the default path? Is there a particular parameter to set in ipython_config.py?
One way is to use your startup.py file. It should be located somewhere like:
C:/Users/yourname/.ipython/profile_default
If it's not there already, create it. Then use
import os
os.chdir('C:/Users/mypath')
Those two lines will then be run at IPython startup.
Source: IPython.org > Startup Files
I’m sure Brad Solomon’s answer is right for his version of IPython, but I’ve just downloaded IPython with pip install ipython and my startup files are in a directory nested one deeper than his. My IPython version is 7.18.1, and the start-up files are located in ~/.ipython/profile_default/startup/. There is a README there which states
This is the IPython startup directory
.py and .ipy files in this directory will be run *prior* to any code
or files specified via the exec_lines or exec_files configurables
whenever you load this profile.
Files will be run in lexicographical order, so you can control the
execution order of files with a prefix, e.g.::
00-first.py
50-middle.py
99-last.ipy
That’s pretty self-descriptive, but I would add to this (for anyone coming here from here) that %load_ext autoreload is an IPython command, so you might want to create a file startup.ipy with contents something like this:
%load_ext autoreload
%autoreload 2
print('IPython startup file - created 2020/9/28')
I've a problem that Jupyter can't see env variable in bashrc file, is there a way to load these variables in jupyter or add custome variable to it?
To set an env variable in a jupyter notebook, just use a % magic commands, either %env or %set_env, e.g., %env MY_VAR=MY_VALUE or %env MY_VAR MY_VALUE. (Use %env by itself to print out current environmental variables.)
See: http://ipython.readthedocs.io/en/stable/interactive/magics.html
You can also set the variables in your kernel.json file:
My solution is useful if you need the same environment variables every time you start a jupyter kernel, especially if you have multiple sets of environment variables for different tasks.
To create a new ipython kernel with your environment variables, do the following:
Read the documentation at https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs
Run jupyter kernelspec list to see a list with installed kernels and where the files are stored.
Copy the directory that contains the kernel.json (e.g. named python2) to a new directory (e.g. python2_myENV).
Change the display_name in the new kernel.json file.
Add a env dictionary defining the environment variables.
Your kernel json could look like this (I did not modify anything from the installed kernel.json except display_name and env):
{
"display_name": "Python 2 with environment",
"language": "python",
"argv": [
"/usr/bin/python2",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env": {"LD_LIBRARY_PATH":""}
}
Use cases and advantages of this approach
In my use-case, I wanted to set the variable LD_LIBRARY_PATH which effects how compiled modules (e.g. written in C) are loaded. Setting this variable using %set_env did not work.
I can have multiple python kernels with different environments.
To change the environment, I only have to switch/ restart the kernel, but I do not have to restart the jupyter instance (useful, if I do not want to loose the variables in another notebook). See -however - https://github.com/jupyter/notebook/issues/2647
If you're using Python, you can define your environment variables in a .env file and load them from within a Jupyter notebook using python-dotenv.
Install python-dotenv:
pip install python-dotenv
Load the .env file in a Jupyter notebook:
%load_ext dotenv
%dotenv
You can setup environment variables in your code as follows:
import sys,os,os.path
sys.path.append(os.path.expanduser('~/code/eol_hsrl_python'))
os.environ['HSRL_INSTRUMENT']='gvhsrl'
os.environ['HSRL_CONFIG']=os.path.expanduser('~/hsrl_config')
This if of course a temporary fix, to get a permanent one, you probably need to export the variables into your ~.profile, more information can be found here
A gotcha I ran into: The following two commands are equivalent. Note the first cannot use quotes. Somewhat counterintuitively, quoting the string when using %env VAR ... will result in the quotes being included as part of the variable's value, which is probably not what you want.
%env MYPATH=C:/Folder Name/file.txt
and
import os
os.environ['MYPATH'] = "C:/Folder Name/file.txt"
If you need the variable set before you're starting the notebook, the only solution which worked for me was env VARIABLE=$VARIABLE jupyter notebook with export VARIABLE=value in .bashrc.
In my case tensorflow needs the exported variable for successful importing it in a notebook.
A related (short-term) solution is to store your environment variables in a single file, with a predictable format, that can be sourced when starting a terminal and/or read into the notebook. For example, I have a file, .env, that has my environment variable definitions in the format VARIABLE_NAME=VARIABLE_VALUE (no blank lines or extra spaces). You can source this file in the .bashrc or .bash_profile files when beginning a new terminal session and you can read this into a notebook with something like,
import os
env_vars = !cat ../script/.env
for var in env_vars:
key, value = var.split('=')
os.environ[key] = value
I used a relative path to show that this .env file can live anywhere and be referenced relative to the directory containing the notebook file. This also has the advantage of not displaying the variable values within your code anywhere.
If your notebook is being spawned by a Jupyter Hub, you might need to configure (in jupyterhub_config.py) the list of environment variables that are allowed to be carried over from the JupyterHub process environment to the Notebook environment by setting
c.Spawner.env_keep = [VAR1, VAR2, ...]
(https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.env_keep)
See also: Spawner.environment
If you are using systemd I just found out that you seem to have to add them to the systemd unit file. This on Ubuntu 16. Putting them into the .profile and .bashrc (even the /etc/profile) resulted in the ENV Vars not being available in the juypter notebooks.
I had to edit:
/lib/systemd/system/jupyer-notebook.service
and put in the variable i wanted to read in the unit file like:
Environment=MYOWN_VAR=theVar
and only then could I read it from within juypter notebook.
you can run jupyter notebook with docker and don(t have to manage dependancy leaks.
docker run -p 8888:8888 -v /home/mee/myfolder:/home/jovyan --name notebook1 jupyter/notebook
docker exec -it notebook1 /bin/bash
then kindly ask jupyter about the opened notebooks,
jupyter notebook list
http:// 0.0.0.0:8888/?token=012456788997977a6eb11e45fffff
Url can be copypasted, verify port if you have changed it.
Create a notebook and paste the following,
into the notebook
!pip install python-dotenv
import dotenv
%load_ext dotenv
%dotenv
Does anyone know if it is possible to load ipython preloaded with custom packages please?
I'm running Python 2.7 on Windows 8.
When I load a DOS prompt, I run ipython preloaded with pylab by typing
ipython --pylab
I've managed to create a shortcut to open a DOS prompt with this automatically fired, thus effectively creating a shortcut to iPython.
However, I'd like iPython to start preloaded with some of my custom packages. So I wonder if there is a way to start iPython and automatically execute the following lines, say:
import package1 as my_package
import package2 as my_second_package
I've had a look online and there's some information on "magic" commands and scripts in iPython which looks like it might help, although I wasn't sure how to use this.
All guidance welcomed.
Many thanks.
What you want is a startup script.
First run ipython locate profile to find the profile folder. Then find a startup folder in there. Create a .py file (any name) in the startup folder with the imports you want, and IPython will run that whenever you it starts.
If you have a look at the documentation, you will find out that IPython will run whatever is inside the file pointed by the PYTHONSTARTUP variable.
Create one, export the variable, and there you go.
References:
http://ipython.org/ipython-doc/stable/interactive/reference.html#ipython-as-your-default-python-environment
https://docs.python.org/2/using/cmdline.html#envvar-PYTHONSTARTUP
Currently, when trying to reference some library code, I'm doing this at the top of my python file:
import sys
sys.path.append('''C:\code\my-library''')
from my-library import my-library
Then, my-library will be part of sys.path for as long as the session is active. If I start a new file, I have to remember to include sys.path.append again.
I feel like there must be a much better way of doing this. How can I make my-library available to every python script on my windows machine without having to use sys.path.append each time?
Simply add this path to your PYTHONPATH environment variable. To do this, go to Control Panel / System / Advanced / Environment variable, and in the "User variables" sections, check if you already have PYTHONPATH. If yes, select it and click "Edit", if not, click "New" to add it.
Paths in PYTHONPATH should be separated with ";".
You should use
os.path.join
to make your code more reliable.
You have already used __my-library__ in the path. So don't use it the second time in import.
If you have a directory structure like this
C:\code\my-library\lib.py and a function in there, e.g.:
def main():
print("Hello, world")
then your resulting code should be
import sys
sys.path.append(os.path.join('C:/', 'code', 'my-library'))
from lib import main
If this is a library that you use throughout your code, you should install it as such. Package it up properly, and either install it in your site-packages directory - or, if it's specific to certain projects, use virtualenv and install it just within the relevant virtualenvs.
To do such a thing, you'll have to use a sitecustomize.py (or usercustomize.py) file where you'll do your sys.path modifications (source python docs).
Create the sitecustomize.py file into the \Lib\site-packages directory of your python installation, and it will be imported each time a python interpreter is launched.
If you are doing this interactively, the best thing to do would be to install ipython and configure your startup settings to include that code. If you intend to have it be part of a script you run from the interpreter, the same thing applies, since it will have access to your namespace.
On the other hand, a stand alone script should not include that automatically. In the future, you or some other maintainer will come along, and all the code should be obvious, and not dependent upon a specific machine setup. The best thing to do would be to set up a skeleton file for new projects that includes all of the basic functionality you need. That, along with oft-used snippets will handle the problem.
All of your code to run the script, will be in the script, and you won't have to think about adding that code every time.
Using jupyter with multiple environments, adding the path to .bashrc didn't work. I had to edit the kernel.json file for that particular kernel and append it to the PYTHONPATH in env section.
This only worked in that kernel but maybe this can help someone else.
My use case is I want to initialize some functions in a file and then start up IPython with those functions defined. Is there a way to do something like this?
ipython --run_script=myscript.py
In recent versions of IPython, you do need to add the -i option to get into the interactive environment afterwards. Without the -i it just runs the code in myfile.py and returns to the prompt.
ipython -i myfile.py
Per the docs, it's trivial:
You start IPython with the command:
$ ipython [options] files
If invoked with no options, it
executes all the files listed in
sequence and drops you into the
interpreter while still acknowledging
any options you may have set in your
ipythonrc file. This behavior is
different from standard Python, which
when called as python -i will only
execute one file and ignore your
configuration setup.
So, just use ipython myfile.py... and there you are!-)
You can use IPython profiles to define startup scripts that will run every time you start IPython. A full description of profiles, is given here. You can create multiple profiles with different startup files.
Assuming you only need one profile, and always want the same startup files every time you start IPython, you can simply modify the default profile. To do this, first find out where your IPython configuration directory is in an ipython session.:
In [1]: import IPython
In [2]: IPython.paths.get_ipython_dir() # As of IPython v4.0
In [3]: print(ipython_config_dir)
/home/johndoe/.config/ipython
For this example, I am using Ubuntu Linux, and the configuration directory is in /home/johndoe/.config/ipython, where johndoe is the username.
The default_profile is in the profile_default subdirectory. Put any starting scripts in profile_default/startup. In the example here, the full path would be /home/johndoe/.config/ipython/profile_default/startup.
Nowadays, you can use the startup folder of IPython, which is located in your home directory (C:\users\[username]\.ipython on Windows). Go into the default profile and you'll see a startup folder with a README file. Just put any Python scripts in there, or if you want IPython commands, put them in a file with an .ipy extension.
You seem to be looking for IPython's %run magic command.
By typing in IPython:
%run hello_world.py
you'll run the hello.py program saved in your home directory. The functions and variables defined in that script will be accessible to you too.
The following is for the case when you want your startup scripts to automatically be run whenever you use IPython (instead of having a script that you must specify each time you run IPython).
For recent versions (i.e., 5.1.0) of IPython, place one or more python scripts you wish to have executed in the IPYTHON_CONFIG_DIR/profile_PROFILENAME/startup folder.
On Linux, for example, you could put your Python startup code into a file named ~/.ipython/profile_default/startup/10-mystartupstuff.py if you want it to run when no IPython profile is specified.
Information on creating and using IPython profiles is available here.
Update to Caleb's answer for Python 3.5 in Ubuntu 14.04 (Trusty Tahr): I made this answer self-contained by copying relevant parts of Caleb's answer.
You can use IPython profiles to define startup scripts that will run every time you start IPython. A full description of profiles, is given here. You can create multiple profiles with different startup files.
Assuming you only need one profile, and always want the same startup files every time you start IPython, you can simply modify the default profile. To do this, first find out where your IPython configuration directory is in an IPython session:
Input:
import IPython
ipython_config_dir = IPython.paths.get_ipython_dir()
print(ipython_cofig_dir)
Output:
/home/johndoe/.ipython
For this example, johndoe is the username.
Inside the /.ipython folder, the default_profile is in the profile_default subdirectory. Put any starting scripts in profile_default/startup. In the example here, the full path would be
/home/johndoe/.ipython/profile_default/startup