I want to use an embedded IPython shell with a user_ns dictionary and a my profile configuration (ipython_config.py and the startup files). The purpose is to run a Django shell with models imported on startup. django-extensions implements a command called shell_plus that does this:
https://github.com/django-extensions/django-extensions/blob/master/django_extensions/management/commands/shell_plus.py
from IPython import embed
embed(user_ns=imported_objects)
The problem is that this does not load my startup files. embed() calls load_default_config() which I figure loads ipython_config.py.
How do I make the embedded IPython instance run my profile startup files?
I used the following workaround to run my own IPython startup script but still take advantage of shell_plus:
Create a file called shell_plus_startup.py in the same directory as manage.py. For example:
# File: shell_plus_startup.py
# Extra python code to run after shell_plus starts an embedded IPython shell.
# Run this file from IPython using '%run shell_plus_startup.py'
# Common imports
from datetime import date
# Common variables
tod = date.today()
Launch shell plus (which launches an embedded IPython shell).
python manage.py shell_plus
Manually run the startup script.
In [1]: %run shell_plus_startup.py
Then you can use variables you've defined, modules you've imported, etc.
In [2]: tod
Out[2]: datetime.date(2012, 7, 14)
Also see this answer: scripting ipython through django's shell_plus
I found a way that works if you are using django-extensions-shell_plus. It is a bit hacky, but with this way your startup file is loaded fully automatically and you don't have to type any run-command at the beginning of your ipython-session.
Therefore I edited the file shells.py from the django_extensions dir, which is in my case located in /usr/local/lib/python2.7/dist-packages/django_extensions/management/shells.py. I added these lines inside the function import_objects(options, style):, so it imports the content of the file startup.py defined by the environment param PYTHONSTARTUP.
def import_objects(options, style):
# (...)
import os, sys, pkgutil
if 'PYTHONSTARTUP' in os.environ:
try:
sys.path.append(os.environ['PYTHONSTARTUP'])
import startup
content = [element for element in dir(startup) if not element.startswith('__')]
for element in content:
imported_objects[element] = getattr(startup, element)
except Exception, ex:
sys.exit("Could not import startup module content, Error:\n%s" % ex)
Now when I launch the shell_plus-shell, I give it the environment variable to my startup python script. My bash script to launch the shell with everything in place looks like this:
#!/bin/bash
export PYTHONSTARTUP=/home/ifischer/src/myproject/startup.py # tells shell_plus to load this file
python /home/ifischer/src/myproject/manage.py shell_plus --ipython
Now I have access to all methods and variables defined in startup.py from the beginning of the ipython session.
So you can reuse that and have custom startup files for every project, pre-loading different aspects.
Maybe there is a cleaner way where to include the lines I added to the shells.py? But this approach works fine for me for now.
It does automatically load your ipython configuration starting from django-extensions==1.5.6. You can also pass additional arguments to ipython via IPYTHON_ARGUMENTS. Docs:
http://django-extensions.readthedocs.org/en/latest/shell_plus.html#configuration
Another way is using a class that derives from InteractiveShellEmbed and InteractiveShellApp. Sample, incomplete code:
from IPython.terminal.embed import InteractiveShellEmbed
from IPython.terminal.ipapp import InteractiveShellApp
class ISE(InteractiveShellEmbed, InteractiveShellApp):
def init_shell(self):
self.shell = self.instance()
self.extra_args = []
ise = ISE()
ise.init_path()
ise.init_shell()
ise.init_code()
ise.shell()
Related
ran into a weird problem where there is a shared-object import error only when I run the script from command line. It succeed if i run the script in the python console using exec(...)
I have a class that needs a shared object: foo.py:
import os
cur_dir = os.curdir()
os.chdir('/tmp/dir_with_shared_object/')
import shared_object_class
os.chdir(cur_dir)
class Useful:
... # use the shared object import
Then there is a script like this:
from foo import Useful
If I enter python console and run:
exec(open('script.py').read())
Everything works fine.
If I run this on command line:
python script.py
I will get
ModuleNotFoundError: No module named 'shared_object_class'
The python is the same. It is 3.7.3, GCC 7.3.0 Anaconda. Anyone knows what is causing this discrepancy in behavior for shared object import?
A standard way of importing from a custom directory would be to include it in the PYTHONPATH environmental variable, with export PYTHONPATH=/tmp/dir_with_shared_object/.
update
It could also be done dynamically with
import sys
p = '/tmp/dir_with_shared_object/'
sys.path.insert(0, p)
PS
I think I have an explanation for why OP's original code didn't work. According to this python reference page, the import system searches, inter alia, in "[t]he directory containing the input script (or the current directory when no file is specified)." So the behavior in the REPL loop is different from how it is when running a script. Apparently the current directory is evaluated each time an import statement is encountered, while the directory containing the input script doesn't change.
My django project fetches credentials from environment variables, now I want to automate this process and store the credentials in the vault(hashivcorp).
I have a python and shell script which fetches data from an API and exports it as environment variables, when I run it using os.system command it runs the shell script but as it runs it in a subprocess, I can't access the variables in the main(parent) process/shell. Only way of doing it by inserting the shell script in the settings.py file.
Is there any way I can do it so that I get those in the main process?
P.s: I did try sourcing, os.system didn't recognise it as a command.
Here's the code I'm running:
import os
os.environ['ENV'] = 'Demo'
os.system('python3 /home/rishabh/export.py')
print(os.environ.get('RDS_DB_NAME'))
output:
None
the python file, shell script works just fine.
One way to do it is to run export.py in the same process, as user1934428 suggested:
import os
import sys
os.environ['ENV'] = 'Demo'
sys.path.append('/home/rishabh/')
import export # runs export.py in the same process
print(os.environ.get('RDS_DB_NAME'))
This assumes there are no __name__ == '__main__' checks inside export.py.
You only need the sys.path line if export.py is in a different directory than your current script.
I'm in IPython and want to run a simple python script that I've saved in a file called "test.py".
I'd like to use the %run test.py command to execute it inside IPython, but I don't know to which folder I need to save my test.py.
Also, how can I change that default folder to something else, for example C:\Users\user\foldername ?
I tried with the .ipython folder (original installation folder) but that's not working.
I found the answer:
import os
filepath = 'C:\\Users\\user\\foldername'
os.chdir(filepath)
%run test.py
%run myprogram works for Python scripts/programs.
To run any arbitrary programs, use ! as a prefix, e.g. !myprogram.
Many common shell commands/programs (cd, ls, less, ...) are also registered as IPython magic commands (run via %cd, %ls, ...), and also have registered aliases, so you can directly run them without any prefix, just as cd, ls, less, ...
I have a .py file in the home directory which contains these three lines:
import os
os.system("cd Desktop/")
os.system("ls")
and I want it to "ls" from the "Desktop" directory but it shows contents of the /home directory.
I looked at these pages:
Calling an external command in Python
http://ubuntuforums.org/showthread.php?t=729192
but I could not understand what to do. Can anybody help me?
The two calls are separate from each other. There is no context kept between successive invocations of os.system because a new shell is spawned for every call. First os.system("cd Desktop/") switches directories to Desktop and exits. Then a new shell executes ls in the original folder.
Try chaining your commands with &&:
import os
os.system("cd Desktop/ && ls")
This will show the contents of directory Desktop.
Fabric
If your application is going to be heavy on os usage you might consider using python-fabric. It allows you to use higher level language constructs like contextmanagers to make command line invocations easier:
from fabric.operations import local
from fabric.context_managers import lcd
with lcd("Desktop/"): # Prefixes all commands with `cd Desktop && `
contents=local("ls", capture=True)
You have to consider that os.system executes the commands in a sub-shell. Hence 1) python starts a sub-shell, 2) the directory is changed, 3) then the sub-shell is completed, 4) return to previous state.
To force the current directory change you should do:
os.chdir("Desktop")
Always try and do it by other means that through os.system (os.listdir), or also by doing other than subprocess (which is an excellent module for command control in the shell)
I am trying to set up a shell-launch script like this:
./practice.py
I intend to debug commands using various modules which can be tedious to import, set up oauth settings, etc.
How do I set example practice.py to launch with this sort of preset:
import module_x
import module_y
from module_z import something
a = something.settings(var_1='123', var_1='456')
>> #start doing stuff
You can use the PYTHONSTARTUP environmental variable. Quoting the documentation:
PYTHONSTARTUP
If this is the name of a readable file, the Python commands in
that file are executed before the first prompt is displayed in
interactive mode. The file is executed in the same namespace where
interactive commands are executed so that objects defined or imported
in it can be used without qualification in the interactive session.
You can also change the prompts sys.ps1 and sys.ps2 and the hook
sys.__interactivehook__ in this file.
How you set environmental variables depends on the OS.
Put a shebang at first line with interactive mode switch:
#!/usr/bin/env python -i
import module_x
import module_y
from module_z import something
a = something.settings(var_1='123', var_1='456')
Then make sure it is executable:
chmod +x practice.py
Then you can run it like this:
./practice.py
python (and the more comfortable ipython) support -i somefile.py as option to execute the contents of a file before opening the interactive prompt.
on Linux/Unix/OS X you could simply set that as the interpreter of your script like this
#! /usr/bin/python -i
import module_x
# other preloaded settings
On Windows you need to create a batch file or similar to wrap it.