Is there a way to enable a renderer except for calling alt.renderers.enable('mimebundle') in code? So if the user imports altair she doesn't have to perform any additional actions?
For example, in plotly you can set an environment variable PLOTLY_RENDERER=plotly_mimetype. Is there something similar in altair?
No, Altair does not currently have any mechanism to specify a renderer aside from calling alt.renderers.enable.
But if you are using Jupyter, you could provide an IPython startup script that does this; for example, you can create a file at the path ~/.ipython/profile_default/startup/start.py with the following contents:
import altair
altair.renderers.enable('notebook')
and this will be executed at the start of any Jupyter/IPython session.
If you don't wish to import Altair in every session, you could instead define in this file a Python import hook that will execute custom code the first time Altair is imported. For example, it might look something like this:
import imp
import os
import sys
class _AltairImportHook(object):
def find_module(self, fullname, path=None):
if fullname != 'altair':
return None
self.module_info = imp.find_module(fullname, path)
return self
def load_module(self, fullname):
"""Loads Altair normally and runs pre-initialization code."""
previously_loaded = fullname in sys.modules
altair = imp.load_module(fullname, *self.module_info)
if not previously_loaded:
try:
altair.renderers.enable('notebook')
except:
pass
return altair
sys.meta_path = [_AltairImportHook()] + sys.meta_path
Related
I have a relatively complex ecosystem of applications and libraries that are scheduled to run in my environment.
I am trying to improve my logging and in particular I'd like to write debug information to a logging file, and I'd like that log to contain the logger.debug("string") lines from all the imported libraries I wrote, but not from libraries I import from pypi.
example:
import sys
import numpy
from bs4 import BeautifulSoup
import logging
import mylibrary
import myotherlibrary
logger = logging.getLogger(application_name) # I don't use _ _ name _ _ in all of them, but I can change this line as necessary
so in this case when I set logger level to debug, I'd like to see debug information from the current script, from mylibrary and from myotherlibrary , but not from bs4,numpy, etc.
bonus: Ideally I would like to not have to hardcode every time the name of the libraries, but just have the script "know" it (from naming convention maybe?)
If anyone has any ideas it'd be greatly appreciated!
Python doesn't really have a concept of "libraries I wrote" vs "libraries imported with pypi" - a library is a library unfortunately.
However, depending on how your libraries are set up, you may be able to get a realllly hacky custom logger?
By default, Python libraries installed with pip go to a central location - usually something like /usr/local/lib or %APPDATA% on windows. In contrast, local libraries are usually within the same directory as the calling script. We can use this to our advantage!
The following code demonstrates a kinda proof-of-concept - I've left a few methods needing implementing as an exercise ;)
#CustomLogger.py
import __main__
import logging
import os
#create a custom log class, inheriting the current logger class
class CustomLogger(logging.getLoggerClass()):
custom_lib = False
def __init__(self, name):
#initialise the base logger
super().__init__(name)
#get the directory we are being run from
current_dir = os.path.dirname(__main__.__file__)
permutations = ['/', '.py', '.pyc']
#check if we are a custom library, or if we are one installed via pip etc
self.custom_lib = self.checkExists(current_dir, permutations)
self.propagate = not self.custom_lib
def checkExists(self, current_dir, permutations):
#loop through each permutation and see if a file matching that spec exists
#currently looks for .py/.pyc files and directories
for perm in permutations:
file = os.path.join(current_dir, self.name + perm)
if os.path.exists(file):
return True
return False
def isEnabledFor(self, level):
if self.custom_lib:
return super().isEnabledFor(level)
return False
#the hackiest part :)
#these are two sample overrides that only log if we're a custom
#library (i.e. one we've written, not installed)
#there are a few more methods that I've not implemented, a full
#list is available at https://docs.python.org/3/library/logging.html#logging.Logger
def debug(self, msg, *args, **kwargs):
if self.custom_lib:
return super().debug(msg, args, kwargs)
def info(self, msg, *args, **kwargs):
if self.custom_lib:
return super().info(msg, args, kwargs)
#most important part - also override the logger class
#this means that any calls to logging.getLogger() will use our new subclass
logging.setLoggerClass(CustomLogger)
You could then use it like this:
import CustomLogger #needs importing first so it ensures the logger is setup
import sys
import numpy
from bs4 import BeautifulSoup
import logging
import mylibrary
import myotherlibrary
logger = logging.getLogger(application_name) #returns type CustomLogger
Trying to add a few imports to my IPython profile so that when I open a kernel in the Spyder IDE they're always loaded. Spyder has a Qt interface (I think??), so I (a) checked to make sure I was in the right directory for the profile using the ipython locate command in the terminal (OSX), and (b) placing the following code in my ipython_qtconsole_config.py file:
c.IPythonQtConsoleApp.exec_lines = ["import pandas as pd",
"pd.set_option('io.hdf.default_format', 'table')",
"pd.set_option('mode.chained_assignment','raise')",
"from __future__ import division, print_function"]
But when I open a new window and type pd.__version__ I get the NameError: name 'pd' is not defined error.
Edit: I don't have any problems if I run ipython qtconsole from the Terminal.
Suggestions?
Thanks!
Whether Spyder uses a QT interface or not shouldn't be related to which of the IPython config files you want to modify. The one you chose to modify, ipython_qtconsole_config.py is the configuration file that is loaded when you launch IPython's QT console, such as with the command line command
user#system:~$ ipython qtconsole
(I needed to update pyzmq for this to work.)
If Spyder maintains a running IPython kernel and merely manages how to display that for you, then Spyder is probably just maintaining a regular IPython session, in which case you want your configuration settings to go into the file ipython_config.py at the same directory where you found ipython_qtconsole_config.py.
I manage this slightly differently than you do. Inside of ipython_config.py the top few lines for me look like this:
# Configuration file for ipython.
from os.path import join as pjoin
from IPython.utils.path import get_ipython_dir
c = get_config()
c.InteractiveShellApp.exec_files = [
pjoin(get_ipython_dir(), "profile_default", "launch.py")
]
What this does is to obtain the IPython configuration directory for me, add on the profile_default subdirectory, and then add on the name launch.py which is a file that I created just to hold anything I want to be executed/loaded upon startup.
For example, here's the first bit from my file launch.py:
"""
IPython launch script
Author: Ely M. Spears
"""
import re
import os
import abc
import sys
import mock
import time
import types
import pandas
import inspect
import cPickle
import unittest
import operator
import warnings
import datetime
import dateutil
import calendar
import copy_reg
import itertools
import contextlib
import collections
import numpy as np
import scipy as sp
import scipy.stats as st
import scipy.weave as weave
import multiprocessing as mp
from IPython.core.magic import (
Magics,
register_line_magic,
register_cell_magic,
register_line_cell_magic
)
from dateutil.relativedelta import relativedelta as drr
###########################
# Pickle/Unpickle methods #
###########################
# See explanation at:
# < http://bytes.com/topic/python/answers/
# 552476-why-cant-you-pickle-instancemethods >
def _pickle_method(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
return _unpickle_method, (func_name, obj, cls)
def _unpickle_method(func_name, obj, cls):
for cls in cls.mro():
try:
func = cls.__dict__[func_name]
except KeyError:
pass
else:
break
return func.__get__(obj, cls)
copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)
#############
# Utilities #
#############
def interface_methods(*methods):
"""
Class decorator that can decorate an abstract base class with method names
that must be checked in order for isinstance or issubclass to return True.
"""
def decorator(Base):
def __subclasshook__(Class, Subclass):
if Class is Base:
all_ancestor_attrs = [ancestor_class.__dict__.keys()
for ancestor_class in Subclass.__mro__]
if all(method in all_ancestor_attrs for method in methods):
return True
return NotImplemented
Base.__subclasshook__ = classmethod(__subclasshook__)
return Base
def interface(*attributes):
"""
Class decorator checking for any kind of attributes, not just methods.
Usage:
#interface(('foo', 'bar', 'baz))
class Blah
pass
Now, new classes will be treated as if they are subclasses of Blah, and
instances will be treated instances of Blah, provided they possess the
attributes 'foo', 'bar', and 'baz'.
"""
def decorator(Base):
def checker(Other):
return all(hasattr(Other, a) for a in attributes)
def __subclasshook__(cls, Other):
if checker(Other):
return True
return NotImplemented
def __instancecheck__(cls, Other):
return checker(Other)
Base.__metaclass__.__subclasshook__ = classmethod(__subclasshook__)
Base.__metaclass__.__instancecheck__ = classmethod(__instancecheck__)
return Base
return decorator
There's a lot more, probably dozens of helper functions, snippets of code I've thought are cool and just want to play with, etc. I also define some randomly generated toy data sets, like NumPy arrays and Pandas DataFrames, so that when I want to poke around with some one-off Pandas syntax or something, some toy data is always right there.
The other upside is that this factors out the custom imports, function definitions, etc. that I want loaded, so if I want the same things loaded for the notebook and/or the qt console, I can just add the same bit of code to exec the file launch.py and I can make changes in only launch.py without having to manually migrate them to each of the three configuration files.
I also uncomment a few of the different settings, especially for plain IPython and for the notebook, so the config files are meaningfully different from each other, just not based on what modules I want imported on start up.
I'm currently writing some kind of tiny api to support extending module classes. Users should be able to just write their class name in a config and it gets used in our program. The contract is, that the class' module has a function called create(**kwargs) to return an instance of our base module class, and is placed in a special folder. But the isinstance check Fails as soon as the import is made dynamically.
modules are placed in lib/services/name
module base class (in lib/services/service)
class Service:
def __init__(self, **kwargs):
#some initialization
example module class (in lib/services/ping)
class PingService(Service):
def __init__(self, **kwargs):
Service.__init__(self,**kwargs)
# uninteresting init
def create(kwargs):
return PingService(**kwargs)
importing function
import sys
from lib.services.service import Service
def doimport( clazz, modPart, kw, class_check):
path = "lib/" + modPart
sys.path.append(path)
mod = __import__(clazz)
item = mod.create(kw)
if class_check(item):
print "im happy"
return item
calling code
class_check = lambda service: isinstance(service, Service)
s = doimport("ping", "services", {},class_check)
print s
from lib.services.ping import create
pingService = create({})
if isinstance(pingService, Service):
print "why this?"
what the hell am I doing wrong
here is a small example zipped up, just extract and run test.py without arguments
zip example
The problem was in your ping.py file. I don't know exactly why, but when dinamically importing it was not accepting the line from service import Service, so you just have to change it to the relative path: from lib.services.service import Service. Adding lib/services to the sys.path could not make it work the inheritance, which I found strange...
Also, I am using imp.load_source which seems more robust:
import os, imp
def doimport( clazz, modPart, kw, class_check):
path = os.path.join('lib', modPart, clazz + '.py')
mod = imp.load_source( clazz, path )
item = mod.create(kw)
if class_check(item):
print "im happy"
return item
In one of my testing scripts in Python I use this pattern several times:
sys.path.insert(0, "somedir")
mod = __import__(mymod)
sys.path.pop(0)
Is there a more concise way to temporarily modify the search path?
You could use a simple context manager:
import sys
class add_path():
def __init__(self, path):
self.path = path
def __enter__(self):
sys.path.insert(0, self.path)
def __exit__(self, exc_type, exc_value, traceback):
try:
sys.path.remove(self.path)
except ValueError:
pass
Then to import a module you can do:
with add_path('/path/to/dir'):
mod = __import__('mymodule')
On exit from the body of the with statement sys.path will be restored to the original state. If you only use the module within that block you might also want to delete its reference from sys.modules:
del sys.modules['mymodule']
Appending a value to sys.path only modifies it temporarily, i.e for that session only.
Permanent modifications are done by changing PYTHONPATH and the default installation directory.
So, if by temporary you meant for current session only then your approach is okay, but you can remove the pop part if somedir is not hiding any important modules that is expected to be found in in PYTHONPATH ,current directory or default installation directory.
http://docs.python.org/2/tutorial/modules.html#the-module-search-path
Here is an alternative implementation of the contextmanager implementation from Eugene Yarmash (use contextlib and pathlib.Path-compatible):
import os
import sys
import contextlib
from typing import Iterator, Union
#contextlib.contextmanager
def add_sys_path(path: Union[str, os.PathLike]) -> Iterator[None]:
"""Temporarily add the given path to `sys.path`."""
path = os.fspath(path)
try:
sys.path.insert(0, path)
yield
finally:
sys.path.remove(path)
with add_sys_path('/path/to/dir'):
mymodule = importlib.import_module('mymodule')
If you're testing uses pytest, they have a great fixture that, among other traits, handles this exact case:
The monkeypatch fixture helps you to safely set/delete an attribute,
dictionary item or environment variable, or to modify sys.path for
importing...
All modifications will be undone after the requesting test function or
fixture has finished. The raising parameter determines if a KeyError
or AttributeError will be raised if the target of the set/deletion
operation does not exist
In describing syspath_prepend:
Use monkeypatch.syspath_prepend to modify sys.path which will also
call pkg_resources.fixup_namespace_packages and
importlib.invalidate_caches().
sample use:
def test_myfunc(monkeypatch):
with monkeypatch.context() as m:
m.syspath_prepend('my/module/path')
mod = __import__(mymod)
# Out here, the context manager expires and syspath is reset
Either it's lack of sleep but I feel silly that I can't get this. I have a plugin, I see it get loaded but I can't instantiate it in my main file:
from transformers.FOMIBaseClass import find_plugins, register
find_plugins()
Here's my FOMIBaseClass:
from PluginBase import MountPoint
import sys
import os
class FOMIBaseClass(object):
__metaclass__ = MountPoint
def __init__(self):
pass
def init_plugins(self):
pass
def find_plugins():
plugin_dir = os.path.dirname(os.path.realpath(__file__))
plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith("Transformer.py")]
sys.path.insert(0, plugin_dir)
for plugin in plugin_files:
mod = __import__(plugin)
Here's my MountPoint:
class MountPoint(type):
def __init__(cls,name,bases,attrs):
if not hasattr(cls,'plugins'):
cls.plugins = []
else:
cls.plugins.append(cls)
I see it being loaded:
# /Users/carlos/Desktop/ws_working_folder/python/transformers/SctyDistTransformer.pyc matches /Users/carlos/Desktop/ws_working_folder/python/transformers/SctyDistTransformer.py
import SctyDistTransformer # precompiled from /Users/carlos/Desktop/ws_working_folder/python/transformers/SctyDistTransformer.pyc
But, for the life of me, I can't instantiate the 'SctyDistTransformer' module from the main file. I know I'm missing something trivial. Basically, I want to employ a class loading plugin.
To dymically load Python modules from arbitrary folders use imp module:
http://docs.python.org/library/imp.html
Specifically the code should look like:
mod = imp.load_source("MyModule", "MyModule.py")
clz = getattr(mod, "MyClassName")
Also if you are building serious plug-in architecture I recommend using Python eggs and entry points:
http://wiki.pylonshq.com/display/pylonscookbook/Using+Entry+Points+to+Write+Plugins
https://github.com/miohtama/vvv/blob/master/vvv/main.py#L104