Here is my question.
I use ipython notebook for daily data processing and analysis. When I create a new notebook, some essential packages must be imported first. After long-time accumulation, some process are interlinked and oriented to different task.
I can summarize the nature of my common project into these classes:
Data processing(numpy,scipy,etc. eg, from scipy import interpolate)
Data tiding(Pandas, csv, etc)
Dealing with scientific format data(netcdf4,pyhdf.eg: from osgeo import gdal)
Basic plotting(Matplotlib,Pylab)
Plotting attribute adjust. eg:
from mpl_toolkits.axes_grid1 import make_axes_locatable
from matplotlib.tri import Triangulation, UniformTriRefiner
from matplotlib.collections import PatchCollection
...
I often meet different tasks with similar working processes. The packages and some user defined function are the same(For example, I often write the correlation function myself for faster speed than pandas.corr). I had to search for past notebooks to find relevant code and copy them. Sometimes, I forget where to find them but I always know my working pattern.
So, my question begins
Is it possible to generate an meta-function library which will represent these features:
When I first work out some problem, I'll devise it into a pervasive function with broad import (one simple case, one user defined colormap can be stored for use another day)
When I use an brand new notebook, I don't need to reproduce the import process (for now, I had to write 41 lines dealing with regular work, and some of them are not essential for this project). I just need to think about every working pattern I had created and import them easily!
For example: Looping and reading specific lines in .csv file can be reproduced easily.
If it's possible, the notebook can be neat and clear!
It is certainly possible. You'll just need to understand the Python import rules- any package that appears as a sub-directory of your PYTHONPATH environment variable can be imported with a traditional import statement.
So, say you had a directory called ~/python/mypatterns containing your set of utility code. You would add ~/python to your PYTHONPATH, and ensure that there is a file called init.py (it doesn't need any contents) in that directory. So...
# Setup your environment
touch ~/mypatterns/__init__.py
export PYTHONPATH=${PYTHONPATH}:~/python
# command to start your notebook...
# in your notebook...
from mypatterns.myfile import myfunction
...
Related
I am trying to build a powerful analysis/visualization workflow with Paraview and and External Python environment. To summarize, paraview.simple package allows connection to a "paraview server", simultaneously a local paraview application gui connects to the same server.
Now, we can run things like this, directly from an external python environment, it doesn't even need to be on the same machine .
from paraview.simple import *
Connect("localhost")
c = Cone()
Show(c)
and everything works as expected. However, due to certain limitations of how client sessions are managed by pvserver, sending pure vtk objects to a pvserver is apparently a non-trivial task. For example an object produced in this way.
# Script
import SimpleITK as sitk
from glob import glob
import itk
import vtk
from vtk.util import numpy_support as vtknp
import numpy as np
from paraview.simple import *
def ReadCSAX3D(PATH, time):
reader = sitk.ImageSeriesReader()
sitk_image = sitk.ReadImage(reader.GetGDCMSeriesFileNames(PATH + '/time_' + str(time)))
return sitk_image
# Based on notebook examples provided # https://github.com/NVIDIA/ipyparaview/blob/master/notebooks/Iso-Surfaces_with_RTX.ipynb
def SitkToVTK(sitk_image):
voldims = np.asarray(sitk_image.GetSize())
npvol = sitk.GetArrayFromImage(sitk_image).reshape(np.prod(voldims))
vtkimg = vtk.vtkImageData()
vtkimg.SetDimensions(voldims)
vtkimg.SetExtent([0,voldims[0]-1, 0,voldims[1]-1, 0,voldims[2]-1])
vtkimg.SetSpacing(np.asarray(sitk_image.GetSpacing()))
vtkimg.SetOrigin(np.asarray(sitk_image.GetOrigin()))
# Get a VTK wrapper around the numpy volume
dataName = 'MRI_STACK'
vtkarr = vtknp.numpy_to_vtk(npvol)
vtkarr.SetName(dataName)
vtkimg.GetPointData().AddArray(vtkarr)
vtkimg.GetPointData().SetScalars(vtkarr)
return vtkimg
sitk_image = ReadCSAX3D("/path/to/image_file", time_step)
vtk_image = SitkToVTK(sitk_image)
Connect("localhost")
tp = TrivialProducer()
tp.GetClientSideObject().SetOutput(vtk_image)
# GetClientSideObject() returns None in this configuration --
Show(tp)
more on this here As explained in one of the answers, the two sessions need to share the same memory space on the server
I figured an interesting workaround based on the Paraview Guide on using programmable sources and filters, which allows sources that have their input being python scripts. This approach worked perfectly provided that paraview python has all the dependencies. So, now we can inject python code directly from an external python environment like this for example.
ps = ProgrammableSource()
ps.Script = "print('Hello')"
Show(ps)
But now what would be the generic way to programmatically inject code from my codebase ?. In terms of : Best approach and maintainbility of the codebase later on (I am currently thinking of using inspect module to get source lines of defined functions as strings and sending them to the programmable source on the fly. However, how would I evaluate parts of the scripts being sent to allow for function parameters, I am worried that this would be very difficult to maintain in the long run.
More generally, what would be a good example of similar problems ? i.e Injecting small python scripts, that might need to change at runtime. Let's assume in anycase that any code that I can run in the external python environment, should also run in the paraview server I can make sure that the python environments/packages are identical.
I want save some classes and functions that I have wrote for Neural Networks, which I want to use in the future whenever I need them. Is there a way to save functions and classes in some library?
So more precisely, I'm looking for a library (let's call it tools), so that I can do:
save my_function in tool
...
from tool import my_function
The way to do that in Python is to simply save your functions in a separate python file (also called a module, see the official docs).
In your case the custom function code could be saved as the file tool.py.
You can then use the syntax you mentioned:
from tool import my_function
To import this specific function, but only if the file tool.py is actually in the same directory as the session you are importing it to (this is an easy way to add the module to your Module Search path, see the offical documentation).
If you want to use the module in another directory, you can append the path where you saved tool.py to your sys.paths:
import sys
sys.path.append('/usr/dir/customcode/')
Then you can from tool import my_function in the same session, if you have tool.py saved in the directory /usr/dir/customcode/.
I have a python script that is becoming rather long. Therefore, the functions defined in the rather large single script were written into individual files for easier maintenance and to easily share them between different main scripts.
In the single script, I import numpy and other modules at the top of the file.
Now, if the function is written into a separate file, I need to import numpy in that separate file. I'd rather avoid that because with multiple functions it will end up importing numpy several times.
Can this be done?
Thanks
Yes it can be done, as described here: Python: Importing an "import file"
In short, you can put all imports in another file and just import that file when you need it.
Note though that each file needs to have numpy imported, one way or another.
EDIT:
Also read this: Does python optimize modules when they are imported multiple times? to understand how python handles multiple imports. Thanks to #EdChum
I have this simple piece of code in a file:
frame = DataFrame(np.random.randn(2, 4),
index=pd.date_range('1/1/2000', periods=2, freq='W-WED'),
columns=['Colorado', 'Texas', 'New York', 'Ohio'])
When I try to run the file I get error messages telling me that it doesn't know what to do with DataFrame, np, or pd. I can fix this easily by adding in "
from pandas import DataFrame
import numpy as np
import pandas as pd
My question: is there a way that I can avoid having to import these libraries within every file that I want to use those tools?
I had imported them in the command line before running this file but this didn't seem to make a difference.
New to Python. Thanks in advance for the help.
Using Python 2 in the Canopy editor.
Generally, if you are going to use a package/module, then you should* import it in every module that needs it. While it might be a little annoying writing the same imports over and over, it's better in the long run because it makes your code a lot more clear. e.g. you know what np or DataFrame is and where it comes from.
I suppose that it's worth noting that you probably can get around this requirement by writing your own import hook, but . . .That's way too complicated a thing to do just to enable you to write code that will likely confuse your co-workers :-)
*There are some conveniences that you can use to get around this in the interactive interpretter, but not in anything that gets imported via the normal mechanisms.
I'm a java developer new to python. In java, you can access all classes in the same directory without having to import them.
I am trying to achieve the same behavior in python. Is this possible?
I've tried various solutions, for example by importing everything in a file which I import everywhere. That works, but I have to type myClass = rootFolder.folder2.folder3.MyClass() each time I want to access a foreign class.
Could you show me an example for how a python architecture over several directories works? Do you really have to import all the classes you need in each file?
Imagine that I'm writing a web framework. Will the users of the framework have to import everything they need in their files?
Put everything into a folder (doesn't matter the name), and make sure that that folder has a file named __init__.py (the file can be empty).
Then you can add the following line to the top of your code:
from myfolder import *
That should give you access to everything defined in that folder without needing to give the prefix each time.
You can also have multiple depths of folders like this:
from folder1.folder2 import *
Let me know if this is what you were looking for.