Access different module versions - python

Imagine I use a pretty standard module, like numpy, and that I have a custom version locally.
Currently, I would store my custom numpy in a folder, say
.../MyPackages/
which I added to sys.path in second position (after ''). This way, any package that I use and which calls numpy, will use my numpy by default. When I don't want to use my numpy as default, I rename .../MyPackages/numpy to .../MyPackages/_numpy. This way, I can still call it manually with
import _numpy
if I want to, but it's not the default behaviour anymore.
This way of storing custom packages works pretty well for me so far, but doesn't allow me to call the usual numpy (not my custom version) without renaming the folder first, which is a bit heavy...
Is there a cleaner/better way to organize my packages to add this flexibility?
Thanks in advance.
_ I described above what I tried._

Related

What is the best way to use different versions of a function from a library in different scripts?

Say I write a Python script for data analysis that includes a function parse_csv which might return a dataframe in a certain format. I can then analyse and plot this output etc.
Some time later, working with different data in the same format, I decide that it's better to parse the csv in a different way, so I improve the function so that it returns a slightly different output to be plotted etc. in a slightly different way. E.g. a dataframe that now contains datetime objects for the columns instead of just the date and time as a string.
I still want all the original scripts to run. I want all of the functions for the old and new scripts to be in one python library for the project. I don't want to go back and change all of the code for the old scripts to work with the new and improved function.
What is the best way to version my functions? I can think of a few options, but I'm wondering if missing anything:
Is there a way to do version control of libraries in python?, e.g. "import myLib version 0.1". Then I can just import different versions of the library for different scripts
I call the functions different things e.g. parse_csv_1, parse_csv_2
I have an argument to the function to redirect to different functions inside the function e.g.:
def parse_csv(csv, version=1):
if version == 1:
return parse_csv_1(csv)
elif version == 2:
return parse_csv_2(csv)
Is there a way to achieve what I want using the first option, as it seems cleaner to me, and would work for improvements to multiple functions across scripts? Or do I need to do this in a different way?
I have seen this question which refers to importing specific versions of libraries, but I haven't been able to see how I would install multiple versions at the same time.
That is a pretty interesting question. Let's address them one by one:
1. Library versioning:
If you are publishing your library to PyPi or any other repository, you can have different versions, and install a specific version using pip. However, you cannot have multiple versions of the same library
2. Calling it different things:
This would the prefered method. However, I don't recommend naming it _1, _2. You should name it in a descriptive way so you understand the difference between the different functions. So based on your description, you'd have parse_csv and parse_csv_date_as_datetime.
3. Redirecting with arguments
In general this is not a good idea. Arguments should not fundamentally change how your function behaves. That can lead to unexpected behaviours. Parameters should be the data you perform computations on.
In conclusion
I recommend just creating a second function. Use the original in your past code, use the new one in the new code.
If you really must have a single function, you could try applying the Strategy Pattern to handle different data formatting tasks, passing in a strategy function and having a default what matches the current behaviour, this however I'd say is probably beyond your current level.

Inject code automatically to a python project and execute

I want to inject some code to all python modules in my project automatically.
I used ast.NodeTransformer and managed to change it quite easily, the problem is that I want to run the project.
An ast node is per module and I want to change all modules in the project and then run; and I have this example
The problem is that it applies to one node, viz. one file. I want to run this file, which imports and uses other files which I want to change too, so I'm not sure how to get it done.
I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
Don't really know how to start, any suggestions?
I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
From 3.9 onward there is ast.unparse, which practically does AST to source conversion after you transform it.

How to make a list of the user-created python files that a module depends on?

I am interested in using doit to automate the build process of a python package.
If possible, I would like doit to re-execute a task if any of the user-created source files it depends on have changed.
From my understanding, the best way to accomplish this would be to use the file_dep key and a list of the dependent source files, however I am having a lot of trouble generating this list.
I've tried using sys.modules and inspect.getmembers(), but these solutions can't deal with import statements that do not import a module, such as from x import Y, which is unfortunately a common occurrence in the package I am developing.
Another route I investigated was to use the snakefood tool, which initially looks like it would do exactly what I wanted, generate a list of file dependencies for every file in a given path.
Unfortunately, this tool seems to have limited Python 3 support, making it useless for my package.
Does anyone have any insight into how to get snakefood-like features in Python 3, or is the only option to change all of my source code to only import modules?
doit tutorial itself is about creating a graph of python module imports!
It uses import_deps package, it is similar to snakefood.
Note that for your use-case you will need to modify file_dep itself during Task action's execution. To achieve that you need to pass the task parameter to your action (as described here).

How to pickle a namespace in python

Is there a way to save and load an entire namespace using pickle allowing me to perform something like this:
import pickle
import numpy as np
a = 1
# other arbitrary code
pickle.dump(namespace,open('my_namespace.p','wb')
Allowing to open a new python prompt and performing the following code:
import pickle
namespace.update(pickle.load(open('my_namespace.p','rb'))
print(np.array(a)+1)
>>2
# using the rest of the namespace
If your intention is to 'freeze' a particular code version of the modules you use I suggest taking a look at Python Virtual Environment (venv). Otherwise, as jasonharper suggested - it is very unlikely to work well (particularly with modules like NumPy which are complex amalgamates of Python code and binary libraries).
venv would help you deal with that particular problem too.
I think that I get the question. You want to be able to restore all variables and interact with a shell in a different interpreter or at a different time. The easiest way to do this is probably to use a notebook like jupyter or ipython. These allow you to save all commands run and reconstitute them. Sorry for it not being a direct answer, but I think it may "equivalently accomplish" what you specified.

Set function in Python 2.2

I have a problem regarding set in Python 2.2.
Currently I am just comparing between two lists like so:
temp3 = set(list1) - set(list2)
But it keeps prompting me that set isn't define. I used 2.7 to create the program.
Updating software is not an option sadly.
Is there an alternative I can use?
You can try third-party modules which provide the missing set functionality. For example, demset.
The simplest way to use this module is to keep it in the same directory as the program you are writing and import the desired contents like so:
from demset import set
The documentation as well as home page mention a way to use Python's built-in set (and frozenset) when available, and only use the versions in the demset module when the built-ins are not available:
try:
type(frozenset)
except NameError:
from demset import set, frozenset
Aside from those imports, your program can stay exactly the same.
Note that I mentioned keeping the demset module in the same directory as your program only because this doesn't require any installation, and if you are stuck on Python 2.2, it sounds like maybe you are not allowed to install anything.
Unless you're doing something massive you can probably just write your own function for this like the one below.
tmp3=[]
for i in list1:
if i not in list2 and i not in tmp3:
tmp3.append(i)

Categories