Is there a way to save and load an entire namespace using pickle allowing me to perform something like this:
import pickle
import numpy as np
a = 1
# other arbitrary code
pickle.dump(namespace,open('my_namespace.p','wb')
Allowing to open a new python prompt and performing the following code:
import pickle
namespace.update(pickle.load(open('my_namespace.p','rb'))
print(np.array(a)+1)
>>2
# using the rest of the namespace
If your intention is to 'freeze' a particular code version of the modules you use I suggest taking a look at Python Virtual Environment (venv). Otherwise, as jasonharper suggested - it is very unlikely to work well (particularly with modules like NumPy which are complex amalgamates of Python code and binary libraries).
venv would help you deal with that particular problem too.
I think that I get the question. You want to be able to restore all variables and interact with a shell in a different interpreter or at a different time. The easiest way to do this is probably to use a notebook like jupyter or ipython. These allow you to save all commands run and reconstitute them. Sorry for it not being a direct answer, but I think it may "equivalently accomplish" what you specified.
Related
Imagine I use a pretty standard module, like numpy, and that I have a custom version locally.
Currently, I would store my custom numpy in a folder, say
.../MyPackages/
which I added to sys.path in second position (after ''). This way, any package that I use and which calls numpy, will use my numpy by default. When I don't want to use my numpy as default, I rename .../MyPackages/numpy to .../MyPackages/_numpy. This way, I can still call it manually with
import _numpy
if I want to, but it's not the default behaviour anymore.
This way of storing custom packages works pretty well for me so far, but doesn't allow me to call the usual numpy (not my custom version) without renaming the folder first, which is a bit heavy...
Is there a cleaner/better way to organize my packages to add this flexibility?
Thanks in advance.
_ I described above what I tried._
OK, prerequisites:
It's my first ever Python project. I used to do some scripting but never anything bigger
So I'm at the very beginning of a learning curve. It's like when you can't kill an ant in Fallout 2 Temple level. On later levels, I was really good in Fall 2:)
Problem:
I can't figure out how to import a module written by me and placed in a different folder
Context:
The project I'm intended to create is meant to do a lot of measures conversions. So I decided to store in DB all data in the same unit system & keep all conversions upon user preferences on a codebase level
In a different folder I decided to store tests. To write the very first one (testing the abovementioned module) I need to import the module, but here is the story begins. I know it's classic, but I'm completely messed with import
Toolkit:
PyCharm Pro (PyCharm 2021.3.1)
Python 3.7 interpreter
macOS 10.15, Intell
Set up:
Settings screenshot provided
Project structure. Folders are marked as Source & Test
I need to import from conversions.py to test_conversions.py
PYTHONPATH settings like this
What do I, for the sake of God, need:
with all the abovementioned, how do I import conversions.py to test_conversions.py or any other place of my project? I read a number of articles and it's getting me anywhere (contradictory, 2.x related, etc). I feel like I need a piece of more foundational info but as well I need a clear walkthrough, a code snippet to import bloody file, I really appreciate any kind of advice
imports are a bit tricky. The issue you have is where your python is looking for packages. I would discourage you to add to your PYTHONPATH a specific project but you could do that locally in your file.
A much easier way is just to launch your test file from the top directory.
In this case your import will just be import conversion.conversion
And then you can launch your test from the root folder with python -m tests.conversion.
In Pycharm you can use the interface to deal with that, see this link. But I like the python -m because it works from anywhere, not only inside Pycharm.
make a class inside a conversion.py, then you can import it from test_conversion.py.
conversion.py
class convert():
def km_to_mm(input):
output = input * 1000000
return output
then import it in test_conversion.py
input = 0.001 # specify your input value
from conversion import convert
converted = convert().a_to_b(input)
converted will have value 1000
make sure you use the same folder. otherwise should use folder name to import. like
import foldername.conversion
from foldername.conversion import convert
I really appreciate all of you who tried to help. I got the problem solved in a very ridiculous manner, below is to someone who might face the same issue in the future.
So, in case you see the next:
You use PyCharm (no idea how other IDEs behave)
You created a module & want to import it into other files of your project
You type import module_name
While you type it, the string looks active and autocomplete even proposes you your module name but as only you finished typing, the import string turns grey, PyCharm throws you a warning saying Unused import statement, yellow bulb next to the import string suggests you delete the import string
--> This does not mean you are not able to import your module, it means you've done it and now can call anything from your module in the code below.
This taught me to pay some more time to read docs before jumping to using anything new and think better about UX in anything I do.
I want to inject some code to all python modules in my project automatically.
I used ast.NodeTransformer and managed to change it quite easily, the problem is that I want to run the project.
An ast node is per module and I want to change all modules in the project and then run; and I have this example
The problem is that it applies to one node, viz. one file. I want to run this file, which imports and uses other files which I want to change too, so I'm not sure how to get it done.
I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
Don't really know how to start, any suggestions?
I know I can use some ast-to-code module, like astor, but all are third party and I don't want to deal with bugs and unexpected issues.
From 3.9 onward there is ast.unparse, which practically does AST to source conversion after you transform it.
I am trying to do a bit of math not covered under the numpy or scipy packages. Since I have to do these calculations many times, I thought I'd create my own function in my own mathystuff.py module and import it into my workspace. Easy enough, except python doesn't like importing the function! Here's what I did:
First, I created my file mathystuff.py which is located in my venv/lib/python3.7/site-packages/ folder like the rest of my modules.
import numpy as np
def mathfunction(input1, input2):
#do some math calculations here - lots of np.stuff()
#and some more math calculations here, you get it
return answer
Simple as that. In my main project, I simply call:
from mathystuff import mathfunction
where I'm met with the error cannot import name 'mathfunction' from 'mathystuff' pointing me to the correct location of the .py file. I've done this a million times before and even done this with other modules I've made in the same project, so why would this happen?
There can be many reasons, but according to limited information provided I will go with the ones I think are most possible. (PS I would've commented to ask questions but unfortunately I don't have enough rep).
Check if:
Your virtual env is activated. (I know it sounds silly but it is possible to forget activating venv)
If it's active, then check the location of python by typing which python into terminal (or which python3 if you are running your code with python3). This will return the directory python (eg: path/to/folder/venv/bin/python3). While working on multiple venvs, it's easy to get confused and work on venvA while venvB is active.
If all above checks, go into venv/lib/ directory and confirm that you have only one python directory there. Like venv/lib/python3.8 or venv/lib/python3.7. If, for some reason, there are two (or more) python versions in venv/lib/ then make sure the one you use is the one your module is located in.
These are the most common errors I came across so far.
I am interested in using doit to automate the build process of a python package.
If possible, I would like doit to re-execute a task if any of the user-created source files it depends on have changed.
From my understanding, the best way to accomplish this would be to use the file_dep key and a list of the dependent source files, however I am having a lot of trouble generating this list.
I've tried using sys.modules and inspect.getmembers(), but these solutions can't deal with import statements that do not import a module, such as from x import Y, which is unfortunately a common occurrence in the package I am developing.
Another route I investigated was to use the snakefood tool, which initially looks like it would do exactly what I wanted, generate a list of file dependencies for every file in a given path.
Unfortunately, this tool seems to have limited Python 3 support, making it useless for my package.
Does anyone have any insight into how to get snakefood-like features in Python 3, or is the only option to change all of my source code to only import modules?
doit tutorial itself is about creating a graph of python module imports!
It uses import_deps package, it is similar to snakefood.
Note that for your use-case you will need to modify file_dep itself during Task action's execution. To achieve that you need to pass the task parameter to your action (as described here).