I would like to save to a file multiple instances of a custom Python class.
The class is Loess, taken from https://github.com/joaofig/pyloess, which performs localised regression.
Below's a MWE of the saving process:
import pickle
import numpy as np
from Loess import Loess
xarr = np.linspace(0, 4, 100) * np.pi
yarr = 2*np.sin(xarr) + np.random.rand(len(xarr))
loess = Loess(xarr, yarr)
with open("localised_regression.pkl", "wb") as output:
pickle.dump(loess, output)
and now the retrieval process
import pickle
with open("localised_regression.pkl", 'rb') as input_:
localised_regression = pickle.load(input_)
When I do this on a jupyter notebook (run the first snippet on one notebook and the second on another) it works perfectly.
Dumping the instance of Loess from a notebook and retrieving it from terminal or another machine, it doesn't work.
I get ModuleNotFoundError: No module named 'Loess' error message
I even tried importing the module in python session where I attempt the retrieval, but nothing changes.
It seems that it only works from within the same location where the dumping was performed.
I'm using Python 3.7.7 and the same conda environment for both Python shell and jupyter notebook.
I examined other answers (like how to save/read class wholly in Python) but no luck.
I've tried saving to numpy file, but same story.
I've also tried dumping with marshal and json, but both complained.
Does anybody have a suggestion on how to solve this? Thank you
Related
I am using Jupyter in a Conda environment:
import igl
import meshplot as mp
import numpy as np
v, f = igl.read_triangle_mesh("./earth.ply")
k = igl.gaussian_curvature(v, f)
mp.plot(v, f, k, return_plot = True)
OUTPUT:
<meshplot.Viewer.Viewer at 0x1b53eb03fa0>
it is not displaying the mesh. it just outputs the location it stored in memory. Please help me.
It seems that you have your meshplot.rendertype set to "OFFLINE".
If you are using this code in a jupyter notebook and want to display the mesh, then just switch rendertype to "JUPYTER", by executing mp.jupyter() somewhere before your plot() command.
If you are running the code as a normal python program, you can export this View object as a HTML frame using the View.to_html() method. Then you can insert this frame into a html file and view it in a browser.
You can check out the source code for switching rendertype here, how the mp.plot function works here. The View class with the to_html method is defined here.
I am trying to do some fairly simple list manipulation, using a Jupyter notebook that calls a DLL function. I'd like my Jupyter notebook/Python code to pass in a Python list to a C++ function, which modifies the list, and then I'd like the Python code to be able to access the new list values.
I can actually read (in Jupyter) the items that were not edited by the C++ code, so there must be some issue with how I'm writing, but every example I can find looks just like my code. When I try to access the item in the list that the C++ code writes, my Jupyter kernel dies with no explanation; I've tried to run the same Python code in the terminal, and the terminal session just exits, again with no explanation.
Running on Windows 10, environment with Python 3.9.2. Here's the Python:
import os
import ctypes
import _ctypes
# Import the DLL
mydll = ctypes.cdll.LoadLibrary(*path to DLL*)
# Set up
data_in = [3,6,9]
mydll.testChange.argtypes = [ctypes.py_object]
mydll.testChange.restype = ctypes.c_float
mydll.testChange(data_in)
# Returns 0.08
After running this and closing the DLL, running data_in[1] returns 6, data_in[2] returns 9, and data_in[0] causes my kernel to die.
C code for the DLL:
float testChange(PyObject *data_out) {
Py_SetPythonHome(L"%user directory%\\anaconda3");
Py_Initialize();
PyList_SetItem(data_out, 0, PyLong_FromLong(1L));
return 0.08;
}
I can also insert a number of print statements in this code that show that I can read out all three items in the DLL both before and after the call to PyList_SetItem using calls like PyLong_AsLong(PyList_GetItem(data_out, 1)). It's not clear to me that any reference counts need changing or anything like that, but perhaps I misunderstand the idea. Any ideas you all have would be greatly appreciated.
I'm currently using a python module called petsc4py (https://pypi.org/project/petsc4py/). My main issue is that none of the typical intellisense features seems to work with this module.
I'm guessing it might have something to do with it being a C extension module, but I am not sure exactly why this happens. I initially thought that intellisense was unable to look inside ".so" files, but it seems that numpy is able to do this with the array object, which in my case is inside a file called multiarray.cpython-37m-x86_64-linux-gnu (check example below).
Does anyone know why I see this behaviour in the petsc4py module. Is there anything that I (or the developers of petsc4py) can do to get intellisense to work?
Example:
import sys
import petsc4py
petsc4py.init(sys.argv)
from petsc4py import PETSc
x_p = PETSc.Vec().create()
x_p.setSizes(10)
x_p.setFromOptions()
u_p = x_p.duplicate()
import numpy as np
x_n = np.array([1,2,3])
u_n = x_n.copy()
In this example, when trying to work with a Vec object from petsc4py, doing u_p.duplicate() cannot find the function and the suggestion is simply a repetition of the function immediately before. However, using an array from numpy, doing u_n.copy() works perfectly.
If you're compiling in-place then you're bumping up against https://github.com/microsoft/python-language-server/issues/197.
I have a script that seemed to run slow and that i profiled using cProfile (and visualisation tool KCacheGrind)
It seems that what is taking almost 90% of the runtime is the import sequence, and especially the running of the _ _ init _ _.py files...
Here a screenshot of the KCacheGrind output (sorry for attaching an image...)
I am not very familiar with how the import sequence works in python ,so maybe i got something confused... I also placed _ _ init _ _.py files in everyone of my custom made packages, not sure if that was what i should have done.
Anyway, if anyone has any hint, greatly appreciated!
EDIT: additional picture when function are sorted by self:
EDIT2:
here the code attached, for more clarity for the answerers:
from strategy.strategies.gradient_stop_and_target import make_one_trade
from datetime import timedelta, datetime
import pandas as pd
from data.db import get_df, mongo_read_only, save_one, mongo_read_write, save_many
from data.get import get_symbols
from strategy.trades import make_trade, make_mae, get_prices, get_signals, \
get_prices_subset
#from profilehooks import profile
mongo = mongo_read_only()
dollar_stop = 200
dollar_target = 400
period_change = 3
signal = get_df(mongo.signals.signals, strategy = {'$regex' : '^indicators_group'}).iloc[0]
symbol = get_symbols(mongo, description = signal['symbol'])[0]
prices = get_prices(
signal['datetime'],
signal['datetime'].replace(hour = 23, minute = 59),
symbol,
mongo)
make_one_trade(
signal,
prices,
symbol,
dollar_stop,
dollar_target,
period_change)
The function get_prices simply get data from a mongo db database, and make_one_trade does simple calculation with pandas. This never poses problem anywhere else in my project.
EDIT3:
Here the Kcache grind screen when i select 'detect cycle' option in View tab:
Could that actually mean that there are indeed circular imports in my self created packages that takes all that time to resolve?
No. You are conflating cumulative time with time spent in the top-level code of the __init__.py file itself. The top-level code calls other methods, and those together take a lot of time.
Look at the self column instead to find where all that time is being spent. Also see What is the difference between tottime and cumtime in a python script profiled with cProfile?, the incl. column is the cumulative time, self is the total time.
I'd just filter out all the <frozen importlib.*> entries; the Python project has already made sure those paths are optimised.
However, your second screenshot does show that in your profiling run, all that your Python code busied itself with was loading bytecode for modules to import (the marshal module provides the Python bytecode serialisation implementation). Either the Python program did nothing but import modules and no other work was done, or it is using some form of dynamic import that is loading a large number of modules or is otherwise ignoring the normal module caches and reloading the same module(s) repeatedly.
You can profile import times using Python 3.7's new -X importtime command-line switch, or you could use a dedicated import-profiler to find out why imports take such a long time.
I'm currently writing a python script which plots a numpy matrix containing some data (which I'm not having any difficulty computing). For complicated reasons having to do with how I'm creating that data, I have to go through terminal. I've done problems like this a million times in Spyder using imshow(). So, I thought I'd try to do the same in terminal. Here's my code:
from numpy import *
from matplotlib import *
def make_picture():
f = open("DATA2.txt")
arr = zeros((200, 200))
l = f.readlines()
for i in l:
j = i[:-1]
k = j.split(" ")
arr[int(k[0])][int(k[1])] = float(k[2])
f.close()
imshow(arr)
make_picture()
Suffice it to say, the array stuff works just fine. I've tested it, and it extracts the data perfectly well. So, I've got this 200 by 200 array of numbers floating around my RAM and I'd like to display it. When I run this code in Spyder, I get exactly what I expected. However, when I run this code in Terminal, I get an error message:
Traceback (most recent call last):
File "DATAmine.py", line 15, in <module>
make_picture()
File "DATAmine.py", line 13, in make_picture
imshow(arr)
NameError: global name 'imshow' is not defined
(My program's called DATAmine.py) What's the deal here? Is there something else I should be importing? I know I had to configure my Spyder paths, so I wonder if I don't have access to those paths or something. Any suggestions would be greatly appreciated. Thanks!
P.S. Perhaps I should mention I'm using Ubuntu. Don't know if that's relevant.
To make your life easier you can use
from pylab import *
This will import the full pylab package, which includes matplotlib and numpy.
Cheers