I'm attempting to create a function to load Sagemaker models within a jupyter notebook using shell commands. The problem arises when I try to store the function in a utilities.py file and source it for multiple notebooks.
Here are the contents of the utilities.py file that I am sourcing in a jupyter lab notebook.
def get_aws_sagemaker_model(model_loc):
"""
TO BE USED IN A JUPYTER NOTEBOOK
extracts a sagemaker model that has ran and been completed
deletes the copied items and leaves you with the model
note that you will need to have the package installed with correct
versioning for whatever model you have trained
ie. if you are loading an XGBoost model, have XGBoost installed
Args:
model_loc (str) : s3 location of the model including file name
Return:
model: unpacked and loaded model
"""
import re
import tarfile
import os
import pickle as pkl
# extract the filename from beyond the last backslash
packed_model_name = re.search("(.*\/)(.*)$" , model_loc)[2]
# copy and paste model file locally
command_string = "!aws s3 cp {model_loc} ."
exec(command_string)
# use tarfile to extract
tar = tarfile.open(packed_model_name)
# extract filename from tarfile
unpacked_model_name = tar.getnames()[0]
tar.extractall()
tar.close()
model = pkl.load(open(unpacked_model_name, 'rb'))
# cleanup copied files and unpacked model
os.remove(packed_model_name)
os.remove(unpacked_model_name)
return model
The error occurs when trying to execute the command string:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_10889/996524724.py", line 1, in <module>
model = get_aws_sagemaker_model("my-model-loc")
File "/home/ec2-user/SageMaker/env/src/utilities/model_helper_functions.py", line 167, in get_aws_sagemaker_model
exec(command_string)
File "<string>", line 1
!aws s3 cp my-model-loc .
^
SyntaxError: invalid syntax
It seems like jupyter isn't receiving the command before exec checks the syntax. Is there a way around this besides copying the function into each jupyter notebook that I use?
Thank you!
You can use transform_cell method of IPython's shell to transform the IPython syntax into valid plain-Python:
from IPython import get_ipython
ipython = get_ipython()
code = ipython.transform_cell('!ls')
print(code)
which will show:
get_ipython().system('!ls')
You can use that as input for exec:
exec(code)
Or directly:
exec(ipython.transform_cell('!ls'))
A ! magic can be included in a function, but can't be performed via exec.
def foo(astr):
!ls $astr
foo('*.py')
will do the same as
!ls *.py
Related
I'm having issues deploying my Panel app to be served up as HTML.
Following instructions at https://panel.holoviz.org/user_guide/Running_in_Webassembly.html, with script.py as
import panel as pn
from mymodule import MyObj
pn.extension('terminal', 'tabulator', sizing_mode="stretch_width")
gspec = pn.GridSpec(sizing_mode='stretch_width', max_height=100, ncols=3)
PI = MyObj()
gspec[0, 0:1] = PI.tabs
gspec[0, 2] = PI.view
pn.Column(
"Text Text Text"
, gspec
).servable()
and mymodule/__init__.py:
__version__ = '0.1.0'
from .my_obj import MyObj
and mymodule/my_obj.py:
from .param_classes import ClassA
from .param_classes import ClassB
# etc
from .compute_obj import ComputeObj
class MyObj(object):
# all the details of the panel build, calling in turn
# param classes detailed in another file, and also calling another module
# to handle all the computation behind the panel
panel serve script.py --autoreload works perfectly, but
$ panel convert script.py --to pyodide-worker --out pyodide
$ python3 -m http.server
does not work. I get a big display at http://localhost:8000/pyodide/script.html: ModuleNotFoundError: no module named 'mymodule', and an infinite loop spinning graphic, and in the Developer Tools Console output:
pyodide.asm.js:10 Uncaught (in promise) PythonError: Traceback (most recent call last):
File "/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception
File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/lib/python3.10/site-packages/_pyodide/_base.py", line 506, in eval_code_async
await CodeRunner(
File "/lib/python3.10/site-packages/_pyodide/_base.py", line 359, in run_async
await coroutine
File "<exec>", line 11, in <module>
ModuleNotFoundError: No module named 'mymodule'
at new_error (pyodide.asm.js:10:218123)
at pyodide.asm.wasm:0xdef7c
at pyodide.asm.wasm:0xe37ae
at method_call_trampoline (pyodide.asm.js:10:218037)
at pyodide.asm.wasm:0x126317
at pyodide.asm.wasm:0x1f6f2e
at pyodide.asm.wasm:0x161a32
at pyodide.asm.wasm:0x126827
at pyodide.asm.wasm:0x126921
at pyodide.asm.wasm:0x1269c4
at pyodide.asm.wasm:0x1e0697
at pyodide.asm.wasm:0x1da6a5
at pyodide.asm.wasm:0x126a07
at pyodide.asm.wasm:0x1e248c
at pyodide.asm.wasm:0x1e00d9
at pyodide.asm.wasm:0x1da6a5
at pyodide.asm.wasm:0x126a07
at pyodide.asm.wasm:0xe347a
at Module.callPyObjectKwargs (pyodide.asm.js:10:119064)
at Module.callPyObject (pyodide.asm.js:10:119442)
at wrapper (pyodide.asm.js:10:183746)
I should add that I'm using poetry to manage packages and build the venv, and I'm operating all the above from within the activated .venv (via poetry shell)
I've tried all the tips and tricks around append the local path to sys.path. Looking at the .js file that the convert utility generates, I gather it would work if all the code were in one file, but forcing bad coding practice doesn't sound right.
I imagine there could be some kind of C++ build-style --include argument to panel convert, but there are no man pages, and the closest I can get with online documentation is --requirements ./mymodule, but no joy.
Could anybody please advise?
I saw a tutorial on YouTube(I can't link it because I can't find it anymore),
So the code is supposed to detect devices that are connected to my Internet/Router, I don't understand a lot about how his(The person who made the tutorial) code works
I also got this error in my console:
File "c:/Users/j/Desktop/Connection-Detection.py", line 6, in
IP_NETWORK = config('IP_NETWORK')
File "C:\Users\j\AppData\Local\Programs\Python\Python38-32\lib\site-packages\decouple.py", line 199, in call
return self.config(*args, **kwargs)
File "C:\Users\j\AppData\Local\Programs\Python\Python38-32\lib\site-packages\decouple.py", line 83, in call
return self.get(*args, **kwargs)
File "C:\Users\j\AppData\Local\Programs\Python\Python38-32\lib\site-packages\decouple.py", line 68, in get
raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: IP_NETWORK not found. Declare it as envvar or define a default value.
PS C:\Users\j\Desktop\python\login>
That's "Detection.py"
import sys
import subprocess
import os
from decouple import config
IP_NETWORK = config('IP_NETWORK')
IP_DEVICE = config('IP_DEVICE')
proc = subprocess.Popen(['ping', IP_NETWORK],stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline
if not line:
break
connected_ip = line.decode('utf-8').split()[3]
if connected_ip == IP_DEVICE:
subprocess.Popen(['say', 'Someone connected to network'])
You need to define an environment variable in same directory as the Detection.py file.
Steps
Install python-decouple - pip install python-decouple.
Create a file called .env
Open the .env file and paste the following into it.
IP_NETWORK=YOUR_IP_NETWORK
IP_DEVICE=YOUR_IP_DEVICE
Replace YOUR_IP_NETWORK and YOUR_IP_DEVICE with your IP_NETWORK and IP_DEVICE
I am trying to load a saved gensim lda mallet:
ldamallet = gensim.models.wrappers.LdaMallet(mallet_path, corpus=corpus, num_topics=n_topics,id2word=id2word)
ldamallet.save('ldamallet')
When testing this for a new query (with the original corpus and dictionary), everything seems fine for the first load.
ques_vec = [dictionary.doc2bow(words) for words in data_words_list]
for i, row in enumerate(lda[ques_vec]):
row = sorted(row, key=lambda x: (x[1]), reverse=True)
On executing the same code afterward, it is this error that pops up:
java.io.FileNotFoundException: /tmp/9f371_corpus.mallet (No such file
or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at cc.mallet.types.InstanceList.load(InstanceList.java:787)
at cc.mallet.classify.tui.Csv2Vectors.main(Csv2Vectors.java:131)
Exception in thread "main" java.lang.IllegalArgumentException:
Couldn't read InstanceList from file /tmp/9f371_corpus.mallet
at cc.mallet.types.InstanceList.load(InstanceList.java:794)
at cc.mallet.classify.tui.Csv2Vectors.main(Csv2Vectors.java:131)
Traceback (most recent call last): File "topic_modeling1.py", line
406, in
topic = get_label(text, id2word, first, ldamallet) File "topic_modeling1.py", line 237, in get_label
for i, row in enumerate(lda[ques_vec]): File "/home/user/sjha/anaconda3/envs/conda_env/lib/python3.6/site-packages/gensim/models/wrappers/ldamallet.py", line 308, in getitem
self.convert_input(bow, infer=True) File "/home/user/sjha/anaconda3/envs/conda_env/lib/python3.6/site-packages/gensim/models/wrappers/ldamallet.py", line 256, in convert_input
check_output(args=cmd, shell=True) File "/home/user/sjha/anaconda3/envs/conda_env/lib/python3.6/site-packages/gensim/utils.py",
line 1806, in check_output
raise error subprocess.CalledProcessError: Command '/home/user/sjha/projects/topic_modeling/mallet-2.0.8/bin/mallet
import-file --preserve-case --keep-sequence --remove-stopwords
--token-regex "\S+" --input /tmp/9f371_corpus.txt --output /tmp/9f371_corpus.mallet.infer --use-pipe-from
/tmp/9f371_corpus.mallet' returned non-zero exit status 1.
Contents of my /tmp/ directory:
/tmp/9f371_corpus.txt /tmp/9f371_doctopics.txt /tmp/9f371_doctopics.txt.infer /tmp/9f371_inferencer.mallet /tmp/9f371_state.mallet.gz /tmp/9f371_topickeys.txt
Also, it seems like the files /tmp/9f371_doctopics.txt.infer and /tmp/9f371_corpus.txt get modified every time I load the model. What could be the possible error source? Or is it some kind of bug in gensim's mallet wrapper?
mallet likes to store important model files (the corpus, etc) in /tmp if prefix is unset, and then when /tmp is cleared (say, by restarting) it throws a fit because it needs those files to run. deleting the model and rerunning the algorithm does not solve it- you first must reinstall gensim...
eg
conda uninstall gensim
conda install gensim
or whatever install manager you prefer.
then delete your saved models (sorry, their corpus etc are already gone...)
important: before rerunning you need to explicitly set the prefix param when initializing mallet:
prefix = {your chosen prefix dir}
if not os.path.isdir(prefix):
os.mkdir(prefix)
ldamallet = models.wrappers.LdaMallet({all your other args}, prefix=prefix, ...)
I have just installed ocropus OCR with all dependencies in my windows 7 machine. (I am using 32bit python 2.7) It seems to be working fine except that I cannot load the default OCR model: en-default.pyrnn.gz. , and receiving a Traceback. I am using the following syntax:
python ocropus-rpred -m en-default.pyrnn.gz book\0001\*.png
here is the error
INFO: #inputs47
# loading object /usr/local/share/ocropus/en-default.pyrnn.gz
Traceback (most recent call last):
File "ocropus-rpred" line 109, in <module>
network = ocrolib.load_object(args.model,verbose=1)
File "C:\anaconda32\lib\site-packages\ocrolib\common.py", line 513, in load_object
return unpickler.load()
EOFError
I have checked the file is not empty; also double checked the binary mode flag enabled i.e. "wb" and "rb"; also converted the newlines of common.py using dos2unix. I am being unable to unable to solve this problem. If anyone have expereinced similar issues, kindly share.
import cPickle
import gzip
def save_object(fname,obj,zip=0):
if zip==0 and fname.endswith(".gz"):
zip = 1
if zip>0:
# with gzip.GzipFile(fname,"wb") as stream:
with os.popen("gzip -9 > '%s'"%fname,"wb") as stream:
cPickle.dump(obj,stream,2)
else:
with open(fname,"wb") as stream:
cPickle.dump(obj,stream,2)
def unpickle_find_global(mname,cname):
if mname=="lstm.lstm":
return getattr(lstm,cname)
if not mname in sys.modules.keys():
exec "import "+mname
return getattr(sys.modules[mname],cname)
def load_object(fname,zip=0,nofind=0,verbose=0):
"""Loads an object from disk. By default, this handles zipped files
and searches in the usual places for OCRopus. It also handles some
class names that have changed."""
if not nofind:
fname = ocropus_find_file(fname)
if verbose:
print "# loading object",fname
if zip==0 and fname.endswith(".gz"):
zip = 1
if zip>0:
# with gzip.GzipFile(fname,"rb") as stream:
with os.popen("gunzip < '%s'"%fname,"rb") as stream:
unpickler = cPickle.Unpickler(stream)
unpickler.find_global = unpickle_find_global
return unpickler.load()
else:
with open(fname,"rb") as stream:
unpickler = cPickle.Unpickler(stream)
unpickler.find_global = unpickle_find_global
return unpickler.load()
UPDATE: Hi, please note that I have used Python's native gzip, and it is working fine. Thank you for pointing that out. Here is the correct syntax that is working on Windows: {with gzip.GzipFile(fname,"rb") as stream:}
Your use of gunzip (in the load_object function) is incorrect. Unless passed the -c argument, gunzip writes the decompressed data to a new file, not to its stdout (which is what you seem to be attempting to do).
As a result, it doesn't write anything to its stdout, and your stream variable contains no data, hence the EOFError.
A quick fix is to change your gunzip command line to give it the -c argument.
More info here: http://linux.die.net/man/1/gzip
That said, why are you even shelling out to gunzip to decompress your data? Python's built-in gzip module should handle that without problems.
I am working on creating a WebGL interface for which I am trying to convert FBX models to JSON file format in an automated process using python file, convert_fbx_three.py (from Mr. Doob's GitHub project) from command line.
When I try the following command to convert the FBX:
python convert_fbx_three.py Dolpine.fbx Dolpine
I get following errors:
Error in cmd:
Traceback (most recent call last):
File "convert_fbx_three.py", line 1625, in <module>
sdkManager, scene = InitializeSdkObjects()
File "D:\xampp\htdocs\upload\user\fbx\FbxCommon.py", line 7, in InitializeSdkObjects
lSdkManager = KFbxSdkManager.Create()
NameError: global name 'FbxManager' is not defined
I am using Autodesk FBX SDK 2012.2 available here on Windows 7.
Can you please try the following:
import FbxCommon
.
.
.
lSdkManager, lScene = FbxCommon.InitializeSdkObjects()
You probably need to add environment variables pointing to the folder that contains fbx.pyd, FbxCommon.py, and fbxsip.pyd prior to calling anything in those modules.