We Keep Coding

Jupyter shell commands in a function

I'm attempting to create a function to load Sagemaker models within a jupyter notebook using shell commands. The problem arises when I try to store the function in a utilities.py file and source it for multiple notebooks.
Here are the contents of the utilities.py file that I am sourcing in a jupyter lab notebook.
def get_aws_sagemaker_model(model_loc):
"""
TO BE USED IN A JUPYTER NOTEBOOK
extracts a sagemaker model that has ran and been completed
deletes the copied items and leaves you with the model
note that you will need to have the package installed with correct
versioning for whatever model you have trained
ie. if you are loading an XGBoost model, have XGBoost installed
Args:
model_loc (str) : s3 location of the model including file name
Return:
model: unpacked and loaded model
"""
import re
import tarfile
import os
import pickle as pkl
# extract the filename from beyond the last backslash
packed_model_name = re.search("(.*\/)(.*)$" , model_loc)[2]
# copy and paste model file locally
command_string = "!aws s3 cp {model_loc} ."
exec(command_string)
# use tarfile to extract
tar = tarfile.open(packed_model_name)
# extract filename from tarfile
unpacked_model_name = tar.getnames()[0]
tar.extractall()
tar.close()
model = pkl.load(open(unpacked_model_name, 'rb'))
# cleanup copied files and unpacked model
os.remove(packed_model_name)
os.remove(unpacked_model_name)
return model
The error occurs when trying to execute the command string:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_10889/996524724.py", line 1, in <module>
model = get_aws_sagemaker_model("my-model-loc")
File "/home/ec2-user/SageMaker/env/src/utilities/model_helper_functions.py", line 167, in get_aws_sagemaker_model
exec(command_string)
File "<string>", line 1
!aws s3 cp my-model-loc .
^
SyntaxError: invalid syntax
It seems like jupyter isn't receiving the command before exec checks the syntax. Is there a way around this besides copying the function into each jupyter notebook that I use?
Thank you!

You can use transform_cell method of IPython's shell to transform the IPython syntax into valid plain-Python:
from IPython import get_ipython
ipython = get_ipython()
code = ipython.transform_cell('!ls')
print(code)
which will show:
get_ipython().system('!ls')
You can use that as input for exec:
exec(code)
Or directly:
exec(ipython.transform_cell('!ls'))

A ! magic can be included in a function, but can't be performed via exec.
def foo(astr):
!ls $astr
foo('*.py')
will do the same as
!ls *.py

Pdb error while debugging python app with serverless launched offline

I have problem with debugging my view function with
import pdb; pdb.set_trace()
placed inside it and serverless launched as
> sls offline start
in console.
Namely, making correspondent GET request I receive the following error:
Python: > /.../handler.py(88)get_results()
-> request_params = event.query_params
Python: (Pdb)
Python: 2019-02-20 18:37:43,648 [ERROR] | ...
Traceback (most recent call last):
...
File ".../handler.py", line 88, in get_results
...
File "/usr/lib/python3.6/bdb.py", line 51, in trace_dispatch
return self.dispatch_line(frame)
File "/usr/lib/python3.6/bdb.py", line 70, in dispatch_line
if self.quitting: raise BdbQuit
bdb.BdbQuit
Google suggests that the problem is in the inability of serverless process to read from stdin, but I don't know how to handle this problem.
Any suggestions?

I found a solution here https://stackoverflow.com/a/26975795/4388451:
create two fifos:
mkfifo fifo_stdin
mkfifo fifo_stdout
in one terminal
In the same terminal open stdout on background, and write to stdin:
cat fifo_stdout & cat > fifo_stdin
In python code create the the pdb object, and use it:
import pdb
mypdb = pdb.Pdb(stdin=open('fifo_stdin', 'r'), stdout=open('fifo_stdout', 'w'))
....
mypdb.set_trace()
Run python code from the folder where fifos were placed (or place fifos in the first step in the folder with python code) in another terminal
Now I am able to use pdb in first console!
PS
It is useful to use --noTimeout option while debugging: sls offline --noTimeout

App Engine python local unit test urlfetch_stub.py fancy_urllib module has no attribute 'InvalidCertificateException'

I'm using linux with google-cloud-sdk-app-engine-python package installed version 196.0.0-0.
I try to create local unit test, following reference from [1].
class ViewsTestCase(unittest.TestCase):
def setUp(self):
self.client = application.test_client()
self.client.testing = True
self.testbed = testbed.Testbed()
self.testbed.activate()
self.testbed.init_urlfetch_stub()
def tearDown(self):
self.testbed.deactivate()
add PYTHONPATH environment variable:
export PYTHONPATH="$PYTHONPATH:/usr/lib/google-cloud-sdk/platform/google_appengine:/usr/lib/google-cloud-sdk/platform/google_appengine/lib:/usr/lib/google-cloud-sdk/platform/google_appengine/lib/yaml/lib"
then I try to run it.
$ python test_views.py
Traceback (most recent call last):
File "test_views.py", line 6, in <module>
from google.appengine.ext import testbed
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/testbed/__init__.py", line 130, in <module>
from google.appengine.api import urlfetch_stub
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch_stub.py", line 32, in <module>
_fancy_urllib_InvalidCertException = fancy_urllib.InvalidCertificateException
AttributeError: 'module' object has no attribute 'InvalidCertificateException'
I have read this similar thread [2].
But I checked, the path of fancy_lib has been fixed.
$ find /usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/__init__.pyc
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/fancy_urllib
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.pyc
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/__init__.py
I have tried to add fancy_urllib to PYTHONPATH
export PYTHONPATH="$PYTHONPATH:/usr/lib/google-cloud-sdk/platform/google_appengine:/usr/lib/google-cloud-sdk/platform/google_appengine/lib:/usr/lib/google-cloud-sdk/platform/google_appengine/lib/yaml/lib:/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib"
But still failed.
any clue?
thank you.
[1]https://cloud.google.com/appengine/docs/standard/python/tools/localunittesting
[2]GAE SDK 1.9.5 and an InvalidCertificateException

If you see the file length of init.py is '0' and if there is sub folder with the same name (i.e. lib/fancy_urllib/fancy_urllib), then copy both files (.py, and .pyc) of the sub folder to upper folder. That worked for me.

Getting "global name 'FbxManager' is not defined"

I am working on creating a WebGL interface for which I am trying to convert FBX models to JSON file format in an automated process using python file, convert_fbx_three.py (from Mr. Doob's GitHub project) from command line.
When I try the following command to convert the FBX:
python convert_fbx_three.py Dolpine.fbx Dolpine
I get following errors:
Error in cmd:
Traceback (most recent call last):
File "convert_fbx_three.py", line 1625, in <module>
sdkManager, scene = InitializeSdkObjects()
File "D:\xampp\htdocs\upload\user\fbx\FbxCommon.py", line 7, in InitializeSdkObjects
lSdkManager = KFbxSdkManager.Create()
NameError: global name 'FbxManager' is not defined
I am using Autodesk FBX SDK 2012.2 available here on Windows 7.

Can you please try the following:
import FbxCommon
.
.
.
lSdkManager, lScene = FbxCommon.InitializeSdkObjects()

You probably need to add environment variables pointing to the folder that contains fbx.pyd, FbxCommon.py, and fbxsip.pyd prior to calling anything in those modules.

Create and use settings module in Python

I am new to Python so bear with me. I use the pyDev plugin fore eclipse. There are three files:
tool.py:
from gui import Tool_Window
import wx
import settings
if __name__ == '__main__':
window = wx.App()
Tool_Window(None, settings.WindowHeader)
window.MainLoop()
Tool_Window.py:
from Tool import settings
import wx
class Tool_Window(wx.Frame):
def __init__(self, parent, title):
super(Tool_Window,self).__init__(parent, title = title)
self.SetDimensions(settings.WindowOpenX,
settings.WindowOpenY,
settings.WindowWidth,
settings.WindowHeight)
settings.py:
WindowHeader = 'The SuperAwesome Tool'
WindowOpenX = 500
WindowOpenY = 100
WindowWidth = 200
WindowHeight = 400
The "tool.py" file is in a package called "Tool", as is the "settings.py" file, and "Tool_Window" is in the package "gui".
I am getting error messages from a previous file I had in the project, now renamed to the "settings.py". I have tried cleaning the project in Eclipse, but nothing happens. The error message looks like:
Traceback (most recent call last):
File "/home/oystein/workspaces/python/awesome.tool/src/Tool/tool.py", line 8, in <module>
Tool_Window(None, settings.WindowHeader)
File "/home/oystein/workspaces/python/awesome.tool/src/gui/__init__.py", line 12, in __init__
# ;-)
AttributeError: class GeneralParameters has no attribute 'WindowParameters'
Previously I had a class named GeneralParameters with a sub-class WindowParameters, as I wanted to access static variables for settings. I relaized Python couldn't do it like that and changed it to the "settings.py" file.
I run the program from "tool.py"
Can anyone see what's wrong here?

You are running stale byte-code, remove the .pyc files and rerun your code.
The traceback reads the source from the .py file but is built from the bytecode, and the fact that is shows that the error is on a line that only consists of a comment is a hint that things are no longer in sync.
Normally, Python will clean up the .pyc file when stale, but only if the .py modification time is newer.