I'd like to include code from another file in another Jupyter .ipynb file on the AWS Elastic MapReduce platform.
But many of the methods I have seen online for including utility functions / common Python code that would seem to work outside of AWS, don't work inside the hosting environment for EMR-based Notebooks. I'm assuming this is file system security/server restrictions. If someone knows of an example of including code from a .py file and/or .ipynb file from the same directory that works on AWS I would love to see an example.
This method did not work for me. The find_notebook returns None.
https://jupyter-notebook.readthedocs.io/en/4.x/examples/Notebook/rstversions/Importing%20Notebooks.html
or this library / method
ipynb import another ipynb file
Is there an AWS "approved" or recommended way of including common Python/PySpark code into a Jupyter Notebook?
Note: this question
How to import from another ipynb file in EMR jupyter notebook which runs a PySpark kernel?
I have ssh'd into the master server and installed different packages like ipynb.
Didn't work even though the module installed fine. And the environment can see it. But the overall technique did not work.
Error:
An error was encountered:
"'name' not in globals"
Traceback (most recent call last):
KeyError: "'name' not in globals"
Related
I am running a python script with bcp to export the data from my local drive to SQL Server. The python script works well when I run it manually on Jupyter notebook. But, when I create a batch file to automatically run the task, I am getting an error.
Below is the batch file:
#echo off
"C:\ProgramData\Anaconda3\python.exe" "C:\Users\Atom\Desktop\Untitled1.ipynb" %*
pause
And the error that I get while running the batch file:
Traceback (most recent call last):
File "C:\Users\Atom\Desktop\Untitled1.ipynb", line 40, in <module>
"execution_count": null,
NameError: name 'null' is not defined
Kindly advise what could be the issue.
According to the IPython web site (emphasis mine):
Notebook documents contains the inputs and outputs of a interactive
session as well as additional text that accompanies the code but is
not meant for execution. In this way, notebook files can serve as a
complete computational record of a session, interleaving executable
code with explanatory text, mathematics, and rich representations of
resulting objects. These documents are internally JSON files and are
saved with the .ipynb extension.
I use the RemoteFS extension in VSCode to connect to my remote SSH server. When I open a .py file on the remote server in VSCode, and then add #%% comment to the .py file, I don't get the option to run a Jupyter cell like I would locally.
Has anybody gotten VSCode's Python extension working with it's built-in Jupyter support and the RemoteFS extension?
We had an overly restrictive file check for when we allowed our "Run Cell" commands to show up and it was limiting it to local files only. I've fixed that issue in the following PR here:
https://github.com/Microsoft/vscode-python/pull/4191
I verified that I was seeing the cell commands using Remote FS after that. Sadly this just missed the cutoff for our recent January release, so it won't show up in the extension until later in February. If you want to check out the fix you can access our daily development build here:
https://github.com/Microsoft/vscode-python/blob/master/CONTRIBUTING.md#development-build
That build has the fix already, but it's not the full public release.
I'm using Jupyter Notebooks within Google CloudDatalab in Google Cloud Platform for my Python scripts. This creates .ipynb files.
I want to be able to create Modules and Packages using Jupyter Notebooks but it is not able to import the scripts.
For e.g.:
mymodule.ipynb has this:
def test_function():
print("Hello World")
Then in myscript.ipynb when I try to import the above:
from mymodule import test_function
It throws an error :
*ImportErrorTraceback (most recent call last) in ()
----> 1 from mymodule import test_function ImportError: No module named mymodule*
How do I create Modules & Packages using Jupyter Notebooks?
Notebooks can't be used as modules. You need to create a python file (e.g. mymodule.py).
If you want you can do this from within a jupyter notebook:
with open('mymodule.py', 'w') as f:
f.write('def test_function():')
f.write(' print("Hello World")')
from mymodule import test_function
test_function()
# Hello World
You cannot import Jupyter Notebook in the same way as Python files (packages).
Check this link: https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Importing%20Notebooks.html
I've received a response on a different forum which answered my question. Here is the answer:
Put your module code into a .py file and make sure that's in the same directory as myscript.ipynb.
To create the .py file from the code currently in a notebook, you can download it from within Jupyter as a .py file and tidy it up in your text editor of choice.
Remember, the import statement is pure python syntax. It knows nothing about - and has no wish to know anything about - jupyter notebooks. It's looking for a .py file.
This has resolved the issue.
Thanks a lot.
I am following this tutorial to install the notebook on google cloud. It shows a wrong user interface and I cannot see blocks when I edit the file and cannot run either.
The mistake is that I open a .py file directly. Instead, we should create a new notebook by clicking new.
When I try to run any of my app engine projects by python GoogleAppEngineLauncher
I got the error log as follows:
Does anyone have any ideas of what's going on?
I tried remove the SDK and reinstall it. Nothing happens. Still got the same error.
Everything is working fine and I don't think I made any changes before this happens.
The only thing that I can think of is that I install bigquery command line tool before this happens. But I don't think this should be the reason of this.
bad runtime process port ['']
Traceback (most recent call last):
File
"/Users/txzhang/Documents/App/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/_python_runtime.py",
line 197, in
_run_file(file, globals()) File "/Users/txzhang/Documents/App/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/_python_runtime.py",
line 193, in _run_file
execfile(script_path, globals_) File "/Users/txzhang/Documents/App/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/runtime.py",
line 175, in
main() File "/Users/txzhang/Documents/App/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/runtime.py",
line 153, in main
sandbox.enable_sandbox(config) File "/Users/txzhang/Documents/App/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py",
line 159, in enable_sandbox
import('%s.threading' % dist27.name) File "/Users/txzhang/Documents/App/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py",
line 903, in load_module
raise ImportError('No module named %s' % fullname) ImportError: No module named google.appengine.dist27.threading
The most probable reason is having another python package that is coming from google. Run
python in verbose mode with python -vvvvv and try following command import google.
If the above import is successful make sure its coming from /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google which is the path of python google appengine libraries on my system (OSX, fresh install of google appengine sdk).
If that is not the case (but the import is successful) then its most likely other existing google libraries which are creating issue and the path of the same would be visible on python prompt after the successful import google command like following:
>>> import google
import google # loaded from Zip
/Library/Python/2.7/site-packages/protobuf-2.4.1-py2.7.egg/google/init.pyc
In this case my google protobuf package was the culprit.
Solution:
Use virtualenv: If you haven't used python virtualenv before, may be this is the best time to use it. (a) Make sure that your PYTHONPATH variable is not set to include anything that has a google package! Unset it running unset PYTHONPATH (in your bash prompt) unless you are very sure what you have there. (b) Create a new python virtualenv, activate it and try to run google appengine commands in it:
virtualenv --no-site-packages /tmp/googleapps
source /tmp/googleapps/bin/activate
dev_appserver.py path_to_google_app
Remove conflicting packages from path: Move conflicting packages a new virtualenv. This is likely to break other stuff so not recommended.
I had hit the same issue today. This being the top result on google but lacked any answers, I add this after fixing same issue on my system. There may be other possible reasons which thankfully I haven't come across.
Good luck!
A recent upgrade of the development SDK started causing this problem for me. After much turmoil, I found that the problem was that the SDK was in a sub-directory of my project code. When I ran the SDK from a different (parent) directory the error went away.