I have uploaded my databricks notebooks to a repo and replace %run sentences with import using the new databrick public available features (Repo integration and python import): https://databricks.com/blog/2021/10/07/databricks-repos-is-now-generally-available.html
But its seems its not working
I already activate the repo integration option in the Admin panel but i Get this error
ModuleNotFoundError: No module named 'petitions'
For simplicity I moved all python files to the same directory. I get the error in the procesado notebook
[
Please try to use full path to file from repo root:
from folders.file import class
If your are using Azure DataBricks and Python Notebooks, you can't import them as modules.
From the documentation:
If you want to import the notebook as a Python module, you must edit
the notebook in a code editor and remove the line # Databricks
Notebook source. Removing that line converts the notebook to a regular
Python file.
Documentation:
https://learn.microsoft.com/en-us/azure/databricks/repos#work-with-python-and-r-modules
Related
I have connected a Github repository to my Databricks workspace, and am trying to import a module that's in this repo into a notebook also within the repo. The structure is as such:
Repo_Name
Checks.py
Test.ipynb
The path to this repo is in my sys.path(), yet I still get ModuleNotFoundError: No module named 'Checks'. When I try to do import Checks. This link explains that you should be able to import any modules that are in the PATH. Does anyone know why it might still not be working?
I have tried doing the same and got a similar error even after following the procedure as given in the link provided in the question.
I have the following python files in my GIT repo (3 files with .py extension).
Now when I add the path /Workspace/Repos/<username>/repro0812 to sys.path and try to import the sample module from this repo, it throws the same error.
This is because, for some reason, this file is not being rendered as a python file. When I open the repo, you can actually see the difference.
There was no problem while I import the other 2 python modules check and sample2. The following is an image for refernce.
Check and make sure that the file is being considered as a .py file after adding your repo.
I have created two new python notebooks for databricks in /workspace/Shared/Notebooks.
I would share some functions between the both notebooks. I have created a python file containing a few generic functions.
It exists a way to import my functions into my both notebooks ?
Thanks for your help.
It is really easy, when it comes to sharing python code between .ipynb files.
Say, you have following files:
source.py
dest_1.ipynb
dest_2.ipynb
With following contents:
source.py
a = "hello from source"
dest_1.ipynb and dest_2.ipynb
import source
print(source.a)
And you can simply run cells in your notebooks.
The main part is that your source file should be located in the same folder with notebooks.
There are two different methods to achieve this:
Use the %run <another_notebook> to include content of another notebook into a current one (doc)
If you use Databricks Repos, it has support for so-called "Files in Repos" - in this case you can use Python or R files (not notebooks!) as Python or R modules, so for Python you can just do import some_file.
Unit tests in this demo repository shows both approaches.
When I import a custom Python package and module in my Jupyter notebook on Google Colab, the Python interpreter reports an error message indicating "ModuleNotFoundError: No module named 'utilities'."
I would like to be able to develop a Jupyter notebook that uses functions from different Python classes/modules from different Python packages that I have developed and tested.
I have simplified my Jupyter notebook that makes a function call to a sole Python module/class in a single Python package stored in the same directory on Google Drive.
The source code for the Jupyter notebook is:
import importlib.util
from google.colab import drive
drive.mount('/content/drive')
import sys
sys.path.append('/content/drive/My\ Drive/Colab\ Notebooks/utilities')
# Module to test if I can import a Python package and module.
from utilities.simple_module import simple
class Try_to_Import_Package:
number_times_executed = 0
# Accessor and Mutator method.
#staticmethod
def get_number_times_executed():
Try_to_Import_Package.number_times_executed = Try_to_Import_Package.number_times_executed + 1
print(" Try_to_Import_Package 'Hello World' function called:",Try_to_Import_Package.number_times_executed,"times.")
if __name__ == "__main__":
for x in range(10):
simple.get_number_times_executed()
Try_to_Import_Package.get_number_times_executed()
In the directory for my Google Drive hosting code for Google Colab Jupyter notebooks (My Drive -> Colab Notebooks), I have a folder named "utilities" with a Python script named "simple_module.py".
The source code for "simple_module.py" is provided as follows:
class simple:
number_times_executed = 0
# Accessor and Mutator method.
#staticmethod
def get_number_times_executed():
simple.number_times_executed = simple.number_times_executed + 1
print(" simple 'Hello World' function has been called:",simple.number_times_executed,"times.")
In the "Colab Notebooks" directory in my Google Drive, I also have a file named: "init.py".
Its contents are:
from .utilities import *
What do I need to do to be able to use modules/classes from Python packages that I created and thoroughly tested.
P/S: The question and solution in How to import custom modules in google colab? does not cover importing Python modules in embedded in Python packages.
I can import Python modules in the directory for my Google Drive hosting code for Google Colab Jupyter notebooks (My Drive -> Colab Notebooks).
However, I have problems including Python modules/classes stored in Python packages (subdirectories of the folder/directory including the Python script with the main function for the Python program.)
As of Feb,2020 you can simply link the google drive to your colab notebook.
Go to left Pane.
Select files.
Click on Mount drive.
Now, navigate to the folder where your .py module is located.
Right click on the directory and click on copy path.
Go to your colab notebook and type:
import sys
sys.path.append('your/folder/path/in/google/drive')
After this you should be able to load the modules in your colab using:
from module_name import *
Hope this helps!
I am doing some machine learning project and want to run the project on google colab since my own machine is too weak for it and hangs when i try to run the project on it. My project has the structure as shown in picture .
project structure.
I have multiple .py files each importing modules from one another. I converted the project to .zip file in my pc and then used the upload tab on google colab to upload the project. i unzipped the file and tried to run one code from the "examples" folder which is importing some function from the modAL function like this
from modAL.models import ActiveLearner.
this import is failing on google colab with error " no module named modAL " . Can someone please tell me how to get around this issue? The code works just fine on my own laptop.
I found this explanation: https://zerowithdot.com/colab-workspace/ - very useful.
After creating a space in your google drive
from os.path import join
from google.colab import drive
ROOT = "/content/drive"
drive.mount(ROOT)
fetch the git repo
GIT_USERNAME = "xxx"
GIT_TOKEN = "xxx"
GIT_REPOSITORY = "Repo"
!mkdir "{PROJECT_PATH}"
!git clone https://{GIT_TOKEN}#github.com/{GIT_USERNAME}/{GIT_REPOSITORY}.git "
{PROJECT_PATH}"
Finally use importlib to get access to definitions
from importlib.machinery import SourceFileLoader
somemodule = SourceFileLoader('somelib', join(PROJECT_PATH,
'utils/somelib.py')).load_module()
If the project is public (probably possible to make it work otherwise too) you can create a package [1] and install it with pip:
!pip install git+https://github.com/myuser/myproject
[1] https://packaging.python.org/tutorials/packaging-projects/
I don't manage to find how import custom class in Python with the Jupyter notebook in anaconda.
In my work folder I have a file 'user.ipynb' that contains a class name User. In an other file in the same folder, I try to import this class with : from user import User.
I get this error: ImportError: No module named user.
I tried to create a file ' _ _init__.py' and _ _init__.ipynb in this folder but it doesn't work.
Do you know how I could do this ?
Thank you in advance
Python modules are either files named *.py or directories with an __init__.py in them. user.ipynb is neither, it's an IPython (Jupyter) notebook, so it is not a module and you can't import it.
The easiest way to get this to work is to convert the notebook to a .py file (this can be done using the export function) and then you'll be able to import it.
An alternative is to provide code that lets you import notebooks directly; such code is available here. Looks like a lot of bother, though. I'm not sure why they didn't just bake this in; it seems like it would be useful.