How to import own modules from repo on Databricks? - python

I have connected a Github repository to my Databricks workspace, and am trying to import a module that's in this repo into a notebook also within the repo. The structure is as such:
Repo_Name
Checks.py
Test.ipynb
The path to this repo is in my sys.path(), yet I still get ModuleNotFoundError: No module named 'Checks'. When I try to do import Checks. This link explains that you should be able to import any modules that are in the PATH. Does anyone know why it might still not be working?

I have tried doing the same and got a similar error even after following the procedure as given in the link provided in the question.
I have the following python files in my GIT repo (3 files with .py extension).
Now when I add the path /Workspace/Repos/<username>/repro0812 to sys.path and try to import the sample module from this repo, it throws the same error.
This is because, for some reason, this file is not being rendered as a python file. When I open the repo, you can actually see the difference.
There was no problem while I import the other 2 python modules check and sample2. The following is an image for refernce.
Check and make sure that the file is being considered as a .py file after adding your repo.

Related

Question about Colab importing your own modules

I am relatively new to Colab & python, so I don't really know how to use it. I have git-cloned a GitHub repository onto my Colab notebook. There are some modules inside the GitHub folder that I have cloned that I need to import them into the notebook. For example, there is a utils and models module that I want to import. I ran the code:
from models import *
from utils import *
However, the Colab editor wrought the error ModuleNotFoundError: No module named 'models' and ModuleNotFoundError: No module named 'utils'. This led me to realize that I haven't imported the modules from the Git Hub clone in Colab Files into Colab. I couldn't find an answer on StackOverflow, w3schools, or the Colab official sites so I
wrought my question here. If any of you have a link to a guide or a solution, please help me. Thank you!
Yes, some of the Jupyter Notebook tricks worked. I used %cd pytorch_objectdetecttrack to get to the modules.
Not an expert in Colab, but for Python import only works for Python modules. So if there's a specific script you're trying to import, you will have to call it like import folder.someModule. The folder you're trying to access should go first, then the module name.
It's the same as saying ./folder/someModule.py.
tl:dr; You have to specify the path to the module if it's not in the current working directory.

Trouble setting up libraries imported from github

I'm trying to clone a repo to my machine to test changes for a pull request.
The repo in question is a clone of pytorch, and I want to add something to one of the files to fix an issue. I figured how to clone the repo, but I can't figure out how to import the pytorch libraries when I write a test file that contains something like:
import torch
x = torch.rand(5, 3)
print(x)
Where am I supposed to create a test.py file? How do I add pytorch (specifically my cloned version of pytorch) to the list of dependencies for Python to run with? I tried just creating a test.py file at the same level as the cloned repo, but i get the error message
"no module named torch.version". I am using VS code.
I'm new to using git and not extremely familiar with the structure of libraries like this. I tried looking through github, stack overflow and the pytorch docs but was unable to find an explanation.
Make sure your current selected interpreter doesn't contain pytorch, then before import torch, add the following code:
import sys
sys.path.append("\the folder that contains pytorch\")
Please have a try.

Unable to 'relative import' a local Python library using a symbolic link

My project has two main folders: sourceCode and lib: Highlighted file tree here
I'm working in \sourceCode\mainFile.ipynb and would like to import a library residing in lib called modifiedLibrary, which has an __init__.py file.
Currently, I'm using a symbolic link for relative-importing the library. The symbolic link is located in \sourceCode and called sym_link with the following content:
../lib/modifiedLibrary/modifiedLibrary
In the project, the library and the symbolic link have the same name.
but when I import in python using
import modifiedLibrary
I receive ModuleNotFoundError: No module named 'modifiedLibrary'
I understand that the same code functions on another device that I do not have access to right now, and I do not seem to find what the issue is.
I successfully included the needed library by:
changing the working directory temporarily to where the library's __init__.py is located,
importing the library
then reverting back to my original directory
but I would like to know what the issue is with the current symbolic link.
Windows 10 / Python 3.7.3 / Jupyter
Relevant Question: Interactive Python - solutions for relative imports
The other solution I found, rather than changing the working directory temporarliy to include a local module, was to add the location of the module on my device to sys.path before importing it:
import sys
sys.path.append('C:/Users/user/myProject/../modifiedLibrary/')
import modifiedLibrary
It doesn't make use of the symbolic link but it seems to do the trick for now. Would be an issue when the code is shared and ran on another device.

Google Colaboratory - AttributeError: module X has no attribute Y

I'm trying to use the Freesound API in a Google Colaboratory notebook (running Python 3) to generate a database of sounds for which to do machine learning with. However I have been unable to use the definitions in a module I imported.
I've looked at other similar questions but they did not seem to address my issue (most were cases of trying to import a module in the standard python library and instead importing a .py file of the same name) and I apologize if this particular issue has been covered somewhere else.
The boilerplate code is as follows:
#clone relevant Git repo
!git clone https://github.com/MoltenMuffins/freesound-python
!ls
#Import packages
import os
import sys
import requests
#Open module file and import module
open('freesound.py','wb')
import freesound
There is some code after that but it is not relevant to the issue. Running this last code block is what gives me the Attribute Error despite FreesoundClient being defined in the freesound.py file cloned from the repo:
freesound_client = freesound.FreesoundClient()
I would greatly appreciate an explanation regarding this issue!
Here's a link to the colabs notebook if you'd like to take a look
I would follow the repo's instructions of using their setup.py to do the installation:
After cloning the git repo, you want to change your working directory to the freesound-python directory and run setup.py
import os
os.chdir('/content/freesound-python')
!python setup.py install
# now import the module
import freesound

Relative imports in Tensorflow Object detection API

I am trying to use the tensorflow object detection APIs for one of my personal projects. Here you can find the link to the repo for the same that needs to be cloned before using it. In this repo, there are multiple directories and are arranged in this fashion:
models
+ research
+ object_detection
+
+ ....
+ ....
Now the object_detection is the one that is of use to me. So, I create a new directory in the parent directory research for my own models and try to use the functionalities given in the object_detection directory. Within the object_detction directory is a sub-directory utils which contains a bunch of .py files for certain utilities. Here is a sample link for such a file in that directory. Now when I try to use this file in my code, it says no module found: object_detection. This is a relative import error but I am not getting how to resolve it. I just want to make sure that I can use each of the functionality without modifying the imports and this is doable because there is an __init__ file in each sub-directory but somehow this isn't working. Please help!!
As per the installation instructions, please add slim/ and models/ to your PYTHONPATH.
Just for clarification, I assume that you already installed tensorflow/models. Otherwise install it (and all its dependencies) either as explained here or in an virtual environment.
Either way you can import code from the object_detection API like so:
from object_detection import <myExample>
For your example that would be:
from object_detection.utils import label_map_util

Categories