I am doing some machine learning project and want to run the project on google colab since my own machine is too weak for it and hangs when i try to run the project on it. My project has the structure as shown in picture .
project structure.
I have multiple .py files each importing modules from one another. I converted the project to .zip file in my pc and then used the upload tab on google colab to upload the project. i unzipped the file and tried to run one code from the "examples" folder which is importing some function from the modAL function like this
from modAL.models import ActiveLearner.
this import is failing on google colab with error " no module named modAL " . Can someone please tell me how to get around this issue? The code works just fine on my own laptop.
I found this explanation: https://zerowithdot.com/colab-workspace/ - very useful.
After creating a space in your google drive
from os.path import join
from google.colab import drive
ROOT = "/content/drive"
drive.mount(ROOT)
fetch the git repo
GIT_USERNAME = "xxx"
GIT_TOKEN = "xxx"
GIT_REPOSITORY = "Repo"
!mkdir "{PROJECT_PATH}"
!git clone https://{GIT_TOKEN}#github.com/{GIT_USERNAME}/{GIT_REPOSITORY}.git "
{PROJECT_PATH}"
Finally use importlib to get access to definitions
from importlib.machinery import SourceFileLoader
somemodule = SourceFileLoader('somelib', join(PROJECT_PATH,
'utils/somelib.py')).load_module()
If the project is public (probably possible to make it work otherwise too) you can create a package [1] and install it with pip:
!pip install git+https://github.com/myuser/myproject
[1] https://packaging.python.org/tutorials/packaging-projects/
Related
I have connected a Github repository to my Databricks workspace, and am trying to import a module that's in this repo into a notebook also within the repo. The structure is as such:
Repo_Name
Checks.py
Test.ipynb
The path to this repo is in my sys.path(), yet I still get ModuleNotFoundError: No module named 'Checks'. When I try to do import Checks. This link explains that you should be able to import any modules that are in the PATH. Does anyone know why it might still not be working?
I have tried doing the same and got a similar error even after following the procedure as given in the link provided in the question.
I have the following python files in my GIT repo (3 files with .py extension).
Now when I add the path /Workspace/Repos/<username>/repro0812 to sys.path and try to import the sample module from this repo, it throws the same error.
This is because, for some reason, this file is not being rendered as a python file. When I open the repo, you can actually see the difference.
There was no problem while I import the other 2 python modules check and sample2. The following is an image for refernce.
Check and make sure that the file is being considered as a .py file after adding your repo.
I am relatively new to Colab & python, so I don't really know how to use it. I have git-cloned a GitHub repository onto my Colab notebook. There are some modules inside the GitHub folder that I have cloned that I need to import them into the notebook. For example, there is a utils and models module that I want to import. I ran the code:
from models import *
from utils import *
However, the Colab editor wrought the error ModuleNotFoundError: No module named 'models' and ModuleNotFoundError: No module named 'utils'. This led me to realize that I haven't imported the modules from the Git Hub clone in Colab Files into Colab. I couldn't find an answer on StackOverflow, w3schools, or the Colab official sites so I
wrought my question here. If any of you have a link to a guide or a solution, please help me. Thank you!
Yes, some of the Jupyter Notebook tricks worked. I used %cd pytorch_objectdetecttrack to get to the modules.
Not an expert in Colab, but for Python import only works for Python modules. So if there's a specific script you're trying to import, you will have to call it like import folder.someModule. The folder you're trying to access should go first, then the module name.
It's the same as saying ./folder/someModule.py.
tl:dr; You have to specify the path to the module if it's not in the current working directory.
When I import a custom Python package and module in my Jupyter notebook on Google Colab, the Python interpreter reports an error message indicating "ModuleNotFoundError: No module named 'utilities'."
I would like to be able to develop a Jupyter notebook that uses functions from different Python classes/modules from different Python packages that I have developed and tested.
I have simplified my Jupyter notebook that makes a function call to a sole Python module/class in a single Python package stored in the same directory on Google Drive.
The source code for the Jupyter notebook is:
import importlib.util
from google.colab import drive
drive.mount('/content/drive')
import sys
sys.path.append('/content/drive/My\ Drive/Colab\ Notebooks/utilities')
# Module to test if I can import a Python package and module.
from utilities.simple_module import simple
class Try_to_Import_Package:
number_times_executed = 0
# Accessor and Mutator method.
#staticmethod
def get_number_times_executed():
Try_to_Import_Package.number_times_executed = Try_to_Import_Package.number_times_executed + 1
print(" Try_to_Import_Package 'Hello World' function called:",Try_to_Import_Package.number_times_executed,"times.")
if __name__ == "__main__":
for x in range(10):
simple.get_number_times_executed()
Try_to_Import_Package.get_number_times_executed()
In the directory for my Google Drive hosting code for Google Colab Jupyter notebooks (My Drive -> Colab Notebooks), I have a folder named "utilities" with a Python script named "simple_module.py".
The source code for "simple_module.py" is provided as follows:
class simple:
number_times_executed = 0
# Accessor and Mutator method.
#staticmethod
def get_number_times_executed():
simple.number_times_executed = simple.number_times_executed + 1
print(" simple 'Hello World' function has been called:",simple.number_times_executed,"times.")
In the "Colab Notebooks" directory in my Google Drive, I also have a file named: "init.py".
Its contents are:
from .utilities import *
What do I need to do to be able to use modules/classes from Python packages that I created and thoroughly tested.
P/S: The question and solution in How to import custom modules in google colab? does not cover importing Python modules in embedded in Python packages.
I can import Python modules in the directory for my Google Drive hosting code for Google Colab Jupyter notebooks (My Drive -> Colab Notebooks).
However, I have problems including Python modules/classes stored in Python packages (subdirectories of the folder/directory including the Python script with the main function for the Python program.)
As of Feb,2020 you can simply link the google drive to your colab notebook.
Go to left Pane.
Select files.
Click on Mount drive.
Now, navigate to the folder where your .py module is located.
Right click on the directory and click on copy path.
Go to your colab notebook and type:
import sys
sys.path.append('your/folder/path/in/google/drive')
After this you should be able to load the modules in your colab using:
from module_name import *
Hope this helps!
I'm trying to use the Freesound API in a Google Colaboratory notebook (running Python 3) to generate a database of sounds for which to do machine learning with. However I have been unable to use the definitions in a module I imported.
I've looked at other similar questions but they did not seem to address my issue (most were cases of trying to import a module in the standard python library and instead importing a .py file of the same name) and I apologize if this particular issue has been covered somewhere else.
The boilerplate code is as follows:
#clone relevant Git repo
!git clone https://github.com/MoltenMuffins/freesound-python
!ls
#Import packages
import os
import sys
import requests
#Open module file and import module
open('freesound.py','wb')
import freesound
There is some code after that but it is not relevant to the issue. Running this last code block is what gives me the Attribute Error despite FreesoundClient being defined in the freesound.py file cloned from the repo:
freesound_client = freesound.FreesoundClient()
I would greatly appreciate an explanation regarding this issue!
Here's a link to the colabs notebook if you'd like to take a look
I would follow the repo's instructions of using their setup.py to do the installation:
After cloning the git repo, you want to change your working directory to the freesound-python directory and run setup.py
import os
os.chdir('/content/freesound-python')
!python setup.py install
# now import the module
import freesound
I am currently working on a project on Machine Learning on Google Colab which uses Tensorflow API. I created a folder and uploaded it on google drive to run on google Colab.
I successfully mounted the google drive and can run the script,
But when I try importing another module from the script present in the same folder, it throws an error
from . import inference
ImportError: cannot import name 'inference'
I tried finding out a solution for this but found results stating how to import a module directly to the colab notebook.
Please tell me what I am missing here.
Edit:
The folder structure is
-nmt
-nmt.py
-train.py
-inference.py
-utils
-evaluation.py
and so on.
And I am running the python file from the nmt folder. I am getting relative import errors.
Try this. First, upload the .py file and then save it locally. Then you may use it as you like it.
from google.colab import files
src = list(files.upload().values())[0]
open('library_you_want_to_use.py','wb').write(src)
import library_you_want_to_use