We Keep Coding

How to import a module into another module in databricks notebook?

This is my config.py in Databricks
DATA_S3_LOCATION='s3://server-data/data1'
DATA_S3_FILE_TYPE='orc'
DATA2_S3_LOCATION='s3://server-data/data2'
DATA2_S3_FILE_TYPE='orc'
I have init . py in this folder as well
I am trying to access these variables in another file
import sys
sys.path.insert(1,'/Users/file')
from file import config
I am facing error , no module named file

There are several aspects here.
If these files are notebooks, then you need to use %run ./config to include notebook from the current directory (doc)
if you're using Databricks Repos and arbitrary files support is enabled, then your code needs to be a Python file, not notebook, and have correct directory layout with __init__.py, etc. In this case, you can use Python imports. Your repository directory will be automatically added into a sys.path and you don't need to modify it.
P.S. I have an example of repository with both notebooks & Python files approaches.

Jupyter Class in different notebook?

i am currently forced in an project to work with jupyter notebook, before I just used PyCharm.
So I create a project and would like to add some classes to it. Should I write all of them in the one main notebook or create different notebooks for each class?
As far as I understand, jupyter is not made for using several notebooks for one project (but I thnink there are options) but if I want to use some of my classes more general for other projects it does not make sense to not have it in a seperate notebook right?
I am sorry, maybe I have a basic misunderstanding of jupyter but I could not figure out yet, what is the best way.

Jupyter will happily allow you to use multiple libraries in other notebooks. You can find the documents here
Though if you want to move the project out to a command line executable, you may want to consider exporting the notebooks as .py files and then importing them as standard python libraries. Jupyter has a nbconvert method for this.
I do this a lot and ended up writing a small script that adds appropriate hashbang lines and strips out any Jupyter "magic" commands. You can find it at github.
Usage: ./nbconvert myNotebook.ipynb
There are also some excellent Jupyter magic commands that make working with libraries much easier.
# cause chagnged modules to be reloaded at execution
# add this to any notebook that sources an external .py file
%load_ext autoreload
%autoreload 2
I also find this useful:
# set %magic alias to convert the current notebook into a .py file
%alias nbconvert ~/devtools/nbconvert thisNotebook.ipynb
Usage:
%nbconvert
[NbConvertApp] Converting notebook searchTools.ipynb to python

You don't necessarily need notebooks, you may create .py files with your class implementations for example myclass.py, and then just normally do import myclass

Downloaded .py file not working in Jupyter

I'm new to programming and now taking the course MIT 6.0001. On problem set 2, they set a .py file, accompanied with a .txt file.
I ran .py file in Spyder with no difficulties (.txt file embedded just fine).
But when I run .py file in Jupyter, things won't run (as in screenshot).
I want to use Jupyter because it's much clearer than Spyder, and can give headers to cells to keep myself on track. Please help me to run things in Jupyter.
Here are link to the problem set (ZIP file)
File not running in Jupyter

Use
%load filename.py
to import the text of your python file into a cell in a Jupyter notebook and then save that notebook.
Jupyter isn't built to run/edit .py files, it's built to run/edit jupyter notebooks (ipython notebooks) (.ipynb)

Where can I put a startup script in Jupyter?

I'm looking for a way to put my startup script in Jupyter. In IPython, I put it under the $IPYTHON_DIR/profile_default/startup/.
In Jupyter, it seems that the config file should be $JUPYTER_CONFIG_DIR/jupyter_notebook_config.py. However, I would like to use my startup file, which import a slew of Python libraries at the launch of the kernel.
Where can I put such file in Jupyter?

You can get the default startup script folder via this in jupyter notebook:
get_ipython().profile_dir.startup_dir
On my Windows PC, the folder is: C:\Users\myusername\.ipython\profile_default\startup
And do read the README file under the folder.
Files will be run in lexicographical order, so you can control
the execution order of files with a prefix, e.g.::
00-first.py
50-middle.py
99-last.ipy
So you can put a file named 00-xx.py under the folder.

Import own .py files in anaconda spyder

I've written my own mail.py module in spider (anaconda). I want to import this py file in other python (spider) files just by 'import mail'
I searched on the internet and couldn't find a clearly solution.

To import any python script, it should exist in the PYTHONPATH. You can check this with the following code:
import sys
print sys.path
To import your Python script:
Put both the scripts (main and the imported python script) in the
same directory.
Add the location of the file to be imported to the
sys.path.
For example, if the script is located as '/location/to/file/script.py':
import sys
sys.path.append('/location/to/file/')
import script

I had the same problem, my files were in same folder, yet it was throwing an error while importing the "to_be_imported_file.py".
I had to run the "to_be_imported_file.py" seperately before importing it to another file.
I hope it works for you too.

Searched for an answer for this question to. To use a .py file as a import module from your main folder, you need to place both files in one folder or append a path to the location. If you storage both files in one folder, then check the working directory in the upper right corner of the spyder interface. Because of the wrong working directory you will see a ModuleNotFoundError.

There are many options, e.g.
Place the mail.py file alongside the other python files (this works because the current working dir is on the PYTHONPATH.
Save a copy of mail.py in the anaconda python environment's "/lib/site-packages" folder so it will be available for any python script using that environment.

I did a slightly different solution approach that is less sophisticated. When I start my anaconda terminal it is at a C prompt. I just did a cd d:\mypython\lib in the beginning window before starting python. once I did that I could simply just import my own classes that I put in that library with "import MyClass as my" then I was off and running. It is interesting, I did 2 days of internet searching in my part time and could not find the answer either, until I asked a friend.
cd d:\mypython\lib
python
>>> import MyClass as my
>>> my1=my.MyClass()
>>> my1.doSomething()
worked for me on my anaconda / windows 10 environment python 3.6.6.

when calling any function from another file, it should be noted to not import any library inside the function

I believe the easiest solution is to place the directory containing your Python files into the Anaconda site-packages folder on your machine. I wrote an article outlining the whole process but, in short, you'll need to create a folder containing your python script and an __init__.py file. Then place that folder inside the site-packages folder in the Anaconda directory.
On Windows the site-packages directory is typically located at:
C:\Users\[your_username]\Anaconda3\Lib\site-packages\
On Mac the site-packages directory is typically located located at:
Users/[your_username]/opt/anaconda3/lib/[python3.8]/site-packages/
Notice, on Mac the Python version matters. You'll need to look for the directory that corresponds to the (base) Python version used by Anaconda. Also, anything I've placed inside of square brackets in the file paths above need to be changed according to your particular machine and Python version.
The file structure should look something like this:
~/
|__site-packages/
|__your_folder/
script.py
__init__.py
After you have the folder containing your script.py and __init__.py file moved into the site-packages subdirectory of Anaconda, you'll be able to import it from any script you run on your machine.