Is there a possibility to run a Jupyter Notebook in one environment and than to call a .py file (out of the JN) from another environment without pulling it over like it normally occurs?
Example:
from PythonScript1 import FunctionFromScript
Edit:
Because I see my problem is unclear described here some further details and the background of my question:
I want to run a matlab file from a jupyter notebook but this only works on condition which does not allow me to use tensorflow in the same JN (Using Matlab.engine and installing tensorflow at the same time).
My idea was to have the tensorflow model in one .py file which works in an anaconda env. (+ other directory) which is designed for it, while I have an JN in an other anaconda environment to call the matlab code.
You can also use SOS kernels in Jupyter Lab. SOS allows you to run multiple kernels in the same notebook and pass variables between the kernels. I was able to run Python and R kernels in a single notebook using SOS. You can use two Python kernels in your case - one with TF and one without.
P.S. I am not affiliated to SOS and am not promoting it. It worked for me and I thought I'd suggest this option.
No, it is not possible because you can't have two interpreters on the same notebook. Actually, you can have two virtual environments and execute the notebook with one or other, but you can't do it with both.
If you are talking about running a module crafted with other version of python interpreter, it depends on the versions compatibility
I found a solution to my problem. If I build my (.py) script as a Flask, then I can run it in a different environment (+ dir.) than my Jupyter Notebook. The only difference is that I can't call the function directly, I have to access the server and import my data with "get" and "post". Thanks for the help everyone!
Related
I am in the process of building a Python package that can be used by Data Scientists to manage their MLOps lifecycle. Now, this package can be used either locally (usually on PyCharm) or on Databricks.
I want a certain functionality of the package to be dependent on where it is running, i.e. I want it to do something different if it is called by a Databricks notebook and something else entirely if it is running locally.
Is there any way I can determine where it is being called from?
I am a little doubtful as to whether we can use something like the following that checks if your code is running on a notebook or otherwise since this will be a package that is going to be stored in your Databricks environment,
How can I check if code is executed in the IPython notebook?
The workaround I've found to work is to check for databricks specific environment variables.
import os
def is_running_in_databricks() -> bool:
return "DATABRICKS_RUNTIME_VERSION" in os.environ
I have been working in Jupyter Notebook/Python for a while now and want to modularize a lot of my scripts, so that I can just call them going forward and have data populate inside or outside of Jupyter.
import mynewdatatable
mynewdatatable.spinon()
-> Wonderful new table
In addition to modularizing my scripts, I'm also trying to get more comfortable in Visual Studio.
Where do I write my code in VS so that it can take advantage of all the modules in Conda and where do I save the modules I create so that they can be used by other scripts?
Also, any good resources on managing/understanding directories would be greatly appreciated.
Additional Details
OS: Windows
I am working in a jupyter notebook with the irkernel.
When I do the same with a Python kernel I usually have at the top of my notebooks the instructions
%load_ext autoreload
%autoreload 2
If I modify a source code that the notebook imports or uses then those functions and pieces of code in the notebook that use that source code will be automatically updated.
Is there an equivalent for this in a jupyter R notebook?
I am using a local package that my notebook uses. I would like to be able to edit the package and have the modifications automatically loaded in my notebook.
In short? Unless jupyter does something that is impossible while working in base R the answer is "No". R cannot dynamically load packages for editing in a similar fashion to how Python does it. The recommended method in R is to modify, install and often run R CMD check. I am not sure how Jupyter implements these, but this is the approach that is also focused upon in the user experience of Rstudio.
Hadley has great (free!) book on how to develop packages in R. I am almost certain he mentions this workflow somewhere in the "Getting started" section.
Is it possible to visualize data in pycharm in the same way that you can do in jupyter?
For instance in jupyter you can run a single line at a time and see the output which can be helpful for working with data sets.
Of course you can just use a function like head() or show() to see what is going on but you have to run the whole file (as opposed to one line at a time in jupyter) and it can make working with data a bit harder to understand.
Does anyone have any recommendations for me in terms of pycharm as that is what I am most familar with, or do you think it is worth me learning something like jupyter?
Why don't you use Atom and install the package Hydrogen? it offers you the same possibilities as Jupyter while working in a script (not a notebook).
You can execute the code line by line like in Jupyter by clicking ctrl+Enter or run it as a script. Here's the documentation of the package.
Atom is a light IDE so it combines both the power of Jupyter and PyCharm. I have used it and it is great and has so many packages like hydrogen, pep8 (helps to write a code that is conform to the pep8) and code beautifiers (for Python, R, JSON,etc), and a lot of great features.
I recommend learning Jupyter. Although PyCharm is an excellent IDE, running Jupyter in it looks clunky and something like this:
I'm trying to use fastai's libraries, but some of the data accessing tools built in to those libraries are dependent on HTML objects. For example, the DataBunch.show_batch attribute produces an HTML object that is easiest to use in Jupyter. I need to run my testing on GCP (or another cloud), and here are the issues:
fastai has some libraries that are dependent on Python3.6 or greater (new string format)
GCP doesn't have a good way to interface with Jupyter NBs. I had it set up with some trouble, but then my computer needed reformat, and now I am questioning if I should set up again. The previous method was largely based on this.
GCP apparently has something meant to provide an interface between it and Jupyter NBs, called Datalab. However, Datalab does not support Py3.6 or greater, per this link.
I see a few options:
Develop my own data visualization techniques by subclassing fastai's libraries and skip Jupyter altogether
Create a Jupyter-to-GCP interface in a different way, basically redoing the steps in link in the second bullet point above.
Use one of the containers (docker) that I keep hearing about on Datalab that allow me to use my own version of Python
Does anyone have other options for how I can make this connection? If not, can anyone provide other links for how to accomplish 1, 2, or 3?
You can follow this guide from fast.ai to create a VM with all the required libraries pre-installed. Then, following the same guide you can access JupyterLab or Jupyter Notebooks in this VM. It's simple, fast and comes with Python 3.7.3.
You could create a notebook using GCP's AI Platform Notebooks.
It should give you a one-click way to create a VM with all the libraries you need preinstalled. It'll even give you a URL that you can use to directly access your notebook.