We Keep Coding

Call and run a jupyter notebook from Excel using xlwings?

In the company I work at, we have just started experimenting with the migration of several computation-heavy projects from Excel to Python. In this process, we have discovered xlwings and the power it can bring to the integration of Excel and Python.
Several of our projects include reading in input data from Excel worksheets, doing some background calculations, and then outputting the results to different sheets in the same workbook. From the
example projects on the xlwings website, I know it's possible to replace VBA macros (which we used so far) with Python, while keeping the Excel front-end.
However, my co-workers, who are primarily financial experts and not programmers, really like the interactivity of jupyter notebooks and would appreciate it if they could work in them during the modeling phase of the projects (rather than switching to PyCharm all of a sudden).
All in all, the ideal workflow would look like this for us:
Inputting from Excel sheets, doing some modeling and model calibration in Python through jupyter notebooks, running some tests, then if we're at a final stage, then outputting to Excel. A key constraint is that the end-users of our models are used to the VBA-like functionality (eg. Run buttons in Excel).
So my question is the following:
Is it possible to call and run a jupyter notebook from Excel as it was a .py file (ie. through the RunPython function)? This way, I assume that we could avoid the intermediate step of "converting" the models from .ipynb to .py, not to mention having two code versions of the same model.
Thank you for any suggestions!

Look here: https://github.com/luozhijian/jupyterexcel with using 'def'. That creates functions callable in Jupyter.

Thanks for the replies!
We started experimenting with nbconvert as #Wayne advised, and basically wrote a small wrapper .py code that can be called in the Excel macro via RunPython and that runs the specified notebook as-is. In the notebook, interaction between Excel and jupyter (eg. reading parameter data in from Excel and outputting values from jupyter to Excel) is handled by xlwings in turn.
I'm not sure it is the most optimal solution concerning execution speed though. However, in our case it works just fine and so far we haven't experienced any additional overhead that would hinder user experience.

This is possible with xlOil (disclaimer: I wrote it). The relevant docs are here.
The summarised procedure is: install xloil
pip install xloil
xloil install
Then edit %APPDATA%\xloil\lxloil.ini to add a reference to xloil.jupyter. Then in an Excel cell type
=xloJpyConnect("MyNotebook.ipynb")
Where you have "MyNotebook.ipynb" loaded in a local jupyter kernel. Then you can execute code on the kernel with:
=xloJpyRun(<result of connect func>, "{} + {} * {}", A1, B1, C1)
Or watch a global variable in the kernel with:
=xloJpyWatch(<result of connect func>, "variable_to_watch")
Or you can add a decorator #xloil.func to functions defined in the kernel to make them appear as Excel worksheet functions.

It sounds like this is what you want https://towardsdatascience.com/python-jupyter-notebooks-in-excel-5ab34fc6439
PyXLL and the pyxll-jupyter package allows you to run a Jupyter notebook inside of Excel and seamlessly interact with Excel from Python. You can also write Python functions in the notebook and call them from Excel as a user defined function or macro.
From the article:
"""
It used to be an “either/or” choice between Excel and Python Jupyter Notebooks. With the introduction of the PyXLL-Jupyter package now you can use both together, side by side.
In this article I’ll show you how to set up Jupyter Notebooks running inside Excel. Share data between the two and even call Python functions written in your Jupyter notebook from your Excel workbook!
"""
If the above link doesn't work you can also find it here https://www.pyxll.com/blog/python-jupyter-notebooks-in-excel/

Visualize data in Pycharm in the same way as in Jupyter?

Is it possible to visualize data in pycharm in the same way that you can do in jupyter?
For instance in jupyter you can run a single line at a time and see the output which can be helpful for working with data sets.
Of course you can just use a function like head() or show() to see what is going on but you have to run the whole file (as opposed to one line at a time in jupyter) and it can make working with data a bit harder to understand.
Does anyone have any recommendations for me in terms of pycharm as that is what I am most familar with, or do you think it is worth me learning something like jupyter?

Why don't you use Atom and install the package Hydrogen? it offers you the same possibilities as Jupyter while working in a script (not a notebook).
You can execute the code line by line like in Jupyter by clicking ctrl+Enter or run it as a script. Here's the documentation of the package.
Atom is a light IDE so it combines both the power of Jupyter and PyCharm. I have used it and it is great and has so many packages like hydrogen, pep8 (helps to write a code that is conform to the pep8) and code beautifiers (for Python, R, JSON,etc), and a lot of great features.

I recommend learning Jupyter. Although PyCharm is an excellent IDE, running Jupyter in it looks clunky and something like this:

Can python import the SPSS and SPSSAux libraries and use them to any value outside of the spss context?

I'm helping my wife try and navigate IBM SPSS and python. She knows SPSS, and I kinda know python -- We might be able to work together. As it stands, I understand that I can call small snippets of python from within an SPSS syntax. While this is useful for looping and conditional branching based on data, it seems a little fuzzy to me. It almost feels like Inversion of Control, but not really.
I was wondering is it possible to have a python script, external to an spss syntax, that can still use the SPSS libraries in any meaningful way, or do I have to keep my scripts confined to the SPSS syntax and runtime?

Yes, you can run Statistics in external mode from a Python or R program. You might have to add the SPSS Python directory to your Python search path, but then just do
import spss
and run your Python code. The only thing you can't do is Viewer and user interface stuff, because there is no SPSS UI in that mode. By default, you will get output as text (which you can turn off when you get the hang of things). If you want better quality output, you can use OMS to capture output in a wide variety of formats.
Note that you need a compatible version of Python if you don't use the one installed with SPSS. That would be 2.7 for most Statistics versions. The Python installed with Statistics is not registered, but you can install a standard version from Python.org and just add the SPSS Python directory to the search path.
HTH

Why autocompletion options in Spyder 3.1 are not fully working in the Editor?

Running on Mac Sierra, the autocompletion in Spyder (from Anaconda distribution), seems quite erratic. When used from the Ipython console, works as expected. However, when used from the editor (which is my main way of writing), is erratic. The autocompletion works (i.e. when pressing TAB a little box appears showing options) for some modules, such as pandas or matplotlib. So writing 'pd.' and hitting TAB, gets the box with options as expected. However, this does not happen with many other objects: for example, after defining a dataframe named 'df', typing 'df.' TAB shows nothing. In the Ipython console, 'df.' TAB would show the available procedures for that dataframe, such as groupby, and also its columns, etc..
So the question is threefold. First, is there any particular configuration that should be enabled to get this to work? I don't think so, given some time spent googling, but just wanna make sure. Second, could someone state what is the official word on what works and what doesn't in terms of autocompletion (e.g. what particular modules do work from the editor, and which ones doesn't?). Finally, what are the technical aspects of the differences between the editor and the Ipython console in the performance of the autocompletion with Spyder? I read something about Jedi vs. PsychoPy modules, so got curious (however, please keep in mind that although I have scientific experience, I am relatively new to computation, so please keep it reasonably simple for an educated but not expert person).
UPDATE: As a side question, it would be great to know why is the autocompletion better in Rodeo (another IDE). It is more new, has way fewer overall options than Spyder, but the autocompletion works perfectly in the editor.

(Spyder developer here)
My answers:
is there any particular configuration that should be enabled to get this to work?
In Spyder 3.1 we added the numpydoc library to improve completions of some objects (like Matplotlib figures and NumPy arrays). If Dataframe completions are not working for you (they are for me), please open an issue in our issue tracker on Github to track and solve this problem.
could someone state what is the official word on what works and what doesn't in terms of autocompletion (e.g. what particular modules do work from the editor, and which ones doesn't?)
The most difficult part is getting completions of definitions when an object is generated by functions or methods developed in C/C++/Fortran and not in Python. I mean, things like
import numpy as np
a = np.array([])
a.<TAB>
As I said, this should be working now for arrays, figures and dataframes, but it doesn't work for all libraries (and most scientific Python libraries are created in C/C++/Fortran and wrapped in Python for speed).
The problem is that the completion libraries we use (Rope and Jedi) can't deal with this case very well because array (for example) can't be introspected in a static way (i.e. without running code involving it). So we have to resort to tricks like analyzing array's docstring to see its return type and introspect that instead.
what are the technical aspects of the differences between the editor and the Ipython console in the performance of the autocompletion with Spyder?
The most important difference is that in the IPython console you have to run your code before getting completions about it. For example, please run this in a fresh IPython console
In [1]: import pandas as pd
...: df = pd.Da<Tab>
and you will see that it won't return you any completions for Da (when it obviously should return Dataframe).
But, after evaluation, it is quite simple to get completions. You can simply run
dir(pd)
to get them (that's what IPython essentially does internally).
On the other hand, Spyder's Editor doesn't have a console to run code into, so it has to get completions by running static analysis tools in your code (like Jedi and Rope). As I said, they introspect your code without running it. While they work very well for pure Python code, they have the problems I described above for compiled libraries.
And trying to evaluate the code you have in the Editor to get completions is usually not a good idea because:
It is not necessarily valid Python code all the time. For example, suppose you left an unclosed parenthesis somewhere, but you want to get completions at some other point. That should work without problems, right?
It could involve a very costly computation (e.g. loading a huge CSV in a Dataframe), so evaluating it every time to get completions (and that's a must because your code is different every time you ask for completions) could consume all your RAM in a blink.
it would be great to know why is the autocompletion better in Rodeo (another IDE)
Last time I checked (a couple of years ago), Rodeo evaluated your code to get completions. However, we'll take a look at what they are doing now to see if we can improve our completion machinery.

Autocompletion works correctly if there are NO white spaces in the project working directory path.

Autocomplete was not working for me at all.
So, I tried Tools -> Reset Sypder to factory defaults and it worked.

how to use IPython

I could not understand the ipython library. This url provide the common feature but I could not core-relate it. http://ipython.org/ipython-doc/stable/interactive/tutorial.html
How to I use IPython to improve my day to day python application experience?

ipython is an improved interactive prompt, not a library. It has features like tab completion and profiles which are not present in the vanilla interactive prompt (which is running python without an input file). All features are listed on the page you cited.
So, it doesn't really improve your day to day python application experience (whatever that means), but it does provide benefits during development.
Also, there is an alternative, called bpython, it has quite great features, too.

do you do any scientific computing? ipython 0.12 has new functionality called ipython notebook that's really useful for data analysis. you can easily print graphs and data inline in your browser and reload your code. You can then print it as a pdf and make a nice report.
it's also useful for learning python due to ipython's functionality. you can quickly test and understand how certain functions operate. A few particularly useful functionality aside from tab completion
object?? gives you more information about the object
%history gives you a list of all your previous commands
%debug if you hit an error, this will put you into the debugger so you can quickly debug