Creating and Using Modules Outside of Jupyter - python

I have been working in Jupyter Notebook/Python for a while now and want to modularize a lot of my scripts, so that I can just call them going forward and have data populate inside or outside of Jupyter.
import mynewdatatable
mynewdatatable.spinon()
-> Wonderful new table
In addition to modularizing my scripts, I'm also trying to get more comfortable in Visual Studio.
Where do I write my code in VS so that it can take advantage of all the modules in Conda and where do I save the modules I create so that they can be used by other scripts?
Also, any good resources on managing/understanding directories would be greatly appreciated.
Additional Details
OS: Windows

Related

How do I make a macOS app out of my Python program?

I've made this question because I had to go through the whole process of creating my own application using Apple's somewhat lacking documentation, and without the use of py2app. I wanted to create the whole application structure so I know exactly what was inside, as well as create an installer for it. The latter of these is still a mystery, so any additional answers with information on making a custom installer would be appreciated. As far as the actual "bundle" structure goes, however, I think I've managed to get the basics down. See the answer below.
Edit: A tutorial has been linked at the end of this answer on using PyInstaller; I don't know how much it helps as I haven't used it yet, but I have yet to figure out how to make a standalone Python application without the use of a tool like this and it may just be what you're looking for if you wish to distribute your application without relying on users knowing how to navigate their Python installations.
A generic application is really just a directory with a .app extension. So, in order to build your application, just make the folder without the extension first. You can rename it later when you're finished putting it all together. Inside this main folder will be a Contents folder, which will hold everything your application needs. Finally, inside Contents, you will place a few things:
Info.plist
MacOS
Resources
Frameworks
Here you can find some information on how to write your Info.plist file. Basically, this is where you detail information about your application.
Inside the MacOS you want to place your main executable. I'm not sure that it matters how you write it; at first, I just had a shell script that called python3 ./../Resources/MyApp.py. I didn't think this was very neat though, so eventually I called the GUI from a Python script which became my executable (I used Tkinter to build my application's GUI, and I wrote several modules which I will get to later). So now, my executable was a Python script with a shebang pointing to the Python framework in my application's Frameworks folder, and this script just created an instance of my custom Tk() subclass and ran the mainloop. Both methods worked, though, so unless someone points out a reason to choose one method over the other, feel free to pick. The one thing that I believe is necessary, is that you name your executable the SAME as your application (before adding the .app). That, I believe, is the only way that MacOS knows to use that file as your application's executable. Here is a source that describes the bundle structure in more detail; it's not a necessary read unless you really want to get into it.
In order to make your executable run smoothly, you want to make sure you know where your Python installation is. If you're like me, the first thing you tried doing on your new Mac was open up Terminal and type in python3. If this is the case, this prompted you to install the Xcode Command Line tools, which include an installation of Python 3.8.2 (most recent on Xcode 12). Then, this Python installation would be located at /usr/bin/python3, although it's actually using the Python framework located at
/Applications/Xcode.app/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3
I believe, but am NOT CERTAIN, that you could simply make a copy of this framework and add it to your Frameworks folder in order to make the app portable. Make a copy of the Python3.framework folder, and add it to your app's Frameworks folder. A quick side note to be wary of; Xcode comes packaged with a lot of useful tools. In my current progress, the tool I am most hurting for is the Fortran compiler (that I believe comes as a part of GCC), which comes with Xcode. I need this to build SciPy with pip install scipy. I'm sure this is not the only package that would require tools that Xcode provides, but SciPy is a pretty popular package and I am currently facing this limitation. I think by copying the Python framework you still lose some of the symlinks that point to Xcode tools, so any additional input on this would be great.
In any case, locate the Python framework that you use to develop your programs, and copy it into the Frameworks folder.
Finally, the Resources folder. Here, place any modules that you wrote for your Python app. You also want to put your application's icon file here. Just make sure you indicate the name of the icon file, with extension, in the Info.plist file. Also, make sure that your executable knows how to access any modules you place in here. You can achieve this with
import os
os.chdir('./../Resources')
import MyModules
Finally, make sure that any dependencies your application requires are located in the Python framework site-packages. These will be located in Frameworks/Python3.framework/Versions/3.X.Y/lib/python3.x.y/site-packages/. If you call this specific installation of Python from the command line, you can use path/to/application/python3 -m pip install package and it should place the packages in the correct folder.
P.S. As far as building the installer for this application, there are a few more steps needed before your application is readily downloaded. For instance, I believe you need to use the codesign tool in order to approve your application for MacOS Gatekeeper. This requires having a developer license and manipulating certificates, which I'm not familiar with. You can still distribute the app, but anyone who downloads it will have to bypass the security features manually and it will seem a bit sketchy. If you're ready to build the installer (.pkg) file, take a look at the docs for productbuild; I used it and it works, but I don't yet know how to create custom steps and descriptions in the installer.
Additional resources:
A somewhat more detailed guide to the anatomy of a macOS app
A guide I found, but didn't use, on using codesign to get your app past Gatekeeper
A RealPython tutorial I found on using PyInstaller to build Python-based applications for all platforms

Want to automatically generate Eclipse project files

I am using the Tasking IDE (based on Eclipse IDE) for my C/C++ project on Windows 10.
I would like to automatically generate the eclipse project files (.cproject and .project) to perform continuous deployment, but the solution is not clear to me.
I think I could generate the project files using Python but I did no manage to find documentation on how those files are structured (schema).
Alternatively, is there a command line tool which I could use to generate those files ? I am wondering if eclipsec could be used for such a task.
I could as well generate the makefiles directly( without generating the project files at all). But I need the project files for debugging purposes.
Thank in advance. Since I am not an Eclipse/Java expert, any help would be greatly appreciated.

Using external modules in a Pycharm project

I am new to Pycharm and need some help. I am working on a project that makes use of a large library of modules (specifically, Schrodinger; which allows for a lot of cool chemistry programs). Schrodinger requires the use of Python 2.7, if that makes any difference.
There are too many modules to install to the project directory. When I move the project directory to the location of the modules, my script becomes stuck on 'initializing'. I have attempted to import it as a package to no avail.
I have also tried to use the sys.path command, however a lot of the modules make use of other modules as well. So I that has become a pain very quickly.
How can I use these modules within Pycharm? And if there is no easy way, do you have a recommendation for an IDE that does have this feature?
Thanks
Pycharm doesn't identifies user defined modules which are not imported to Pycharm.
I usually mask the module as a Sources Root see the picture for more details. if the modules are in same project.
Alternative way: In your case import the external modules using File -> Open modules with open -> open in current window -> add to currently opened project this looks like two different projects. Now you can mark Sources Root for the complete module (i.e. learning) which you have imported.
import stackoverflow
Now pycharm can identifies the user defined modules.

How to use both Jupyter Notebook and Sublime for Data Science projects?

I'm learning some Data Science and for that I'm using Python with Jupyter Notebook. Which I think it's great for data analysis, mainly because it’s super easy to run step-by-step code. You can see everything that is happening.
On the other hand, to do more complex projects, like a web crawler or an Object Oriented program to extract information from an API, I’m using Sublime Text3. IMO it’s simple, clean, light… perfect. Also I think that .py is better than .ipynb for that (I don't even know if it's possible to do OO with Jupyter).
My problem now is integrating these two tools. The best I can do now is convert the dictionnaires in some .csv file and read it manually in Jupyter notebook. Obviously it doesn't sounds very smart and it is like a temporary solution just for experimentation.
This is the first time I'm dealing with a project which I need to integrate more than one environment and not only working with the same languages with all the files in the same folder etc. so I'm not very familiar on how to approach that.
If someone could explain the right way of integrating these two IDEs, how to make all the process more 'automatic', if it's better to use some database and then extract with SQL or something like that I'd appreciate very much.
PS: Also, if you guys have any material on how should a Python Data Science project be organized it would be awesome. Thanks!
I use ipython magic commands to help me switch between a text editor and a ipython notebook.
Specifically, I like to experiment with the code in the Notebook for the reasons you mentioned, and then when I'm ready to integrate it as a class in a bigger system I use the %%writefile filename.py command which will export that cell into a .py file.
You can also use %load filename.py and %run myfile.py to bring .py files into the notebook.

PTVS: How to reference or use Python source code in one project from a second project

In Visual Studio with PTVS I have two separate Python projects, one contains a Python source file named lib.py for use as a library of functions and the other is a main that uses the functions in the library. I am using an import statement in the main to reference the functions in the library project but get the following error:
No module named lib
I primarily program in F# using Visual Studio so my mindset is adding references to other .NET projects.
How do I think in the Pythonic way to accomplish this?
Python does not use references like .NET does but uses a path which is searched. The search path needs to be modified to include the directory containing the source file. See: The Module Search Path
Looking at the project in Visual Studio with Solution Explorer shows Search Paths for each project.
To modify the search path:
Get the directory for the Python file containing the source code to import.
e.g. lib.py
In Solution Explorer right click on lib.py and select Copy Path
Now for the project that will import the module
e.g. ConsoleDriver_Python
Right click Search Paths and select Add Folder to Search Path...
which displays a select folder dialog
Right click and paste in the path from the clipboard. Also change it to a directory by removing the file name.
Click Select Folder
Now check the project to make sure Search Path was updated.
The import error should now be cleared.
I just wanted to add the below in addition to the verified answer, for a very specific scenario.
I was recently asked to fix the same problem that the OP was experiencing for a work machine, which had recently had the user accounts migrated over to a new domain.
Setup:
Visual Studio 2013
PTVS 2.2.30718
Anaconda 3.5
Basically, Anaconda was installed for localmachine/UserA.
Once the users were migrated over to the new domain (newdomain/UserA), the Python environment had to be updated from within VS2013, by clicking View > Other Windows > Python Environments.
Once that was setup, the python scripts would run as expected, although none of the Search Folder references would work. They were then removed and re-added but to no avail.
Various other things were tried, including setting up totally fresh projects, and linking them using the Search Paths, but to no avail.
The only thing that fixed the problem was to reinstall the Python Environment (in my case Anaconda3) outside of a user account (by clicking the "for all users, using administrator privileges" option during the install).
Then I restarted, removed and re-added the search folders, and the python worked as expected, including all the search paths.
I hope that helps someone, as I just wasted hours solving it...
D :)
Or you can do this in code with the following:
sys.path.append("search path")
So that the code can be run outside the IDE.

Categories