We Keep Coding

How do I make a macOS app out of my Python program?

I've made this question because I had to go through the whole process of creating my own application using Apple's somewhat lacking documentation, and without the use of py2app. I wanted to create the whole application structure so I know exactly what was inside, as well as create an installer for it. The latter of these is still a mystery, so any additional answers with information on making a custom installer would be appreciated. As far as the actual "bundle" structure goes, however, I think I've managed to get the basics down. See the answer below.

Edit: A tutorial has been linked at the end of this answer on using PyInstaller; I don't know how much it helps as I haven't used it yet, but I have yet to figure out how to make a standalone Python application without the use of a tool like this and it may just be what you're looking for if you wish to distribute your application without relying on users knowing how to navigate their Python installations.
A generic application is really just a directory with a .app extension. So, in order to build your application, just make the folder without the extension first. You can rename it later when you're finished putting it all together. Inside this main folder will be a Contents folder, which will hold everything your application needs. Finally, inside Contents, you will place a few things:
Info.plist
MacOS
Resources
Frameworks
Here you can find some information on how to write your Info.plist file. Basically, this is where you detail information about your application.
Inside the MacOS you want to place your main executable. I'm not sure that it matters how you write it; at first, I just had a shell script that called python3 ./../Resources/MyApp.py. I didn't think this was very neat though, so eventually I called the GUI from a Python script which became my executable (I used Tkinter to build my application's GUI, and I wrote several modules which I will get to later). So now, my executable was a Python script with a shebang pointing to the Python framework in my application's Frameworks folder, and this script just created an instance of my custom Tk() subclass and ran the mainloop. Both methods worked, though, so unless someone points out a reason to choose one method over the other, feel free to pick. The one thing that I believe is necessary, is that you name your executable the SAME as your application (before adding the .app). That, I believe, is the only way that MacOS knows to use that file as your application's executable. Here is a source that describes the bundle structure in more detail; it's not a necessary read unless you really want to get into it.
In order to make your executable run smoothly, you want to make sure you know where your Python installation is. If you're like me, the first thing you tried doing on your new Mac was open up Terminal and type in python3. If this is the case, this prompted you to install the Xcode Command Line tools, which include an installation of Python 3.8.2 (most recent on Xcode 12). Then, this Python installation would be located at /usr/bin/python3, although it's actually using the Python framework located at
/Applications/Xcode.app/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3
I believe, but am NOT CERTAIN, that you could simply make a copy of this framework and add it to your Frameworks folder in order to make the app portable. Make a copy of the Python3.framework folder, and add it to your app's Frameworks folder. A quick side note to be wary of; Xcode comes packaged with a lot of useful tools. In my current progress, the tool I am most hurting for is the Fortran compiler (that I believe comes as a part of GCC), which comes with Xcode. I need this to build SciPy with pip install scipy. I'm sure this is not the only package that would require tools that Xcode provides, but SciPy is a pretty popular package and I am currently facing this limitation. I think by copying the Python framework you still lose some of the symlinks that point to Xcode tools, so any additional input on this would be great.
In any case, locate the Python framework that you use to develop your programs, and copy it into the Frameworks folder.
Finally, the Resources folder. Here, place any modules that you wrote for your Python app. You also want to put your application's icon file here. Just make sure you indicate the name of the icon file, with extension, in the Info.plist file. Also, make sure that your executable knows how to access any modules you place in here. You can achieve this with
import os
os.chdir('./../Resources')
import MyModules
Finally, make sure that any dependencies your application requires are located in the Python framework site-packages. These will be located in Frameworks/Python3.framework/Versions/3.X.Y/lib/python3.x.y/site-packages/. If you call this specific installation of Python from the command line, you can use path/to/application/python3 -m pip install package and it should place the packages in the correct folder.
P.S. As far as building the installer for this application, there are a few more steps needed before your application is readily downloaded. For instance, I believe you need to use the codesign tool in order to approve your application for MacOS Gatekeeper. This requires having a developer license and manipulating certificates, which I'm not familiar with. You can still distribute the app, but anyone who downloads it will have to bypass the security features manually and it will seem a bit sketchy. If you're ready to build the installer (.pkg) file, take a look at the docs for productbuild; I used it and it works, but I don't yet know how to create custom steps and descriptions in the installer.
Additional resources:
A somewhat more detailed guide to the anatomy of a macOS app
A guide I found, but didn't use, on using codesign to get your app past Gatekeeper
A RealPython tutorial I found on using PyInstaller to build Python-based applications for all platforms

How to distribute C++ application which calls Python?

I know there is some way to call Python from C++, like Python/C API or Boost.Python. My question is, how can I distribute the application? For example, does user still need to install Python and Python packages on their machine?
My user case is: I want to use some Python code from my C++ code. The main application is written in C++. Then I am going to deploy my app. The goal is to make the app self contained, and user don't need to install Python and Python packages at all.
The possible steps may be :
1, calling Python from C++ via Python/C API or boost.Python from source code.
2, bring Python/C libraries together with application.
I hope after these 2 steps, my app will be a self-contained and standalone software. User can just copy the app folder to any other machines which has no Python installed.
Note that due to license issue, I can not use PyInstaller. I also meet some problems when trying to use "Nuitka" to make the Python part self contained. So I am now trying directly calling Python from C++. I know it will run on my developer machine. But needs to confirm that this solution can also make app self-contained and won't ask user to install Python.
Update: Now I feel I need to do something to make my app self-contained if I use Python/C to call python from C++ :
1, I need to bring all needed runtime with my app. (C++ runtime of course, and the python_version.dll)
2, I need to deploy a Python interpreter inside my app. Simply copy the Python folder from Python installation and remove some not needed files (like header files, lib files)
3, use Py_SetPythonHome function to points to the copied Python interpreter inside the app.

I'd say you're on the right track. Basically, you should obtain a Python (shared or static) library, compile your program with it, and of course bundle the Python dependencies you have with your program. The best documentation I've read is available here: https://docs.python.org/3.8/extending/embedding.html#embedding-python-in-another-application. Roughly, the process is:
Get a Python library from python.org and compile with ./configure --enable-shared (I believe omitting --enable-shared does only produce the python binary).
Compile your program. Have it reference the headers under Include and link the library. Note that you can obtain the compiler and linker flags you need as described here.
Call Python code from within your application using e.g. PyRun_SimpleString() or other functions from the C API. Note that you may also depend on the Python standard library (under Lib in the distribution) if there's any functionality you use from it.
If you linked against Python statically, at this point you're done, aside from bundling any Python code you depend on, which I'm not sure is relevant in your case.

I am suffering from the same problem, I had a project which is made up of C++ and python(embedded) and there is a problem of deployment/distribution.
After research, I got a solution which is not perfect (means it will be helpful to run your app in other system)
change visual studio in release mode and compile(you got a folder in your working directory)
install pyinstaller (pip install pyinstaller)
then navigate to pyinstaller folder and command:-pyinstaller.exe "your script_file_path.py"
-it will create a dist folder
copy that folder in working folder where exe exists.
remember.
dist folder and c/python code compiled by same version of python.
now good to go.
it will work.

Is vscode-python extension for VSCode is a Language Service or just an extension only?

I want to write an extension for VSCode that reuse refactor/rename function of vscode-python extension. So that, when the user executes my command, my extension will do rename on a variable of .py file. I don't know if vscode-python is just an extension only or it's implementing Language Server Protocol LSP. And how I call the rename/refactor function of vscode-python. Can you give me some hints? Thank you very much!

It appears that the Python extension is implementing the VSCode API directly for features such as rename, rather than using the LSP abstraction layer. See for instance its renameProvider.ts, which implements vscode.RenameProvider:
https://github.com/Microsoft/vscode-python/blob/c976b6c98e34e041b7ee826ec22f9820ba90f8ac/src/client/providers/renameProvider.ts
I don't think how it's implemented really matters to you as a caller though - It seems like you're in luck, the ability to programatically trigger a rename was added just recently and will be released soon as part of VSCode 1.25.0 / the June release:
API to programmatically begin rename (#50856)
It's also part of the release notes for the upcoming version, which give the following usage example:
vscode.commands.executeCommand(
'editor.action.rename',
[vscode.Uri.file('/my/file.abc'), new vscode.Position(14, 7)]
)

Fully embedded SymPy+Matplotlib+others within a C/C++ application

I've read the Python documentation chapter explaining how to embed the Python interpreter in a C/C++ application. Also, I've read that you can install Python modules either in a system-wide fashion, or locally to a given user.
But let's suppose my C/C++ application will use some Python modules such as SymPy, Matplotlib, and other related modules. And let's suppose end users of my application won't have any kind of Python installation in their machines.
This means that my application needs to ship with "pseudo-installed" modules, inside its data directories (just like the application has a folder for icons and other resources, it will need to have a directory for Python modules).
Another requirement is that the absolute path of my application installation isn't fixed: the user can "drag" the application bundle to another directory and it will run fine there (it already works this way but that's prior to embedding Python in it, and I wish it continues being this way after embedding Python).
I guess my question could be expressed more concisely as "how can I use Python without installing Python, neither system-wide, nor user-wide?"

There are various ways you could attempt to do this, but none of them are general solutions. From the (docs):
5.5. Embedding Python in C++
It is also possible to embed Python in a C++ program; precisely how this is done will depend on the details of the C++ system used; in general you will need to write the main program in C++, and use the C++ compiler to compile and link your program. There is no need to recompile Python itself using C++.
This is the shortest section in the document, and is roughly equivalent to: 'left as an exercise for the reader`. I do not believe you will find any straight forward solutions.
Use pyinstaller to gather the pieces:
This means that my application needs to ship with "pseudo-installed" modules, inside its data directories (just like the application has a folder for icons and other resources, it will need to have a directory for Python modules).
If I needed to tackle this problem, I would use pyinstaller as a base. (Disclosure: I am an occasional contributer). One of the major functions of pyinstaller is to gather up all of the needed resources for a python program. In onedir mode, all of the things needed to let the program run are gathered into one directory.
You could include this tool into your make system, and have it place all of the needed pieces into your python data directory in your build tree.

How to debug C extensions for Python on Windows

I have a problem with a segfault in pyodbc and would like to debug it in Windows XP x86. However, the information online seems primarily Linux-centric. What is the best way to go about this?

So I was able to successfully resolve my issue by using Visual Studio 2008. I loosely followed the steps listed here -
http://www.velocityreviews.com/forums/t329214-debugging-python-extensions.html
And some tips on workarounds here -
Compiling python modules whith DEBUG defined on MSVC
Here is my version of the steps for anyone else who may encounter this problem.
In case you haven't already, be sure to setup the Python header and libs directories in VS
a. Go to Tools > Options > Projects and Solutions > VC++ Directories. Be sure to add your include and libs path to the Include and Library files' path, respectively. (e.g. C:\Python27\include, C:\Python27\libs)
Go to your Python include folder (once again, e.g. C:\Python27\include) and edit pyconfig.h. Comment out the line # define Py_DEBUG and save. Go to your libs folder (e.g. C:\Python27\libs) and make a copy of python27.lib. Name the copy python27_d.lib.
Create a new project. Choose Win32 Project and name it the module name (in my case pyodbc. Click Next then choose DLL for Application type and check Empty Project.
In the Solution Explorer, right-click on Header Files and choose Add > Existing Item. Select all of the header files that you need. Do the same for Source Files.
Go to Project > Properties, then under Configuration Properties -
a. General - ensure that you are using the correct Character Set. For me it was Use Multi-Byte Character Set. Python 3 probably needs Use Unicode Character Set.
b. Debugging - enter the path to Python in the Command field. (e.g. C:\Python27\python.exe). Then set Attach to Yes.
c. Linker > General - change the Output File to end in .pyd instead of .dll.
Ensure that your configuration is set to Debug. Go to Build > Build Solution.
Open a cmd and cd into the directory where your pyd file was compiled. Start python from the cmd window. To attach debugger on this running python process, go back to Visual Studio and click the green play button to start debugging. You can also use Debugging -> Attach to Process... Now go back to Python and import your module. Play, test, and try to break it!

Debugging workflow with WinDbg
This workflow will create debugging information for a Release build, so you
don't have to mess with the original include and library files of Python.
Download and install Debugging Tools for Windows
Get the symbol files for your Python version and extract them. For Python
2.7.3 this would be http://www.python.org/ftp/python/2.7.3/python-2.7.3-pdb.zip.
Modify setup.py to generate debugging files. You have to add '/Zi' to
extra_compile_args and '/DEBUG' to extra_link_args. Example:
ext_modules = [Extension('pyuv', sources=['src/pyuv.c'],
extra_compile_args=['/Zi'],
extra_link_args=['/DEBUG'])
]
Build the extension as always (python setup.py ...).
Start WinDbg and specify the Symbol Search Path (Ctrl + S).
C:\Path\To\Extension_pdb
C:\Path\To\Extracted\python-2.7.3-pdb
srv*;SRV*c:\tmp*http://msdl.microsoft.com/download/symbols
The last line will download and cache required symbols for Windows modules.
Start the Python executable (Ctrl + E). You can directly execute a
script or run in interactive mode.
Skip the initial breakpoint with "Go" (F5).
If there is a Segmentation fault the execution will break and you will
see something like Access violation - code c0000005 (first chance)
in the WinDbg console.
You can get detailed exception information by typing !analyze -v
in the WinDbg console and the current stack trace with kb.
Here is an
example of such an output.
You should be able to combine this approach with pyrospade's
answer to debug with
Visual Studio if you omit his second step and build the project with
Release configuration.
A further tutorial for WinDbg could be found here.

Segfaults are especially mysterious, as there is no way to trap for them from your Python code, or even to get much stacktrace information on the C side of things. One thing that can give you at least a little more info is to use the Google breakpad C library to report a C stack trace when the segfault occurs.

You may want to try David Malcolm's tool CPyChecker which statically analyses C extensions for memory leaks and other bugs. The tool is documented here.