Python .pyc files and Windows UAC - python

I'm working on an Inno Setup installer for a Python application for Windows 7, and I have these requirements:
The app shouldn't write anything to the installation directory
It should be able to use .pyc files
The app shouldn't require a specific Python version, so I can't just add a set of .pyc files to the installer
Is there a recommended way of handling this? Like give the user a way to (re)generate the .pyc files? Or is the shorter startup time benefit from the .pyc files usually not worth worrying about?

PYC files aren't guaranteed to be compatible for different python versions. If you don't know that all your customers are running the same python versions, you really don't want to distribute pyc's directly. So, you have to choose between distributing PYCs and supporting multiple python versions.
You could create build process that compiles all your files using py_compile and zips them up into a version-specific package. You can do this with setuptools.; however it will be awkward to do because you'll have to run py_compile in every version you need to support.
If you are basically distributing a closed application and don't want people to have trivial access to your source code, then py2exe is probably a simpler alternative. If your python is supposed to be integrated into the user's python install, then it's probably simpler to just create a zip of your .py files and add a one-line .py stub that imports the zipped package(s) using zipfile
if it makes you feel better, PYC doesn't provide much extra security and it doesn't really boost perf much either :)

If you haven't read PEP 3147, that will probably answer your questions.
I don't mean the solution described in that PEP and implemented as of Python 3.2. That's great if your "multiple Python versions" just means "3.2, 3.3, and probably future 3.x". Or even if it means "2.6+ and 3.1+, but I only really care about 3.2 and 3.3, so if I don't get the pyc speedups for other ones that's OK".
But when I asked your supported versions, you said, "2.7", which means you can't rely on PEP 3147 to solve your problems.
Fortunately, the PEP is full of discussion of earlier attempts to solve the problem, and the pitfalls of each, and there should be more than enough there to figure out what the options are and how to implement them.
The one problem is that the PEP is very linux-centric—mainly because it's primarily linux distros that tried to solve the problem in the past. (Apple also did so, but their solution was (a) pretty much working, and (b) tightly coupled with the whole Mac-specific "framework" thing, so they were mostly ignored…)
So, it largely leaves open the question of "Where should I put the .pyc files on Windows?"
The best choice is probably an app-specific directory under the user's local application data directory. See Known Folders if you can require Vista or later, CSIDL if you can't. Either way, you're looking for the FOLDERID_LocalAppData or CSIDL_LOCAL_APPDATA, which is:
The file system directory that serves as a data repository for local (nonroaming) applications. A typical path is C:\Documents and Settings\username\Local Settings\Application Data.
The point is that it's a place for applications to store data that's separate for each user (and inside that user's profile directory), and also for each machine the user's roaming profile might end up on, which means you can safely put stuff there and know that the user has the permissions to write there without UAC getting involved, and also know (as well as you ever can) that no other user or machine will interfere with what's there.
Within that directory, you create a directory for your program, and put whatever you want there, and as long as you picked a unique name (e.g., My Unique App Name or My Company Name\My App Name or a UUID), you're safe from accidental collision with other programs. (There used to be specific guidelines on this in MSDN, but I can no longer find them.)
So, how do you get to that directory?
The easiest way is to just use the env variable %LOCALAPPDATA%. If you need to deal with older Windows, you can use %USERPROFILE% and tack \Local Settings\Application Data onto the end, which is guaranteed to either be the same, or end up in the same place via junctions.
You can also use pywin32 or ctypes to access the native Windows APIs (since there are at least 3 different APIs for this and at least two ways to access those APIs, I don't want to give all possible ways to write this… but a quick google or SO search for "pywin32 SHGetFolderPath" or "ctypes SHGetKnownFolderPath" or whatever should give you what you need).
Or, there are multiple third-party modules to handle this. The first one both Google and PyPI turned up was winshell.

Re-reading the original question, there's a much simpler answer that probably fits your requirements.
I don't know much about Inno, but most installers give you a way to run an arbitrary command as a post-copy step.
So, you can just use python -m compileall to create the .pyc files for you at install time—while you've still got elevated privileges, so there's no problem with UAC.
In fact, if you look at pywin32, and various other Python packages that come as installer packages, they do exactly this. This is an idiomatic thing to do for installing libraries into the user's Python installation, so I don't see why it wouldn't be considered reasonable for installing an executable that uses the user's Python installation.
Of course if the user later decides to uninstall Python 2.6 and install 2.7, your .pyc files will be hosed… but from your description, it sounds like your entire program will be hosed anyway, and the recommended solution for the user would probably be to uninstall and reinstall anyway, right?

Related

How do I make a macOS app out of my Python program?

I've made this question because I had to go through the whole process of creating my own application using Apple's somewhat lacking documentation, and without the use of py2app. I wanted to create the whole application structure so I know exactly what was inside, as well as create an installer for it. The latter of these is still a mystery, so any additional answers with information on making a custom installer would be appreciated. As far as the actual "bundle" structure goes, however, I think I've managed to get the basics down. See the answer below.
Edit: A tutorial has been linked at the end of this answer on using PyInstaller; I don't know how much it helps as I haven't used it yet, but I have yet to figure out how to make a standalone Python application without the use of a tool like this and it may just be what you're looking for if you wish to distribute your application without relying on users knowing how to navigate their Python installations.
A generic application is really just a directory with a .app extension. So, in order to build your application, just make the folder without the extension first. You can rename it later when you're finished putting it all together. Inside this main folder will be a Contents folder, which will hold everything your application needs. Finally, inside Contents, you will place a few things:
Info.plist
MacOS
Resources
Frameworks
Here you can find some information on how to write your Info.plist file. Basically, this is where you detail information about your application.
Inside the MacOS you want to place your main executable. I'm not sure that it matters how you write it; at first, I just had a shell script that called python3 ./../Resources/MyApp.py. I didn't think this was very neat though, so eventually I called the GUI from a Python script which became my executable (I used Tkinter to build my application's GUI, and I wrote several modules which I will get to later). So now, my executable was a Python script with a shebang pointing to the Python framework in my application's Frameworks folder, and this script just created an instance of my custom Tk() subclass and ran the mainloop. Both methods worked, though, so unless someone points out a reason to choose one method over the other, feel free to pick. The one thing that I believe is necessary, is that you name your executable the SAME as your application (before adding the .app). That, I believe, is the only way that MacOS knows to use that file as your application's executable. Here is a source that describes the bundle structure in more detail; it's not a necessary read unless you really want to get into it.
In order to make your executable run smoothly, you want to make sure you know where your Python installation is. If you're like me, the first thing you tried doing on your new Mac was open up Terminal and type in python3. If this is the case, this prompted you to install the Xcode Command Line tools, which include an installation of Python 3.8.2 (most recent on Xcode 12). Then, this Python installation would be located at /usr/bin/python3, although it's actually using the Python framework located at
/Applications/Xcode.app/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3
I believe, but am NOT CERTAIN, that you could simply make a copy of this framework and add it to your Frameworks folder in order to make the app portable. Make a copy of the Python3.framework folder, and add it to your app's Frameworks folder. A quick side note to be wary of; Xcode comes packaged with a lot of useful tools. In my current progress, the tool I am most hurting for is the Fortran compiler (that I believe comes as a part of GCC), which comes with Xcode. I need this to build SciPy with pip install scipy. I'm sure this is not the only package that would require tools that Xcode provides, but SciPy is a pretty popular package and I am currently facing this limitation. I think by copying the Python framework you still lose some of the symlinks that point to Xcode tools, so any additional input on this would be great.
In any case, locate the Python framework that you use to develop your programs, and copy it into the Frameworks folder.
Finally, the Resources folder. Here, place any modules that you wrote for your Python app. You also want to put your application's icon file here. Just make sure you indicate the name of the icon file, with extension, in the Info.plist file. Also, make sure that your executable knows how to access any modules you place in here. You can achieve this with
import os
os.chdir('./../Resources')
import MyModules
Finally, make sure that any dependencies your application requires are located in the Python framework site-packages. These will be located in Frameworks/Python3.framework/Versions/3.X.Y/lib/python3.x.y/site-packages/. If you call this specific installation of Python from the command line, you can use path/to/application/python3 -m pip install package and it should place the packages in the correct folder.
P.S. As far as building the installer for this application, there are a few more steps needed before your application is readily downloaded. For instance, I believe you need to use the codesign tool in order to approve your application for MacOS Gatekeeper. This requires having a developer license and manipulating certificates, which I'm not familiar with. You can still distribute the app, but anyone who downloads it will have to bypass the security features manually and it will seem a bit sketchy. If you're ready to build the installer (.pkg) file, take a look at the docs for productbuild; I used it and it works, but I don't yet know how to create custom steps and descriptions in the installer.
Additional resources:
A somewhat more detailed guide to the anatomy of a macOS app
A guide I found, but didn't use, on using codesign to get your app past Gatekeeper
A RealPython tutorial I found on using PyInstaller to build Python-based applications for all platforms

Deploying Python modules for 3D software

I have been developing a fairly extensive library of python modules that automate the more time consuming parts of "3D character development" for games/film/tv.
All of my code up until a few months ago has been run within Maya's dedicated python interpreter, however, my GUIs are built in PySide/PyQt, and so, run just fine in mac/windows/linux or a few other Graphics programs such as Nuke, XSI, Max.
What I would really like to figure out is a "simple" way to distribute my code to various different people ---> using various different operating Systems ---> potentially using various applications (Nuke, XSI, Max), which, in turn, have their own dedicated python interpreters.
The obvious option would be pip and easy_install.. These modules are clearly the "right" way to go, but its not really clear how a user would install/run them under the dedicated python installs that ship with Maya/Nuke/ etc...Though, it does seem possible (as explained here). Still Its going to be a pretty big barrier for a less-technical user.
Any help or points in the right direction would be immensely appreciated..
I would not say that pip/easy_install are the 'right' way for this problem. They are pretty good (not quite 'great') tools for motivated, technically inclined users -- but even in that context they have issues (such as unintended upgrades or deletions). Most importantly, they are opt-in methods: nobody can make you pip unless you want to. This means users can accidentally or deliberately get themselves into very different positions from each other, which makes support and maintenance a nightmare.
I've had very good luck in Maya distributing a zipped file containing a complete environment - all the modules etc. userSetup.py adds that zip to the path and the Python's native zipimport functionality handles the rest. This makes sure that there is only one file to maintain and distribute. It also fixes the common problem of leftover .pyc files creating havok after .py files get moved or renamed. Since this is all standard python, I'd assume this will work for any app-specific python that uses a 2.6+ version of python, though I've never tried it in Nuke or Max.
The main wrinkle will be modules with .pyd or other binary components, typically these don't work inside the zip files. I include a bootstrap routine which unpacks those to a (disposable) location on the user's disk and adds that to the path.
There's a detailed discussion of the method here and some background here

Py2App Can't find standard modules

I've created an app using py2app, which works fine, but if I zip/unzip it, the newly unzipped version can't access standard python modules like traceback, or os. The manpage for zip claims that it preserves resource forks, and I've seen other applications packaged this way (I need to be able to put this in a .zip file). How do I fix this?
This is caused by building a semi-standalone version that contains symlinks to the natively installed files and as you say, the links are lost when zipping/unzipping unless the "-y" option is used.
An alternate solution is to build for standalone instead, which puts (public domain) files inside the application and so survives zipping/unzipping etc. better. It also means the app is more resilient to changes in the underlying OS. The downside is that it is bigger, of course, and is more complicated to get it set up.
To build a stand alone version, you need to install the python.org version which can be repackaged.
An explanation of how to do this is here, but read the comments as there have been some changes since the blog post was written.
use zip -y ... to create the file whilst preserving symlinks.
You probably need to give it your full PYTHONPATH.
Depends on your os. Here's how to find out:
import os [or any other std module]
os.file()

Trimming Python Runtime

We've got a (Windows) application, with which we distribute an entire Python installation (including several 3rd-party modules that we use), so we have consistency and so we don't need to install everything separately. This works pretty well, but the application is pretty huge.
Obviously, we don't use everything available in the runtime. I'd like to trim down the runtime to only include what we really need.
I plan on trying out py2exe, but I'd like to try and find another solution that will just help me remove the unneeded parts of the Python runtime.
One trick I've learned while trimming down .py files to ship: Delete all the .pyc files in the standard library, then run your application throughly (that is, enough to be sure all the Python modules it needs will be loaded). If you examine the standard library directories, there will be .pyc files for all the modules that were actually used. .py files without .pyc are ones that you don't need.
Both py2exe and pyinstaller (NOTE: for the latter use the SVN version, the released one is VERY long in the tooth;-) do their "trimming" via modulefinder, the standard library module for finding all modules used by a given Python script; you can of course use the latter yourself to identify all needed modules, if you don't trust pyinstaller or py2exe to do it properly and automatically on your behalf.
This py2exe page on compression suggests using UPX to compress any DLLs or .pyd files (which are actually just DLLs, still). Obviously this doesn't help in trimming out unneeded modules, but it can/will trim down the size of your distribution, if that's a large concern.

Distribute a Python program with a minimal environment

I want to distribute a Python application to windows users who don't have Python or the correct Python version.
I have tried py2exe conversion but my Python program is really complex and involve code import on the fly by xmlrpc process so it is not suitable for py2exe.
The complete Python folder takes around 80MB but this includes docs and a lot of non-essential things.
Do you know if there exists a small package of a minimal Python interpreter I can include with my program ? Include a folder of 80MB is a bit big ;)
PyInstaller is a py2exe "competitor" that has many extras (such as being cross-platform, supporting popular third party packages "out of the box", and explicitly supporting advanced importing options) -- it might meet your needs. Just be sure to install the SVN trunk -- the existing (1.3) release is way, WAY obsolete (PyInstaller is under active development again since quite a while, but I can't convince the current maintainers to stop and do a RELEASE already -- they're kind of perfectionists and keep piling more and more great goodies, optimizations, enhancements, etc, into the SVN trunk instead;-).
Have a look at Portable Python. This will install a Python programming environment in a local folder. I am sure that you could strip many unwanted things off.
I recommend however that you give py2exe another chance.
..involve code import on the fly by xmlrpc process so it is not suitable for py2exe
Py2exe can deal with situations like this. You just have to tell it which modules are being imported at runtime, so that it includes them in the distribution. Your code should then be able to import from these modules dynamically.
püy2exe is bad and incompabilite to Windows 10 now.
I suggest you use BoxedApp Packer until 22 mb small without runtimes....
enter link description here
It is almost better than py2exe because py2exe need many py files and opened data files...

Categories