I am new to Python and am totally lost as to where to even start to get this done.
I have written many small modules (a toolset for maya) that need to be compiled into on single .pyc file. Is there a module that just does this? Or can you tell me where to go to start? A tutorial? I don't even know what terms to look for.
You don't even need to make an egg, you can just zip up your files and put the zip file onto your python path. Maya's version of python includes the zipimport module by default so it 'just works' as long as maya can find your zip file.
Here are some discussions of the whole topic of tools distribution you might find useful:
http://tech-artists.org/forum/showthread.php?3271-Distribution-techniques-for-external-Python-tools&highlight=distribute
http://tech-artists.org/forum/showthread.php?3987-Maya-GitHub-and-Script-Paths-for-Mel-and-Python-How-Would-You-Do-It&highlight=distribute
http://tech-artists.org/forum/showthread.php?3752-Best-Way-to-Share-Your-Scripts&highlight=distribution
The compileall module, present in the standard library, will compile all files in a directory, but it will still not generate a single .pyc file. An example of its usage is also given at effbot.org. I don't know if it's possible at all to create a single pyc file out of multiple modules.
However, my guess is that you are looking into creating a python egg, which does make a single file out of a series of files grouped together in a package, which is what you want, I think.
Related
I have a .py file that imports from other python modules that import from config files, other modules, etc.
I am to move the code needed to run that .py file, but only whatever the py file is reading from (I am not talking about packages installed by pip install, it's more about other python files in the project directory, mostly classes, functions and ini files).
Is there a way to find out only the external files used by that particular python script? Is it something that can be found using PyCharm for example?
Thanks!
Static analysis tools (such as PyCharm's refactoring tools) can (mostly) figure out the module import tree for a program (unless you do dynamic imports using e.g. importlib.import_module()).
However, it's not quite possible to statically definitively know what other files are required for your program to function. You could use Python's audit events (or strace/ptrace or similar OS-level functions) to look at what files are being opened by your program (e.g. during your tests being run (you do have tests, right?), or during regular program use), but it's likely not going to be exhaustive.
I am making a password manager in python but I thought of putting the hashes of the master password or the security keys directly into the program then I will convert it into an exe and use it.
So my question is that after converting it to the exe using the pyinstaller or py2exe module will any one be able to reconvert it backwards into python code and see it or will my python code be seeable after converting it to the exe.
I know what I am doing is not a good practice but it was also out of my curiosity.
Yes, they almost certainly will.
To create the exe, PyInstaller basically compiles your files to .pyc files, bundles them with any other resources. When you run the file, it extracts them to a temporary folder and runs them. You can learn more about that process from the docs for PyInstaller. Someone that knows what they're doing can look at all those extracted files while the application is running, or just extract it themselves using a tool like pyinstxtractor. The strings for the hashes themselves will still appear in the .pyc files, and someone who knows what kind of pattern to look for can probably find them without too much trouble.
If you want to look into some options to make it more difficult, you could explore some suggested options from PyInstaller such as compiling to Cython or adding a key.
But if you're only storing the hashes, they can only get the hashes. Whether or not they can derive the original key depends on the hashing algorithm you use.
I write and maintain a Python library for quantum chemistry calculations called PyQuante. I have a fairly standard Python distribution with a setup.py file in the main directory, a subdirectory called "PyQuante" that holds all of the Python modules, and one called "Src" that contains source code for C extension modules.
I've been lucky enough to have some users donate code that uses Cython, which I hadn't used before, since I started PyQuante before either it or Pyrex existed. On my suggestion, they put the code into the Src subdirectory, since that's where all the C code went.
However, looking at the code that generates the extensions, I wonder whether I should have simply put the code in subdirectories of the Python branch instead. And thus my question is:
what are the best practices for the directory structure of python distributions with both Python and Cython source files?
Do you put the .pyx files in the same directory as the .py files?
Do you put them in in a subdirectory of the one that holds the .py files?
Do you put them in a child of the .py directory's parent?
Does the fact that I'm even asking this question betray my ignorance at distributing .pyx files? I'm sure there are many ways to make this work, and am mostly concerned with what has worked best for people.
Thanks for any help you can offer.
Putting the .pyx files in the same directory as .py files makes the most sense to me. It's what the authors of scikit-learn have done and what I've done in my py-earth module. I guess I think of Cython modules as optimized replacements for Python modules. I will often begin by writing a package in pure Python, then replace some modules with Cython if I need better performance. Since I'm treating Cython modules as replacements for Python modules, it makes sense to me to keep them in the same place. It also works well for test builds using the --inplace argument.
I want to know what a pyc file(python bytecode) is. I want to know all the details.
I want to know about how pyc files interface with the compiler. Is it a replacement for exe?
Does it need to be run by python?
Is it as portable as the .py file is?
Where should I use this?
To supplement Mike Graham's answer there are some interesting comments here giving some information on pyc files. Most interestingly I suspect for you is the line:
A program doesn't run any faster when it is read from a ‘.pyc’ or ‘.pyo’ file than when it is read from a ‘.py’ file; the only thing that's faster about ‘.pyc’ or ‘.pyo’ files is the speed with which they are loaded.
Which hits the nail on the head w.r.t. the crux of a pyc file. A pyc is a pre-interpreted py file. The python bytecode is still the same as if it was generated from a py file - the difference is that when using a pyc file you don't have to go through the process of creating that pyc output (which you do when running a py file). Read as you don't have to convert the python script to python bytecode.
If you've come across .class files in java this is a similar concept - the difference is in java you have to do the compiling using javac before the java interpreter will execute the application. Different way of doing things (the internals will be very different as they're different languages) but same broad idea.
Python bytecode requires Python to run, cannot be ran standalone without Python, and is specific to a particular x.y release of Python. It should be portable across platforms for the same version. There is not a common reason for you to use it; Python uses it to optimize out parsing of your .py file on repeated imports. Your life will be fine ignoring the existence of pyc files.
From the docs:
As an important speed-up of the start-up time for short programs that use a lot of standard modules, if a file called spam.pyc exists in the directory where spam.py is found, this is assumed to contain an already-“byte-compiled” version of the module spam. The modification time of the version of spam.py used to create spam.pyc is recorded in spam.pyc, and the .pyc file is ignored if these don’t match.
See the ref for more info. But some specific answers:
The contents of the spam.pyc file are platform independent, so a Python module directory can be shared by machines of different architectures.
It's not an executable; it's used internally by the compiler as an intermediate step.
In general, you don't make .pyc files by hand: the interpreter makes them automatically.
We've got a (Windows) application, with which we distribute an entire Python installation (including several 3rd-party modules that we use), so we have consistency and so we don't need to install everything separately. This works pretty well, but the application is pretty huge.
Obviously, we don't use everything available in the runtime. I'd like to trim down the runtime to only include what we really need.
I plan on trying out py2exe, but I'd like to try and find another solution that will just help me remove the unneeded parts of the Python runtime.
One trick I've learned while trimming down .py files to ship: Delete all the .pyc files in the standard library, then run your application throughly (that is, enough to be sure all the Python modules it needs will be loaded). If you examine the standard library directories, there will be .pyc files for all the modules that were actually used. .py files without .pyc are ones that you don't need.
Both py2exe and pyinstaller (NOTE: for the latter use the SVN version, the released one is VERY long in the tooth;-) do their "trimming" via modulefinder, the standard library module for finding all modules used by a given Python script; you can of course use the latter yourself to identify all needed modules, if you don't trust pyinstaller or py2exe to do it properly and automatically on your behalf.
This py2exe page on compression suggests using UPX to compress any DLLs or .pyd files (which are actually just DLLs, still). Obviously this doesn't help in trimming out unneeded modules, but it can/will trim down the size of your distribution, if that's a large concern.