I am wondering what options I have to speed up kivy after I package it.
I was looking into cython and pypy because those should speed up python. Also, I was reading that kivy can be a little slower and bigger than most android applications because it includes python and the interpreter. I usually just search around forever until i find an answer but it can be hard to find things about kivy.
Can anyone with experience recommend something to get better speeds out of this framework? I'm dealing with a lot of code so it could be cumbersome testing a lot of this stuff out.
edit: 132137
asked Nov 24 '17 at 19:13
I have a lot of this application packaged now. I wouldn't so much worry about cython until you are packaging it. I would also try to package the application incrementally to make sure everything works. More than anything, I was just really worried about how things would go when I started packaging it. I should have started earlier. The size hasn't been too much of an issue. I would try and write it on ubuntu or a linux distribution(buildozer doesn't work with windows) and not everything will run the same cross all platforms(I had some issues with some of the modules I was working with). I love kivy this is like an eli5 thing I wish I'd known at the time.
After messing around with it some I got it down to 16mb. So I'm really happy with the framework. I guess i didn't need to include the buildozer folder in the build. I'm new to programming but I'm pretty happy with how everything turned out.
When it comes to responsiveness, make sure your python code is optimised. That means doing things like not loading Screens or other widgets until you need them (or even doing it in the background as much as possible).
For speeding up python itself, the main supported method is cython. python-for-android does not support pypy.
kivy can be a little slower and bigger than most android applications because it includes python and the interpreter.
A basic Kivy-using APK is about 7MB. And the delay from starting the interpreter manifests largely during the startup of the application, which can take a few seconds, especially on older devices.
Is it possible to deploy python applications such that you don't release the source code and you don't have to be sure the customer has python installed?
I'm thinking maybe there is some installation process that can run a python app from just the .pyc files and a shared library containing the interpreter or something like that?
Basically I'm keen to get the development benefits of a language like Python - high productivity etc. but can't quite see how you could deploy it professionally to a customer where you don't know how there machine is set up and you definitely can't deliver the source.
How do professional software houses developing in python do it (or maybe the answer is that they don't) ?
You protect your source code legally, not technologically. Distributing py files really isn't a big deal. The only technological solution here is not to ship your program (which is really becoming more popular these days, as software is provided over the internet rather than fully installed locally more often.)
If you don't want the user to have to have Python installed but want to run Python programs, you'll have to bundle Python. Your resistance to doing so seems quite odd to me. Java programs have to either bundle or anticipate the JVM's presence. C programs have to either bundle or anticipate libc's presence (usually the latter), etc. There's nothing hacky about using what you need.
Professional Python desktop software bundles Python, either through something like py2exe/cx_Freeze/some in-house thing that does the same thing or through embedding Python (in which case Python comes along as a library rather than an executable). The former approach is usually a lot more powerful and robust.
Yes, it is possible to make installation packages. Look for py2exe, cx_freeze and others.
No, it is not possible to keep the source code completely safe. There are always ways to decompile.
Original source code can trivially be obtained from .pyc files if someone wants to do it. Code obfuscation would make it more difficult to do something with the code.
I am surprised no one mentioned this before now, but Cython seems like a viable solution to this problem. It will take your Python code and transpile it into CPython compatible C code. You also get a small speed boost (~25% last I checked) since it will be compiled to native machine code instead of just Python byte code. You still need to be sure the user has Python installed (either by making it a pre-requisite pushed off onto the user to deal with, or bundling it as part of the installer process). Also, you do need to have at least one small part of your application in pure Python: the hook into the main function.
So you would need something basic like this:
import cython_compiled_module
if __name__ == '__main__':
cython_compiled_module.main()
But this effectively leaks no implementation details. I think using Cython should meet the criteria in the question, but it also introduces the added complexity of compiling in C, which loses some of Python's easy cross-platform nature. Whether that is worth it or not is up to you.
As others stated, even the resulting compiled C code could be decompiled with a little effort, but it is likely much more close to the type of obfuscation you were initially hoping for.
Well, it depends what you want to do. If by "not releasing the source code" you mean "the customer should not be able to access the source code in any way", well, you're fighting a losing battle. Even programs written in C can be reverse engineered, after all. If you're afraid someone will steal from you, make them sign a contract and sue them if there's trouble.
But if you mean "the customer should not care about python files, and not be able to casually access them", you can use a solution like cx_Freeze to turn your Python application into an executable.
Build a web application in python. Then the world can use it via a browser with zero install.
I have an application written in Python and 'compiled' with PyInstaller. It also uses PyQt for the GUI framework.
Running this application has a delay of about 10 seconds before the main window loads and is shown. As far as I can tell, this is not due to slowness in my code. Instead, I suspect this is due to the Python runtime initializing.
The problem is that this application is started with a custom laucncher / taskbar application. The user will click the button to launch the app, see nothing appear to happen, and click elsewhere on another application. When my application shows it's window, it cannot come to the foreground due to the rules for SetForegroundWindow.
I have access to the source for the PyInstaller win32 loader, the Python code, and even the launcher code.
My questions are:
How can I make this application start faster?
How can I measure the time spend i the first few seconds of the process's lifetime?
What is the generally accepted technique for reducing time until the first window is shown?
I'd like to avoid adding a splash screen for two reasons - one, I expect it won't help (the overhead is before Python code runs) and two, I just don't like splash screens :)
If I need to, I could probably edit the PyInstaller loader stub to create a window, but that's another route i'd rather not take.
I suspect that you're using pyinstaller's "one file" mode -- this mode means that it has to unpack all of the libraries to a temporary directory before the app can start. In the case of Qt, these libraries are quite large and take a few seconds to decompress. Try using the "one directory" mode and see if that helps?
Tell PyInstaller to create a console-mode executable. This gives you a working console you can use for debugging.
At the top of your main script, even before the first import is run, add a print "Python Code starting". Then run your packaged executable from the command line. This way you can get a clear picture whether the time is spent in PyInstaller's bootloader or in your application.
PyInstaller's bootloader is usually quite fast in one-dir mode, but it can be much slower in one-file mode, because it depacks everything into a temporary directory. On Windows, I/O is very slow, and then you have antiviruses that will want to double check all those DLL files.
PyQt itself is a non-issue. PyQt is generated by SIP which generates very fast lazy bindings; importing the whole PyQt is faster than any other GUI library because it basically does nothing: all bindings to classes/functions are dynamically created when (and if!) you access them, saving lots of memory too.
If your application is slow at coming up, that will be true without PyInstaller as well. In that case, your only solution is either a splash screen (import just PyQt, create QApplication, create a display the splashscreen, then import the rest of your program and run it), or rework your code. I can't help you much without details.
I agree with above answers. My Qt python program needed about 5 seconds to start up on a decent PC when using onefile mode. After I change to --onedir, it only took around one second to start; almost immediately after user double clicks the exe file. But the drawback is that there are many files in that directory which is not so neat.
For my application, the long startup time almost entirely was caused by the antivirus system. Switching it off reduced the startup in my case from 3 minutes to less than 10secs!
To bring these measurements into perspective: My application was bundled with extra data files (about 150 files with a payload of 250MB), besides carrying around Qt and numpy (that may depend on Intel MKL, which alone adds another 200MB to the bundle!) dependencies. It even didn't help much that the tested system was running with a solid state drive...
In conclusion: if you have a large application with lots of dependencies, the startup time may be affected strongly by the antivirus system!
In case anyone is still have this issue, I resolved mine by running the exe locally and not on any sharedrives. This took the startup time from over a minute to under 10 seconds.
I solved this by adding an exception to the anti-virus monitoring software (F-Secure). It removed the wait of several minutes before running.
I have 'compiled' a few wxPython apps using py2exe and cx_Freeze, None of them take more than 4 seconds to start.
Are you sure sure it's not your code ?
maybe some network or some I/O resource call holding your app ?
Have you tried other machine than yours? Even the fastest hardware can be slow sometimes with the wrong software config, apps or OS, try it.
Try timing it with timeit module.
I never used pyQT, but with wxPython the startup speed is OK, and after the first initialize if I close and open again, it's faster than the first time.
C++ is fast. Qt is fast. Python interpreter startup is relatively fast (2-3 seconds on my computer). PyQt is thin wrapper around Qt. Why then PyQt app startup (about 80 lines, even in .pyc form) is so slow (6-12 seconds)? I'm interested in reasons for this and, most importantly, ways to improve this situation.
There is a known bug in Qt 4.6.3 that cripples application start times. Maybe 4.4.3 is also affected. Try to upgrade to 4.7.0, which fixes that bug.
Your problem might also come from dynamic library relocation. If you compile either Qt or Python yourself, try to give them a base address different from the default (0x10000000), to prevent relocation to occur when the two libraries are loaded together.
Under Linux, you could have used prelink, but to my knowledge it has no equivalent on Windows.
If I write a python script, anyone can simply point an editor to it and read it. But for programming written in C, one would have to use decompilers and hex tables and such. Why is that? I mean I simply can't open up the Safari web browser and look at its code.
Note: The author disavows a deep expertise in this subject. Some assertions may be incorrect.
Python actually is compiled into bytecode, which is what gets run by the python interpreter. Whenever you use a Python module, Python will generate a .pyc file with a name corresponding to the module. This is the equivalent of the .o file that's generated when you compile a C file.
So if you want something to disassemble, the .pyc file would be it :)
The process that Python goes through when compiling a module is pretty similar to what gcc or another C compiler does with C source code. The major difference is that it happens transparently as part of execution of the file. It's also optional: when running a non-module, i.e. an end-user script, Python will just interpret the code rather than compiling it first.
So really your question is "Why are python programs distributed as source rather than as compiled modules?" Or, put another way, "Why are C applications distributed as compiled binaries rather than as source code?"
It used to be very common for C applications to be distributed as source code. This was back before operating systems and their various subentities (i.e. linux distributions) became more established. Some distros, for example gentoo, still distribute apps as source code. Apps which are a bit more cutting edge or obscure are still distributed as source code for all platforms they target.
The reason for this is compatibility, and dependencies. The reason you can run the precompiled binary Safari on a Mac, or Firefox on Ubuntu Linux, is because it's been specifically built for that operating system, architecture (e.g. x86_64), and set of libraries.
Unfortunately, compilation of a large app is pretty slow, and needs to be redone at least partially every time the app is updated. Thus the motivation for binary distributions.
So why not create a binary distribution of Python? For one thing, as Aaron mentions, modules would need to be recompiled for each new version of the Python bytecode. But this would be similar to rebuilding a C app to link with a newer version of a dynamic library — Python modules are analogous in this sense to C libraries.
The real reason is that Python compilation is very much quicker than C compilation. This is in part, I think, because of the dynamic nature of the language, and also because it's not as thorough of a compilation. This has its tradeoffs: in particular, Python apps run much more slowly than do their C counterparts, because Python has to interpret the compiled bytecode into instructions for the processor, whereas the C app already contains such instructions.
That all being said, there is a program called py2exe that will take a Python module and distribution and build a precompiled windows executable, including in it the logic of the module and its dependencies, including Python itself. I guess the point of this is to avoid having to coerce people into installing Python on their Windows system just to run your app. Under linux, or I think even OS/X, Python is usually already installed, so precompilation is not really necessary. Linux systems also have super-dandy package managers that will transparently install dependencies such as Python if they are not already installed.
Python is a script language, runs in a virtual machine through an interpeter.
C is a compiled language, the code compiled to binary code which the computer can run without all that extra stuff Python needs.
This is sorta a big topic. You should look into your local friendly Computer Science curriculum, you'll find a lot of great stuff on this subject there.
The short answer is the Python is an "interpreted" language, which means that it requires a machine language program (the python interpreter) to run the python program, adding a layer of indirection. C or C++ are different. They are compiled directly to machine code, which runs directly on your processor.
There is a lot of additional voodoo to be learned here, however. Technically Python is compiled to a bytecode, and modern interpreters do more and more "Just in Time" compilation, so the boundaries between compiled and interpreted code are getting fuzzier all the time.
In several comments you asked: "Is it then possible to compile python to an executable binary file and then simply distribute that?"
From a theoretical viewpoint, there's no question the answer is yes -- a Python program could be compiled to, and distributed as, fully compiled machine code.
From a practical viewpoint, it's open to a lot more question. There are a few things like Unladen Swallow, Psyco, Shed Skin, and PyPy that you might want to know about though.
Unladen Swallow is primarily an attempt at making Python run faster, but part of the plan to do so involves using LLVM for its back-end. LLVM can (among other things) produce native machine code output. The last couple of releases of Unladen Swallow have used LLVM for native code generation, but 1) the most recent update on the web site is from late 2009, and 2) the release notes for that version say: "The Unladen Swallow team does not recommend wide adoption of the 2009Q3 release."
Psyco works as a plug-in for Python that basically does JIT compilation, so even though it can speed up execution (quite a lot in some cases), it doesn't produce a machine-code executable you can distribute. In short, while it's sort of similar to what you want, it's not intended to do exactly what you've asked for.
Shed Skin Python-to-C++ produces C++ as its output, and you then compile the C++ and (potentially) distribute the result of that. Shedskin is currently at version 0.5 -- i.e., nobody's claiming that it's a finished, released product. On the other hand, development is ongoing, and each release does seem to include pretty substantial improvements.
PyPy is a Python implementation written in Python. Their intent is to allow code production to be "plugged in" without affecting the rest of the implementation -- but while they currently support 4 different code generation models, I don't believe any of them results in producing native machine code that runs directly on the hardware.
Bottom line: work has been done and is being done with the intent of doing what you asked about, but at least to my knowledge there's not really anything I could reasonably recommend as a finished product that you can really depend on to do the job right now. The primary emphasis is really on execution speed, not producing standalone executables.
Yes, you can - it's called disassembling, and allows you to look at the code of Safari perfectly well. The thing is, C, among other languages, compiles to native code, i.e. code that your CPU can "understand" and execute.
More or less obviously, the level of abstraction present in the instruction set of your CPU is much smaller than that of a high level language like Python. The CPU instructions are not concerned with "downloading that URI", but more "check if that bit is set in a hardware register".
So, in conclusion, the level of complexity present in a native application is much higher when looking at the machine code, so many people simply can't make any sense of what is going on there, it's hard to get the big picture. With experience and time at your hands, it is possible though - people do it all the time, reversing applications and all.
you can't open up and read the code that actually runs for python either. Try
import dis
def foo():
for i in range(100):
print i
print dis.dis(foo)
That will show you the (human readable) bytcode of the foo program. equivalently, you can save the file and import it from the interactive python interpreter. This will create a .pyc file with the same basename as the script. open that with a hex editor and you are looking at the actually python bytecode.
The reason for the difference is that python changes up it's byte code between releases so that you would either need to distribute a different version of a binary only release for each version of python. This would be a pain.
With C, it's compiled to native code and so the byte code is much more stable making binary only releases possible.
because C code is complied to object (machine) code and python code is compiled into an intermediate byte code. I am not sure if you are even referring to the byte code of python - you must be referring to the source file itself which is directly executable (hiding the byte code from you!). C needs to be compiled and linked.
Python scripts are parsed and converted to binary only when they're run - i.e., they're text files and you can read them with an editor.
C code is compiled and linked to an executable binary file before they can be run. Normally, only this executable binary file is distributed - hence you need a decompiler. You can always view the source code, if you've access to it.
Not all C programs require decompilers. There's lots of C code distributed in source form. And some Python programs do require decompilers, if distributed as bytecode (.pyc files).
But, to the extent that your assumptions are valid, it's because C is a compiled language while Python is an interpreted language.
Python scripts are analogous to a man looking at a to-do list written in English (or language he understands). The man has to do all the work, every time that list of things has to be done.
If the man, instead of doing the steps on his own each time, creates and programs a robot which can carry out those steps again and again (and probably faster than him), that robot is analogous to the C program.
The man in the python case is called the "interpreter" and in the C case is called the "compiler", and the C robot is called the compiled program/executable.
When you look at the python program source, you see the to-do list. In case of the robot, you see the gears, motors and batteries, etc, which look very different from the to-do list. If you could get hold of the C "to-do" list, it looks somewhat like the python code, just in a different language.
G-WAN executes ANSI C scripts on the fly -making it just like Python scripts.
This can be server-side scripts (using G-WAN as a Web server) or any general-purpose C program and you can link any existing library.
Oh, and G-WAN C scripts are much faster than Python, PHP or Java...