Is it possible to extend CPython with pypy to make use of the C modules in CPython and to write fast number crunching calculations in PyPy for its convince and ease of use. If so how can it be done?
I have searched the web throughly before asking to make sure i do not waste anyone's time but extending CPython with PyPy sounds like a good idea.
I read somewhere you could run python without an OS. How would I do this? Would I need to compile it? Can I run it raw? And if I did need to compile it, what tool would I use and what format would I compile it to?
As far as I know there's not really any way to do this easily but I could be wrong. There are "portable" versions of python but these are operating system dependent. I think what you're referencing is some guys at PyCon managed to run python from the GRUB Bootloader. Your best bet would be installing some minimalist Linux distribution, with essentially only Python and some core packages required to run. The problem is that there's a lot of types of hardware out there, all with their own drivers and assembly language. Python can work as a low level language when you need it to but it seems like configuration would be a nightmare. I haven't looked into it super thoroughly but it seems difficult and impractical. Having an OS above python gives you access to the package managers IDEs and compilier options that make python worth using.
yea that's one of the options, pretty much all the "light" distros will be similar, if you want more to try out try here. Not sure why you're worried about speed though, if you're having speed issues it's far more likely to be the IDE you're using or your code bogging down the computer, not any sort of compiler issues.
I'll preface this by saying I am quite new to PyPy, though fairly experienced with Python.
I'm looking to run a web app where I run untrusted Python code. The PyPy sandboxing features look ideal for what I'm doing.
The PyPy docs on sandboxing indicate that you can call a PyPy sandbox from either Python or PyPy. This seems to imply that there's some separate program or executable that is the sandbox.
I'm wondering, is it possible to call a PyPy sandbox from a non-Python language? I'm looking at Haskell in particular, but it's also very possible that I could use C or C++ as an intermediate.
Yes, that's possible. The PyPy sandbox is a separate process communicating only via stdin/stdout. If you want to rewrite the "external" part, you can; it's not using anything that should be too heavily Python-related.
Note that the sandboxing feature of PyPy is not being maintained any more, see http://www.pypy.org/features.html#sandboxing
I was looking at PyPy and I was just wondering why it hasn't been adopted into the mainline Python distributions. Wouldn't things like JIT compilation and lower memory footprint greatly improve the speeds of all Python code?
In short, what are the main drawbacks of PyPy that cause it to remain a separate project?
PyPy is not a fork of CPython, so it could never be merged directly into CPython.
Theoretically the Python community could universally adopt PyPy, PyPy could be made the reference implementation, and CPython could be discontinued. However, PyPy has its own weaknesses:
CPython is easy to integrate with Python modules written in C, which is traditionally the way Python applications have handled CPU-intensive tasks (see for instance the SciPy project).
The PyPy JIT compilation step itself costs CPU time -- it's only through repeated running of compiled code that it becomes faster overall. This means startup times can be higher, and therefore PyPy isn't necessarily as efficient for running glue code or trivial scripts.
PyPy and CPython behavior is not identical in all respects, especially when it comes to "implementation details" (behavior that is not specified by the language but is still important at a practical level).
CPython runs on more architectures than PyPy and has been successfully adapted to run in embedded architectures in ways that may be impractical for PyPy.
CPython's reference counting scheme for memory management arguably has more predictable performance impacts than PyPy's various GC systems, although this isn't necessarily true of all "pure GC" strategies.
PyPy does not yet fully support Python 3.x, although that is an active work item.
PyPy is a great project, but runtime speed on CPU-intensive tasks isn't everything, and in many applications it's the least of many concerns. For instance, Django can run on PyPy and that makes templating faster, but CPython's database drivers are faster than PyPy's; in the end, which implementation is more efficient depends on where the bottleneck in a given application is.
Another example: you'd think PyPy would be great for games, but most GC strategies like those used in PyPy cause noticeable jitter. For CPython, most of the CPU-intensive game stuff is offloaded to the PyGame library, which PyPy can't take advantage of since PyGame is primarily implemented as a C extension (though see: pygame-cffi). I still think PyPy can be a great platform for games, but I've never seen it actually used.
PyPy and CPython have radically different approaches to fundamental design questions and make different tradeoffs, so neither one is "better" than the other in every case.
For one, it's not 100% compatible with Python 2.x, and has only preliminary support for 3.x.
It's also not something that could be merged - The Python implementation that is provided by PyPy is generated using a framework they have created, which is extremely cool, but also completely disparate with the existing CPython implementation. It would have to be a complete replacement.
There are some very concrete differences between PyPy and CPython, a big one being how extension modules are supported - which, if you want to go beyond the standard library, is a big deal.
It's also worth noting that PyPy isn't universally faster.
See this video by Guido van Rossum. He talks about the same question you asked at 12 min 33 secs.
Highlights:
lack of Python 3 compatibility
lack of extension support
not appropriate as glue code
speed is not everything
After all, he's the one to decide...
One reason might be that according to PyPy site, it currently runs only on 32- and 64-bit Intel x86 architecture, while CPython runs on other platforms as well. This is probably due to platform-specific speed enhancements in PyPy. While speed is a good thing, people often want language implementations to be as "platform-independent" as possible.
I recommend watching this keynote by David Beazley for more insights. It answers your question by giving clarity on nature & intricacies of PyPy.
In addition to everything that's been said here, PyPy is not nearly as rock solid as CPython in terms of bugs. With SymPy, we've found at about a dozen bugs in PyPy over the past couple of years, both in released versions and in the nightlies.
On the other hand, we've only ever found one bug in CPython, and that was in a prerelease.
Plus, don't discount the lack of Python 3 support. No one in the core Python community even cares about Python 2 any more. They are working on the next big things in Python 3.4, which will be the fifth major release of Python 3. The PyPy guys still haven't gotten one of them. So they've got some catching up to do before they can start to be contenders.
Don't get me wrong. PyPy is awesome. But it's still far from being better than CPython in a lot of very important ways.
And by the way, if you use SymPy in PyPy, you won't see a smaller memory footprint (or a speedup either). See https://bitbucket.org/pypy/pypy/issues/1447/.
I'm having some trouble with distributing my python programs. for linux it's easy enough, just tell people which packages they need and provide the source. for windows however it's a different story altogether. I can get my program compiled with py2exe. the problem there however is that it always seems to miss one dll file or the other.
my question therefore is, is there some easy way to package a python script for multiple platforms? right now I keep having to go on a wild goosechase after dll files I don't have. and this isn't even mentioning the 32bit vs. 64bit problem which leaves a lot to desire.
any and all pointers are welcome. the ideal solution would be some sort of script that would create two zipfiles. one being linux/source and one for windows64bit/windows32bit. I really don't care all that much about mac support, but it would be welcome.
If you support linux, you probably support mac in the same way.
I typically use py2exe to distribute to windows, but there are alternatives, such as cxfreeze, and others. I haven't bothered to care much about 64/32 bit and just deliver a 32 bit solution.
Here are a list of packagers:
http://www.freehackers.org/Packaging_a_python_program