Wouldn't it be possible to have an OS entirely in Python if the Python VM itself is build into a hardware? Something like the good old Lisp Machine?
Suppose I have a cpu that is the hardware implementation of the python virtual machine, then all programs written in python would perform with the speed of assembly, won't it (but Python is mostly interpreted but we can compile it)?
If we have such a 'python-microprocessor', what about the memory and other subsystems? Would it be compatible with the current memory.
Is there any information on the registers and the Python VM architecture, something similar to what we have for 8086?
Wouldn't it be possible to have an OS
entirely in Python if the Python VM
itself is build into a hardware?
Something like the good old Lisp
Machine?
Yes, theoretically it would be possible.
Suppose I have a cpu that is the
hardware implementation of the python
virtual machine, then all programs
written in python would perform with
the speed of assembly, won't it (but
Python is mostly interpreted but we
can compile it)?
Python doesn't have a speed, it's a language. The speed of the interpreter (in this case the processor) can be tested. But just as it's difficult to compare the performance of a RISC and a CISC processor, comparing Assembly with Python will be difficult too.
If we have such a
'python-microprocessor', what about
the memory and other subsystems? Would
it be compatible with the current
memory.
The python microprocessor would have to do the memory management (and thus the garbage collection). Since that's normally done by the interpreter, now the microprocessor has to do it.
Is there any information on the
registers and the Python VM
architecture, something similar to
what we have for 8086?
Normally you don't access the memory directly in Python, so the registers shouldn't be relevant here.
Similar things were tried for Java, but none really took the world by storm.
Yeah, it might be possible, but designing new hardware is expensive. Would the return on investment justify building such a toy? I'd guess not, otherwise someone would have tried it by now. :)
Suppose I have a cpu that is the hardware implementation of the python virtual machine, then all programs written in python would perform with the speed of assembly, won't it (but Python is mostly interpreted but we can compile it)?
Yes it would be assembly speed. See this link for a comparison with an avr microcontroller assembly code. http://pycpu.wordpress.com/code-examples/speed-pycpu-vs-8bit-avr/.
It is a hardware implemntation of a cpu that can do very very limited python bytecode. But enought for ifs conditions and while loops with simple integers.
In the 70ties such ideas were quite popular. The idea was to close the semantic gap between compilers/virtual machines and instruction set architectures, and thereby bring programming languages and hardware closer together. However, when Patterson and Ditzel published The Case for the Reduced Instruction Set Computer (PDF, 672KB) and after the success of RISC and the microprocessor, the idea of closing the semantic gap was basically dead.
Now, with ever increasing transistor counts the idea may become interesting again. But, as others already noted, designing chips is costly. You need a very good reason to sink so much money. But it is definitely possible. IBM and Azul have shown this with their massively parallel Java Chips.
I guess you should call Google and convince them that they urgently need a Python processor. ;-)
New operating systems are interesting and cool, and basing one off of python would be cool. Then again, linux is so good and has so much development for it already. It would have to be the "right time".
Related
I need pypy to speed up my python code. While the pypy doesn't support a lot of modules I need (e.g. GNU Radio). Could I use pypy to speed up parts of my python code. Could I use pypy to only speed up some of my python files? How can I do that?
No, you can't. You can only have one interpreter instance running all of the code in a single program at a time. The exception is if you break out some of your functionality into a totally separate program that communicates with the other part of your code through some form of inter-process communication; then you can run those totally separate programs however you like. But for code that is not separated like that, it's not possible.
It will probably be more straightforward to adapt the entirety of your code to work with PyPy one way or another, instead of trying to break out bits and pieces. If that's absolutely not possible, then PyPy probably can't help you.
No, you can't. And GNU Radio does the signal processing and scheduling in C++, so that's totally opaque to your python interpreter. Also, GNU Radio itself is highly optimized and contains specialized implementations for most of the CPU intense tasks for SSE, SSE4, and some NEON.
I need pypy to speed up my python code.
I doubt that. If your program runs too slow, it's probably nothing your Python interpreter can solve -- you might have to look into what could take so much time, and solve this on a higher level.
I was looking at PyPy and I was just wondering why it hasn't been adopted into the mainline Python distributions. Wouldn't things like JIT compilation and lower memory footprint greatly improve the speeds of all Python code?
In short, what are the main drawbacks of PyPy that cause it to remain a separate project?
PyPy is not a fork of CPython, so it could never be merged directly into CPython.
Theoretically the Python community could universally adopt PyPy, PyPy could be made the reference implementation, and CPython could be discontinued. However, PyPy has its own weaknesses:
CPython is easy to integrate with Python modules written in C, which is traditionally the way Python applications have handled CPU-intensive tasks (see for instance the SciPy project).
The PyPy JIT compilation step itself costs CPU time -- it's only through repeated running of compiled code that it becomes faster overall. This means startup times can be higher, and therefore PyPy isn't necessarily as efficient for running glue code or trivial scripts.
PyPy and CPython behavior is not identical in all respects, especially when it comes to "implementation details" (behavior that is not specified by the language but is still important at a practical level).
CPython runs on more architectures than PyPy and has been successfully adapted to run in embedded architectures in ways that may be impractical for PyPy.
CPython's reference counting scheme for memory management arguably has more predictable performance impacts than PyPy's various GC systems, although this isn't necessarily true of all "pure GC" strategies.
PyPy does not yet fully support Python 3.x, although that is an active work item.
PyPy is a great project, but runtime speed on CPU-intensive tasks isn't everything, and in many applications it's the least of many concerns. For instance, Django can run on PyPy and that makes templating faster, but CPython's database drivers are faster than PyPy's; in the end, which implementation is more efficient depends on where the bottleneck in a given application is.
Another example: you'd think PyPy would be great for games, but most GC strategies like those used in PyPy cause noticeable jitter. For CPython, most of the CPU-intensive game stuff is offloaded to the PyGame library, which PyPy can't take advantage of since PyGame is primarily implemented as a C extension (though see: pygame-cffi). I still think PyPy can be a great platform for games, but I've never seen it actually used.
PyPy and CPython have radically different approaches to fundamental design questions and make different tradeoffs, so neither one is "better" than the other in every case.
For one, it's not 100% compatible with Python 2.x, and has only preliminary support for 3.x.
It's also not something that could be merged - The Python implementation that is provided by PyPy is generated using a framework they have created, which is extremely cool, but also completely disparate with the existing CPython implementation. It would have to be a complete replacement.
There are some very concrete differences between PyPy and CPython, a big one being how extension modules are supported - which, if you want to go beyond the standard library, is a big deal.
It's also worth noting that PyPy isn't universally faster.
See this video by Guido van Rossum. He talks about the same question you asked at 12 min 33 secs.
Highlights:
lack of Python 3 compatibility
lack of extension support
not appropriate as glue code
speed is not everything
After all, he's the one to decide...
One reason might be that according to PyPy site, it currently runs only on 32- and 64-bit Intel x86 architecture, while CPython runs on other platforms as well. This is probably due to platform-specific speed enhancements in PyPy. While speed is a good thing, people often want language implementations to be as "platform-independent" as possible.
I recommend watching this keynote by David Beazley for more insights. It answers your question by giving clarity on nature & intricacies of PyPy.
In addition to everything that's been said here, PyPy is not nearly as rock solid as CPython in terms of bugs. With SymPy, we've found at about a dozen bugs in PyPy over the past couple of years, both in released versions and in the nightlies.
On the other hand, we've only ever found one bug in CPython, and that was in a prerelease.
Plus, don't discount the lack of Python 3 support. No one in the core Python community even cares about Python 2 any more. They are working on the next big things in Python 3.4, which will be the fifth major release of Python 3. The PyPy guys still haven't gotten one of them. So they've got some catching up to do before they can start to be contenders.
Don't get me wrong. PyPy is awesome. But it's still far from being better than CPython in a lot of very important ways.
And by the way, if you use SymPy in PyPy, you won't see a smaller memory footprint (or a speedup either). See https://bitbucket.org/pypy/pypy/issues/1447/.
I am working with an ARM Cortex M3 on which I need to port Python (without operating system). What would be my best approach? I just need the core Python and basic I/O.
Golly, that's kind of a tall order. There are so many services of a kernel that Python depends upon, and that you'd have to provide yourself. I'd think you'd be far better off looking for a lightweight OS -- maybe Minix 3? -- to put on your embedded processor.
Failing that, I'd be horribly tempted to think about hand-translating to C and building the essentials on that.
You should definitely look at eLua:
http://www.eluaproject.net
"Embedded power, driven by Lua
Quickly prototype and develop embedded software applications with the power of Lua and run them on a wide range of microcontroller architectures"
There are a few projects that have attempted to port Python to the situation you mention, take a look at python-on-a-chip, PyMite or tinypy. These are aimed at lower power microcontrollers without an OS and tend to focus on slightly older versions of the Python language and reduced library support.
One possible approach is to build your own stack machine in software to interpret and execute Python byte code directly. Certainly not a porting job and quite labor-intensive to implement, but a self-contained Python byte code stack processor built for your embedded system gets you around needing an operating system.
Another approach is writing your own low level executive (one step below a general purpose OS) that contains the bare minimum in services that a core Python interpreter port requires. I am not certain if this is more or less labor intensive than building a stack processor.
I am not recommending either of these approaches - personally, I like Charlie Martin's Minix 3 approach best since it is a balanced requirements compromise. On the other hand, what I suggest might be interesting if your project absolutely requires Python without an operating system and if the project has an excellent time and money budget.
Update 5 Mar 2012: Given a strict adherence to your Python/No OS requirements, another possibility of a path to a solution may lie in using an OS-less Java VM (e.g., jnode, currently in beta) and use Jython to create Java byte code from Python. Certainly not an ideal off-the-shelf solution, and it does seem to meet an OS-less Python requirement.
Compile it to c :)
http://shed-skin.blogspot.com/
fyi I just ported CPython 2.7x to non-POSIX OS. That was easy.
You need write pyconfig.h in right way, remove most of unused modules. Disable unused features.
Then fix compile, link errors. Then it just works after fixing some simple problems on run.
If You have no some POSIX header, write one by yourself. Implement all POSIX functions, that needed, such as file i/o.
Took 2-3 weeks in my case. Although I have heavily customized Python core. Unfortunately cannot opensource it :(.
After that I think Python can be ported easily to any platform, that has enough RAM.
Why should I use VM, like Parrot, for a dynamic language I use (Python, Perl, ...) if I already have an interpreter? What can I potentially gain, for the cost of having different VM between my code and my machine, and by using a separate interpreter?
(I am new in VM issue, so maybe the answer is obvious)
EDIT
What's the benefit of Parrot VM for end-users?
Why should I use VM, like Parrot, for a dynamic language I use (Python, Perl, ...) if I already have an interpreter?
First, if you're starting a project, then you may not already have an interpreter.
However, assuming you have an interpreter and are considering whether to add functionality to it or rewrite it to use Parrot, the tradeoffs that come to mind are:
In general, Parrot is probably better tested than the interpreter in question (better optimizer, better garbage collector, etc.)
In general, Parrot's developers know more about cross-platform issues than run-of-the-mill programmers
In general, Parrot has solved most problems you're likely to run into
Parrot was designed with complete generality in mind, and that added a ton of complexity; you may not need the extra generality
Personally, Parrot's optimizer (and register-based design, largely to make optimizations easier) and well tested cross platform codebase would be enough to convince me.
Parsing the ASCII source code is slow. It is faster if the source file gets parsed once, and then the interpreter uses a binary structure. In Python this structure gets stored in .pyc files for fast reuse.
There are two steps:
Parse the source, create byte code
Run (interpret) the byte code.
This is used by e.g. scala: There is no scala-VM. Scala is just a new syntax. The scala compiler creates java byte code.
I have a python program that processes image frames with Python 2.7, PIL, OpenCV, and numpy/scipy. To the best of my knowledge, it does not maintain any lists of previous frame. Nevertheless, memory consumption increases steadily as the program processes more and more frames.
There are several good discussions of memory profiling solutions for Python, but they seem to focus on 32-bit or Linux solutions. What should I use with 64-bit Python 2.7 on Windows? Initial investigations suggest that the issue is with a C library. I am particularly interested in tools to help detect C library leaks or experience finding leaks in Python / OpenCV / PIL.
I've found the tools discussed here very helpful: http://mg.pov.lt/blog/hunting-python-memleaks.html
There is a version of his code here with some additions for measure numpy array sizes.
I had a similar sort of problem tracking down a severe memory leak in a numpy/scipy heavy code where none of the usual Python memory management tools and diagnostics detected the leak or hinted at its source.
In my case, the source of the leak was scipy interface code to the UMFPACK solver package, which was calling a C language initialization routine at every call of the interface object constructor, but never calling the de-initialization routine when the interface object was destroyed, resulting in scratch space and internal allocations leaking at the rate of about 15Mb a call. In an application with 10-20k calls, the impact was severe. Because the memory allocation was not done via the Python memory manager, things like heapy could not detect the leak.
I wound up having to use valgrind + "printf" style debugging to track down the culprit. You might need to look at non-python memory use analyzers and instrumentation tools to find out where the leak is coming from. I don't work in the Windows environment, and am not familiar with the standard toolchains, so I can't really suggest what to use. Perhaps someone else could chip in with some suggestions.
David Malcom did a talk this year at PyCon 2011, called Dude, Where's My RAM?. He talks about debugging memory usage in Python as well as showing a tool he developed for analyzing memory usage called gdb-heap which can track memory usage down to individual bytes. Really, really great talk. I would guess it'd be difficult to use gdb-heap on Windows (maybe useful to test on another platform and possibly debug there?), but the talk covers a lot of the common issues, resolutions, etc.