Choice of parallelism in python [closed] - python

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
It seems like there are many options of parallelizing Python. I have seen these options below:
shared memory:
therading, multiprocessing, joblib, cython.parallel
distributed memory:
mpi4py, parallelpython (pp)
any CUDA, OpenCL options?
Does anyone have experiences in using these or other parallel libraries? How do they compare to each other? I am particularly interested in the use of python in computation-intensive applications in scientific computing field.

any CUDA, OpenCL options?
There is pyCUDA for CUDA anyway.
There is pyOpenCL also. (I'm less familiar with OpenCL, there may be others.)
There are pyCUDA and pyOpenCL tags here on SO.
pyCUDA and pyOpenCL are basically "wrappers" AFAIK, but it's unclear what exactly you're looking for -- your scope appears to be wide.

As far as I know, pyPar and/or pyMPI are the two most frequently used libraries for computation intensive applications in the scientific field.
pyPar tends to be easier to use, while pyMPI is more feature complete - so the first is used more frequently for less complex computations.
Iirc, they are just python wrappers for the relevant C libraries, making them the highest performing/most efficient ones to use.

Related

Why doesn't Python come pre-built with required libraries like pandas, numpy etc [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
What is the reason packages are distributed separately?
Why do we have separate 'add-on' packages like pandas, numpy?
Since these modules seem so important, why are these not part of Python itself?
Are the "single distributions" of Python to come pre-loaded?
If it's part of design to keep the 'core' separate from additional functionality, still in that case it should at least come 'pre-imported' as soon as you start Python.
Where can I find such distributions if they exist?
Many of these tools, including core Python, are separately developed and distributed by different team, so it is up to aggregators to curate them and put them into a single distribution. Here are some notable examples:
The Anaconda distribution from Continuum Analytics
The Canopy distribution from Enthought
ActivePython from ActiveState
PythonXY for scientific programming
WinPython for Windows
PyIMSL Studio from Rogue Wave Software
This is a bit like asking "Why doesn't every motor come with a car around it?"
While a car without a motor is pretty useless, the inverse doesn't hold: Most motors aren't even used for cars. Of course one could try selling a complete car to people who want to have a generator, but they wouldn't buy it.
Also the people designing cars might not be the best to build a motor and vice versa.
Similarly with python. Most python distributions are not used with numpy, scipy or pandas. Distributing python with those packages would create a massive overhead.
However, there is of course a strong demand for prebuilt distributions which combine those modules with a respective python and make sure everything interacts smoothly. Some examples are Anaconda, Canopy, python(x,y), winpython, etc. So an end user who simply wants a car that runs, best chooses one of those, instead of installing everything from scratch. Other users who do want to always have the newest version of everything might choose to tinker them together themselves.
You can make the interactive interpreted launch with "pre-imported" modules, as well as with pre-run code, using The Interactive start-up file.
Alternatively, you can use The Customization modules to pre-run code on every invocation of python.
Regarding whether pandas and numpy should be part of the standard library - it's a matter of opinion.
PyPi currently has over 100,000 libraries available. I'm sure someone thinks each of these is important.
Why do you need or want to pre-load libraries, considering how easy a pip install is especially in a virtual environment?

Schematic link between Python, C, Theano, CUDA, Windows 7, Machine Code, CPU, GPU and many more? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I do not have a thorough computer science background and I am trying to find my way trough deep learning. I was at the point that I wanted to make a decision on which framework I want to use to start coding some networks. Nevertheless, going through many discussions on fora and other websites raised another question that kept me busy for two days and which seems to be answered nowhere.
At this point I still can't get a complete understanding on how my Theano code in Python is finally executed on the CPU and GPU (is this "machine code"?).
As an answer I imagine a schematic view (of course can be in words) on the link between how all the parts play together in a Theano program, mainly:
my code
Theano
Python interpreter
C
NVCC
CPU
GPU
CUDA
Also I'd like to understand the role of Python bytecode in the context of Theano, and how it differs from the code (machine code?) that is acutally executed on the CPU or the GPU.
If it helps, I am working on a Windows 7 64 bit.

Python vs Perl for portability? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'm working on a large-ish project. I currently have some functional tests hacked together in shell scripts. They work, but I'd like to make them a little bit more complicated. I think it will be painful to do in bash, but easy in a full-blown scripting language. The target language of the project is not appropriate for implementing the tests.
I'm the only person who will be working on my branch for the foreseeable future, so it isn't a huge deal if my tests don't work for others. But I'd like to avoid committing code that's going to be useless to people working on this project in the future.
In terms of test harness "just working" for the largest number of current and future contributors to this project as possible, am I better off porting my shell scripts to Python or Perl?
I suspect Perl, since I know that the default version of Python installed (if any) varies widely across OSes, but I'm not sure if that's just because I'm less familiar with Perl.
Every modern Linux or Unixlike system comes with both installed; which you choose is a matter of taste. Just pick one.
I would say that whatever you choose, you should try to write easily-readable code in it, even if you're the only one who will be reading that code. Now, Pythonists will tell you that writing readable code is easier in Python, which may be true, but it's certainly doable in Perl as well.
This is more personal, but I always used python and will use it till something shows me it's better to use other. Its simple, very expansible, strong and fast and supported by a lot of users.
The version of python doesn't matter much since you can update it and most OSes have python 2.5 with expands the compatibility a lot. Perl is also included in Linux and Mac OS, though.
I think that Python will be good, but if you like perl and have always worked with it, just use perl and don't complicate your life.

Python code sharing and repositories? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I do scientific computing and am currently using MATLAB. I have however recently started learning Python. I find the MATLAB Central file exchange very helpful and people share a lot of their programs there.
Are there equivalent code repositories and places where people share the programs they have written for the Python community? Especially for engineering and scientific computing.
Since programming in Python revolves around using modules and packages, you might find Python's package index PyPI very useful. It allows you to easily integrate other people's code into your programs.
If you're more interested in finding out how some thing were implemented, then you might want to explore code on github. There is a ton of code out there, so you should be able to profit from it.
In case you're interested in figuring out how to use some library for scientific computing, you can find detailed examples for numpy, scipy, matplotlib and so on.
This might serve you as a good starting point for exploring Python code and examples.
The scipy Cookbook has small pieces of code, but they should only receive general-purpose and good-quality code:
http://wiki.scipy.org/Cookbook
Or the python package index:
https://pypi.python.org/pypi

How to modify a python code to a cython code? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
i have been working on a project on python for physics, using enthought canopy, i wished to enhance it by modifying it to a cython code. Could someone pleases tell me how to rewrite a python to cython codes in canopy? or do i need a separate software?
Just to add, it's probably better to use libraries which already incorporate Cython. NumPy for example, has virtually any array handling you can think of and has been optimized around things like matrix multiplication. Smart people have already done the work for you, so see if you can get that module to do what you need then as a last resort rewrite your code using Cython.
The Canopy Python distribution is bundled with Cython. However to use it, you will need a C compiler on your machine, which you may or may not already have. The Cython documentation has a good overview of Cython basics, including a brief description of where to get/find a C compiler for you operating system.

Categories