I am running into some errors when trying to set up my own client using jobqueue PBS Cluster instead of using a default local cluster (i.e., client = Client()).
When setting the default, my own modules were recognized, but I realized my workers in the PBS Cluster could not find them. This page and other research was helpful in understanding what I might be able to do.
I organized my modules into a package and used pip install -e . since I'll still be developing it. I confirmed my python environment site-packages directory has my package (via an .egg-link file).
I hoped installing the package would make my modules available, but I received the same error when I run my code after setting up a basic PBS Cluster:
cluster = PBSCluster(cores=x,memory=y)
cluster.scale(n)
client=Client(cluster)
Is my basic idea of installing the modules as a package not enough?
I looked into client.upload_file based on this answer as another means to make the reference to my module file explicit. Will I need to do something like this still to install modules directly on the workers?
Apologies for length, I am very new to both dask and operating on a HPC.
Thanks for any help.
First, just a sanity check: When using an HPC cluster, there is typically a shared filesystem, which all workers can access (and so can your client machine). Is that the case for your cluster? If so, make sure your conda environment is in a shared location that all workers can access.
I organized my modules into a package and used pip install -e .
That should work, as long as your source code is also on the shared filesystem. The directory pointed to by the .egg-link file should be accessible from the worker machines. Is it?
Related
I am currently working on a git repo located in
/home/user/bin/git/my-project
so i am saving the project as I work on it. But i have an issue, my project depends on python libraries, crontab configuration, bashrc/.profile variables and if i need to transfer all of this on another linux system I was wondering if it was possible to host the whole directory right from root? like:
git/home/user/bin/git/my-project
so that when i want to retrieve the exact same copy and update the second machine, it works exactly like the first? Is it possible? are there any downsides?
thanks in advance for your help
EDIT: Thanks for your answers, by root I meant the directory which does not have a parent directory. On my raspberry pi I probably meant the place where you find (etc, var, usr, boot, dev...). So how would you proceed, if you wanted to (for example) have another RPI with the exact same function without having to duplicate what's in the SD card? and everyone working on a different system have the exact same functionnality, up to date?
Thanks!
Downsides of making a git repo in the root (ie. in /)
Some IDE applications work horribly slow, because they "autodetect" if they are inside a git repo and then try to run git status to check for changes. This ends up indexing the whole filesystem. I remember a problem with eclipse and NetBeans (I think NetBeans didn't start at all, because it tried to open read-write some git files, and they were owned by root, but I'm not sure).
how would you proceed, if you wanted to (for example) have another RPI with the exact same function without having to duplicate what's in the SD card?
Create a package for the distribution installed on the platform. Create a publicly available repository with that package. Add that repository to the native package manager of that platform and install it like a normal package.
Package manager is the way of distributing stuff to systems. Package managers have dependency resolution, easy-updates, pre-/post-install scripts and hooks, all the functionality you need to distribute programs. And package managers have great support of removing files and updates.
my project depends on python libraries
So install them, via pip pyenv or the best via the package manager available on the system, the system administrators choice. Or request the user to setup a venv after installing your application.
, crontab configuration
(Side note: see systemd timers).
, bashrc/.profile variables
If your library depends on my home customization, you have zero chances of installing it on any of my systems. There is zero need to touch home directory stuff if you want to distribute a library to systems. Applications have /etc for configuration, and most distributions have /etc/profile.d/ as the standard directory for drop-in shell configuration. Please research linux directory structure, see Filesystem Hierarchy Specification and XDG specifications.
Still, requesting users to do manual configuration after installing a package is normal. Some applications to work need environmental variables.
If you want to propagate change to different computers and manage that change, then use ansible, puppet, chef and such automation tools.
I have a python app with its setup.py that's working just fine to install it through setuptools. I am then packaging it up in DEB and PKGNG using the excellent Effing package management. I've also made some quick tests with setuptools-pkg and that seems to work too.
Now I have a need to distribute the packages including init scripts to start/stop/manage the service. I have my init scripts in the source repo and, according to what seems to be best practice, I'm not doing anything with them in setuptools and I'm handling them in the os-specific packaging: for debian-based systems I use the --deb-init, --deb-upstart and --deb-systemd FPM options as needed.
How can I build a FreeBSD package that includes the correct rc.d script, using FPM or through any other means?
All the examples I've seen are adding the rc.d script when building a package through the ports collection but this is an internal app and is not going to be published to the Ports or on PyPi. I want to be able to check out the repository on a FreeBSD system, launch a command that gives me a package, distribute it to other FreeBSD systems, install it using pkg and have my init script correctly deployed to /usr/local/etc/rc.d/<myappname>. There's no need to keep using FPM for that, anything works as long as it gives me a well-formed package.
I would highly suggest creating your package as if it were any other port either if is going to be published or not.
One of the advantages you can inherit by doing this is that you could also include all your test and automate the deployment having out of the box the base for a continues integration/delivery setup.
Check out poudriere. You could indeed maintain a set of custom ports with your very own settings and distribute them across your environments without any hassle:
pkg install -r your-poudriere yourpkg
In case this is probably too much or probably doesn't adapt well to your use case, you can always fallback to ansible, in where you could create a custom rc.d within a template of an ansible role.
If you just want to build and deploy something, let's say a microservice, then probably pkg is not the best tool, maybe you just need a supervisor that can work on all your platforms (sysutils/immortal) so that you could just distribute your code and have a single recipe for starting/stoping the service.
nbari's answer is probably the Right Way™ to do this and I'd probably create my own "port" and use that to build the package on a central host.
At the time of my original question I had taken a different approach that I'm reporting here for the sake of completeness.
I am still building the applications package (ie. myapp-1.0.0.txz) with fpm -s python -t freebsd, which basically uses Python's setuptools infrastructure to get the necessary informations, and I don't include any rc.d file in it.
I also build a second package which I will call myapp-init-1.0.0.txz with the source directory type (ie. fpm -s dir -t freebsd) and I only include the init script in that package.
Both packages get distributed to hosts and installed, thus solving my distribution issue.
I have a GPU cluster with one storage-node and several computing nodes each has 8 GPU. I am configuring the cluster.
One of the task is to configure the python, what we need is several versions of Python and some python packages, and for some packages we may require several versions of it, such as different version of tensorflow.
So the question is how to configure the python and the packages so that it' convenient to use different version of the package I want to use.
I have installed both python2.7 and python3.6 in each computing node and in the storage node. But I think it is a good way if one has a huge amount of computing node to configure. One of the solution is to install python in the share directory of the cluster instead of the default /usr/local path.
Anyone has a better way to do this?
What I use now is OpenPBS(Torque) and I am new to HPC.
Thanks a lot.
You can install Modules software environment in a shared directory accessible on every node. Then it will be easy to load a specific version of python or TensorFlow:
module load lang/Python/3.6.0
module load lib/Tensorflow/1.1.0
Then, if for some packages we may require several versions of it, you can have a look at Python virtualenv that permits to install several version of the same package. To share it on all the nodes, consider to create your virtualenv on a shared mount point.
You could install each piece of software on the storage node under a certain directory and mount that directory on the compute nodes. Then you don't have to install each software several times.
A common solution to this problem are Environment Modules. You install your software as a module. This means that the software is installed in a certain directory (e.g /opt/modules/python/3.6/) together with a modulefile. When you do module load python/3.6, the modulefile sets environment variables such that Python3.6 is in PATH, PYTHONPATH, etc.
This results in a nice separation of your software stack and also enables you to install newer versions of tensorflow without messing up the environment.
I have been using Anaconda Python a lot but also have made many package upgrades (like PANDAS). I've written some tools that I want to turn into a MapReduce job and I've researched how to go about the python EMR bootstrapping for package dependencies.
I thought about a possible workaround: just getting and installing the Anaconda distribution. But then I remembered that I'd have to do all the necessary upgrading.
My last effort in possibly making this easy is this question: is there a way to "rebundle" the upgraded anaconda (or one its environments) so that it can be stored on S3 and used as in the EMR bootstrap action?
Thanks for any help!
ADDED: I suppose it would require a license to be able to wrap up an Anaconda distro like this and use it on various machines, be they in my office network or on AWS. Here's an open source version of this question (I just learned the main package manager to the Anaconda distro is actually OS):
Suppose I have a virtual (or conda) environment running with various modules and extensions installed. What is the proper way, if any, to encapsulate/bundle this virtual environment so that I can efficiently deploy it as needed? I have come across 'pip bundle' and there is 'conda clone' and 'conda create' as well. Also, there appears the concept of conda channels. It's just not clear to me if I can put these together for efficient deployment on EMR and if so, how.
The license allows you to do this, if that's what you're asking.
You might also look at http://continuum.io/anaconda-cluster and http://continuumio.github.io/conda-cluster/.
I'm trying to connect RabbitMQ to Graphite(0.9.9) using https://github.com/somic/graphite-rabbitmq
However, I'm not entirely sure which directory in Graphite the graphite-rabbitmq files should be placed.
When I run carbon-agent-rabbitm1.py I get
Failed to import the graphite package. Please verify that this package
was properly installed and that your PYTHONPATH environment variable
includes the directory in which it is installed.
For example, you may need to run the following command:
export PYTHONPATH="/home/myusername/lib/python/:$PYTHONPATH"
Help would be very much appreciated
The convention with python modules, you can just put the downloaded files in an arbitrary temp directory, cd into that directory, and run:
python setup.py install
The standard distutils package will do the work of making sure everything gets to the right place.
You may also want to download pip which will manage the process of downloading and installing these packages for you, in which case you can then just type:
pip install graphite-web
But aside from all that, you really can put the files anywhere, so long as you add the directory to an environment variable called PYTHONPATH, just like the error message you quoted says.
Unless you've got a specific reason for using the scripts in that github repo, I wouldn't as they appear to be about 3 years old and graphite now has support for RabbitMQ by way of the Twisted Python AMQP library (txamqp), which makes those scripts completely unnecessary.