We Keep Coding

How to run the same task in parallel on multiple hosts with different parameters using fabric?

demo.py
from fabric.api import env, run,execute
env.hosts = ['10.1.1.100','10.1.1.200']
env.remotePath = {'10.1.1.100':'/home','10.1.1.200':'/var'}
env.parallel=True
def mytask(remotePath):
run('ls %s' % remotePath)
def test():
execute(mytask,env.remotePath[env.host])
fab -f demo.py test
I want to execute command ls /home at 10.1.1.100,and ls /var at 10.1.1.200 in parallel using #parallel decorator,is there any way to make it possible?

Use host_string to get the current host, then the command/argument you want to use.
#parallel
def mytask():
host_ip = env.host_string
remote_path = env.remotePath[host_ip]
run('ls %s' % remote_path)
According to Fabric's API doc: host_string
Defines the current user/host/port which Fabric will connect to when
executing run, put and so forth. This is set by fab when iterating
over a previously set host list, and may also be manually set when
using Fabric as a library.
Hope this helps :)

Fabric check user has permissions on file/folder

Is there an equivalent in fabric to os.access('/path/to/folder', os.W_OK) to check whether a remote folder has correct permissions given an remote user/group?
Currently I could try getting into the folder with with cd and catching the exception but I dont like that way...

You cannot use the os lib for obvious reasons; but you can use test. I made a quick 2-min example on how to use it.
from fabric.api import env, task, run, sudo as _sudo, settings, hide
env.user = 'vagrant'
env.key_filename = '~/.vagrant/machines/default/virtualbox/private_key'
env.host_string = '127.0.0.1'
env.port = '2222'
def is_file_writable(filepath, sudo=False):
fn = run if not sudo else _sudo
with settings(warn_only=True), hide('everything'):
response = fn('test -w ' + filepath)
return response.return_code == 0
#task
def sometask():
print is_file_writable('/etc/sudoers')
print is_file_writable('/etc/sudoers', sudo=True)
output:
$ fab sometask
False
True
Done.
Disconnecting from 127.0.0.1:2222... done.

How to prevent fabric form waiting for the process to return

I always use fabric to deploy my processes from my local pc to remote servers.
If I have a python script like this:
test.py:
import time
while True:
print "Hello world."
time.sleep(1)
Obviously, this script is a continuous running script.
And I deploy this script to remote server and execute my fabric script like this:
...
sudo("python test.py")
The fabric will always wait the return of test.py and won't exit.How can I stop the fabric script at once and ignore the return of test.py

Usually for this kind of asynchronous task processing Celery is preferred .
This explains in detail the use of Celery and Fabric together.
from fabric.api import hosts, env, execute,run
from celery import task
env.skip_bad_hosts = True
env.warn_only = True
#task()
def my_celery_task(testhost):
host_string = "%s#%s" % (testhost.SSH_user_name, testhost.IP)
#hosts(host_string)
def my_fab_task():
env.password = testhost.SSH_password
run("ls")
try:
result = execute(my_fab_task)
if isinstance(result.get(host_string, None), BaseException):
raise result.get(host_string)
except Exception as e:
print "my_celery_task -- %s" % e.message

sudo("python test.py 2>/dev/null >/dev/null &")
or redirect the output to some other file instead of /dev/null

This code worked for me:
fabricObj.execute("(nohup python your_file.py > /dev/null < /dev/null &)&")
Where fabricObj is an object to fabric class(defined internally) which speaks to fabric code.

Passing a Fabric env.hosts sting as a variable is not work in function

Passing a Fabric env.hosts sting as a variable is not work in function.
demo.py
#!/usr/bin/env python
from fabric.api import env, run
def deploy(hosts, command):
print hosts
env.hosts = hosts
run(command)
main.py
#!/usr/bin/env python
from demo import deploy
hosts = ['localhost']
command = 'hostname'
deploy(hosts, command)
python main.py
['localhost']
No hosts found. Please specify (single) host string for connection:
But env.host_string works!
demo.py
#!/usr/bin/env python
from fabric.api import env, run
def deploy(host, command):
print host
env.host_string = host
run(command)
main.py
#!/usr/bin/env python
from demo import deploy
host = 'localhost'
command = 'hostname'
deploy(host, command)
python main.py
localhost
[localhost] run: hostname
[localhost] out: heydevops-workspace
But the env.host_string is not enough for us, it's a single host.
Maybe we can use env.host_string within a loop, but that's not good.
Because we also want to set the concurrent tasks number and run them parallelly.
Now in ddep(my deployment engine), I only use MySQLdb to get the parameters then execute the fab command like:
os.system("fab -f service/%s.py -H %s -P -z %s %s" % (project,host,number,task))
This is a simple way but not good.
Because if I use the fab command, I can't catch the exceptions and failures of the results in Python, to make my ddep can "retry" the failed hosts.
If I use the "from demo import deploy", I can control and get them by some codes in Python.
So now "env.host " is the trouble. Can somebody give me a solution?
Thanks a lot.

Here's my insight.
According to docs, if you're calling fabric tasks from python scripts - you should use fabric.tasks.execute.
Should be smth like this:
demo.py
from fabric.api import run
from fabric.tasks import execute
def deploy(hosts, command):
execute(execute_deploy, command=command, hosts=hosts)
def execute_deploy(command):
run(command)
main.py
from demo import deploy
hosts = ['localhost']
command = 'hostname'
deploy(hosts, command)
Then, just run python main.py. Hope that helps.

Finally, I fixed this problem by using execute() and exec.
main.py
#!/usr/bin/env python
from demo import FabricSupport
hosts = ['localhost']
myfab = FabricSupport()
myfab.execute("df",hosts)
demo.py
#!/usr/bin/env python
from fabric.api import env, run, execute
class FabricSupport:
def __init__(self):
pass
def hostname(self):
run("hostname")
def df(self):
run("df -h")
def execute(self,task,hosts):
get_task = "task = self.%s" % task
exec get_task
execute(task,hosts=hosts)
python main.py
[localhost] Executing task 'hostname'
[localhost] run: hostname
[localhost] out: heydevops-workspace

How can I properly set the `env.hosts` in a function in my Python Fabric `fabfile.py`?

When I run this fabfile.py...
from fabric.api import env, run, local, cd
def setenv(foo):
env.hosts = ['myhost']
def mycmd(foo):
setenv(foo)
print(env.hosts)
run('ls')
with this command fab mycmd:bar. I get this output...
['myhost']
No hosts found. Please specify (single) host string for connection:
What, what?! I don't get it? I've set the env.hosts and it seems to be valid "inside" the mycmd function, but for some reason that run command doesn't know about the hosts I've specified.
Color me confused. Any help would be appreciated!

#Chris, the reason you're seeing this behavior is because the host list is constructed before the task function is called. So, even though you're changing env.hosts inside the function, it is too late for it to have any effect.
Whereas the command fab setenv:foo mycmd:bar, would have resulted in something you would have expected:
$ fab setenv:foo mycmd:bar
[myhost] Executing task 'mycmd'
['myhost']
[myhost] run: ls
This is the same as the accepted answer, but because of the way setenv is defined, an argument is needed.
Another example:
from fabric.api import env, run, local, cd
env.hosts = ['other_host']
def setenv(foo):
env.hosts = ['myhost']
def mycmd(foo):
setenv(foo)
print('env.hosts inside mycmd: %s' % env.hosts)
run('ls')
The output of this is:
$ fab mycmd:bar
[other_host] Executing task 'mycmd'
env.hosts inside mycmd: ['myhost']
[other_host] run: ls
Fatal error: Name lookup failed for other_host
Underlying exception:
(8, 'nodename nor servname provided, or not known')
Aborting.
As you can see, the host-list is already set to ['other_host', ] when fabric starts to execute mycmd.

The way you are doing it is not normally how I would use Fabric.
from fabric.api import *
def hostname():
env.hosts = ['myhosts']
def mycmd():
print env.hosts
run('ls -l')
To run this I would then do
fab hostname mycmd
this allows you to seperate which host/hosts you want to perform the command on.
hope it helps.

Have you tried to used the hosts decorator?
from fabric.api import env, run, hosts
#hosts('myhost')
def mycmd(foo):
print(env.hosts)
run('ls')

I know this question is super old, but just in case someone stumbles across this, I have found that you don't need to call this as a fab file per se (your file doesn't need to be called "fabfile.py" and command doesn't need to be fab setenv(foo) mycmd(bar). Since you are importing the needed fab elements, you can call the file anything you want (let's call it "testfile.py") and simply use the execute function in the file. That would make your command python testfile.py.
Inside the testfile, set everything up like you would normally, but start the function using the execute keyword. Your file would look something like this:
from fabric.api import env, run
def setenv(foo):
env.hosts = ['myhost']
execute(mycmd, bar)
def mycmd(bar):
run('ls')
setenv(foo)
** It's important to note that the execute command does look like a regular function call. It will call the function and send the arguments in a single comma separated line. You can find more information here
So you'd start your program which would first call setenv, then setenv would execute the mycmd function. With this, you can also set multiple hosts within the same array. Something like:
env.hosts=['myhost1','myhost2','myhost3']

I have figured out how to make it work:
from fabric.api import env, run, local, cd
def setenv(foo):
env.hosts = ['myhost']
return env
def mycmd(foo):
env = setenv(foo)
print(env.hosts)
run('ls')