Fabric – Wrappers and more error handling

I wrote about Fabric’s error handling in this post and there task execution errors were handled like this:

env.warn_only=True

def cmd(cmd):
    if run(cmd).failed:
        sudo(cmd)

What if the task is a bit more complex and has multilple parts that can go wrong? You might want to abort the execution and do some kind of rollback action.

The task could be aborted by just using env.warn_only=False but then then the task would be aborted before we can do any rollback actions.

Without a wrapper it could be done like this:

env.warn_only=True

def multitask():
    task1()
    task2()
    task3()
    task4()

def task1():
    if sudo("apt-get update").failed:
        rollback()
        abort("Fail!")
    if sudo("apt-get -y install php5").failed:
        rollback()
        abort("Fail!")

def task2():
    if sudo("./myscript").failed:
        rollback()
        abort("Fail!")

def rollback():
    sudo("apt-get -y autoremove php5")
    sudo("rm -r mydir")

Well as you can see this is not very convenient. Instead, you should create a wrapper function for the error handling. This way we don’t need to write those “if run(something).failed” parts.

First we need to do a small import:

from contextlib import contextmanager

Then we can create the wrapper:

env.warn_only=False

@contextmanager
def rollbackwrap():
    try:
        yield
    except SystemExit:
        rollback()
        abort("Fail!")

Notice that now the warn_only setting is set to False. When the setting is set to False, Fabric will give us the SystemExit exception if there’s errors in the task execution. The wrapper will execute the rollback action when the SystemExit exception is raised. If warn_only is set to True, Fabric will only give a warning and not raise any exceptions.

Now we just need to wrap our task:

@task
def multitask():
    with rollbackwrap():
        task1()
        task2()
        task3()
        task4()

def task1():
    sudo("apt-get update")
    sudo("apt-get -y install php5")

Now if anything in one of the tasks fails, the rollback will be executed and then the Fabric execution will be aborted.

Wrappers like this can be of course used for more than just error handling. You can wrap all your settings in them:

@contextmanager
def settingwrap():
    with settings(warn_only=True, user="simo"):
        yield

@task
def deploy():
    with settingwrap():
        run("whoami")
        run("hostname")

Complete fabfile for this post:

from fabric.api import *
from contextlib import contextmanager

env.warn_only=False

@contextmanager
def rollbackwrap():
    try:
        yield
    except SystemExit:
        rollback()
        abort("Fail!")

@task
def multitask():
    with rollbackwrap():
        task1()
        task2()
        task3()
        task4()

def task1():
    sudo("apt-get update")
    sudo("apt-get -y install php5")

def task2():
    sudo("./myscript")

def task3():
    put("foo.conf","/tmp/")

def task4():
    sudo("service apache2 restart")

def rollback():
    sudo("apt-get -y autoremove php5")
    sudo("rm -r mydir")
    sudo("rm /tmp/foo.conf")

@contextmanager
def settingwrap():
    with settings(warn_only=True, user="simo"):
        yield

@task
def deploy():
    with settingwrap():
        run("whoami")
        run("hostname")

Also available here.

Tested with:
Ubuntu 12.04
Fabric 1.4.1

This blog post was inspired by a question on stackoverflow that can be found here:
http://stackoverflow.com/questions/10827112/handling-failures-with-fabric