[ACCEPTED]-Recover from task failed beyond max_retries-django-celery

Accepted answer
Score: 18

The issue is that celery is trying to re-raise 5 the exception you passed in when it hits 4 the retry limit. The code for doing this 3 re-raising is here: https://github.com/celery/celery/blob/v3.1.20/celery/app/task.py#L673-L681

The simplest way around 2 this is to just not have celery manage your 1 exceptions at all:

def mytask():
    except Exception as e:
        except MaxRetriesExceededError:
Score: 16

You can override the after_return method 3 of the celery task class, this method is 2 called after the execution of the task whatever 1 is the ret status (SUCCESS,FAILED,RETRY)

class MyTask(celery.task.Task)

    def run(self, xml, **kwargs)
        #Your stuffs here

    def after_return(self, status, retval, task_id, args, kwargs, einfo=None):
        if self.max_retries == int(kwargs['task_retries']):
            #If max retries are equals to task retries do something
        if status == "FAILURE":
            #You can do also something if the tasks fail instead of check the retries



Score: 15

With Celery version 2.3.2 this approach 1 has worked well for me:

class MyTask(celery.task.Task):
    abstract = True

    def after_return(self, status, retval, task_id, args, kwargs, einfo):
        if self.max_retries == self.request.retries:
            #If max retries is equal to task retries do something

@task(base=MyTask, default_retry_delay=5, max_retries=10)
def request(xml):
    #Your stuff here
Score: 11

I'm just going with this for now, spares 2 me the work of subclassing Task and is easily 1 understood.

# auto-retry with delay as defined below. After that, hook is disabled.
@celery.shared_task(bind=True, max_retries=5, default_retry_delay=300)
def post_data(self, hook_object_id, url, event, payload):
    headers = {'Content-type': 'application/json'}
        r = requests.post(url, data=payload, headers=headers)
    except requests.exceptions.RequestException as e:
        if self.request.retries >= self.max_retries:
            log.warning("Auto-deactivating webhook %s for event %s", hook_object_id, event)
            return False
        raise self.retry(exc=e)
    return True

More Related questions