The title pretty much says it, but:
- I can see one of my celery tasks being skipped roughly every other time it should run. I can see this in both a debugger (the debugger only starts up every other time), and with a primitive/simple little logging module I wrote (it only logs anything every other time).
- I wrote a little Celery on_success handler that shows its 4 parameters, including task_id. I can see it logging the same task_id twice (everything else is the same too, even the timestamps), then 0 times, then twice, then 0 times.
The problem comes and goes. I don’t know why. It consistently does the above for a while, and then suddenly starts working again, with what seems to be no code or input changes.
I’ve got the task in question decorated the same as most of the other tasks in this program:
@asynchronous.task(base=TPMplus, bind=True, max_retries=None, reject_on_worker_lost=True)
…except that TPMplus, which just gives on_success and on_failure methods.
I need it to fire 100% of the times it’s called.
For CPython 2.7 I’m using:
celery==3.1.26.post2
- billiard [required: >=3.3.0.23,<3.4, installed: 3.3.0.23]
- kombu [required: >=3.0.37,<3.1, installed: 3.0.37]
- amqp [required: >=1.4.9,<2.0, installed: 1.4.9]
- anyjson [required: >=0.3.3, installed: 0.3.3]
- pytz [required: >dev, installed: 2018.3]
And for CPython 3.10 we’re using:
celery==5.2.3
├── billiard [required: >=3.6.4.0,<4.0, installed: 3.6.4.0]
├── click [required: >=8.0.3,<9.0, installed: 8.0.3]
├── click-didyoumean [required: >=0.0.3, installed: 0.3.0]
│ └── click [required: >=7, installed: 8.0.3]
├── click-plugins [required: >=1.1.1, installed: 1.1.1]
│ └── click [required: >=4.0, installed: 8.0.3]
├── click-repl [required: >=0.2.0, installed: 0.2.0]
│ ├── click [required: Any, installed: 8.0.3]
│ ├── prompt-toolkit [required: Any, installed: 3.0.30]
│ │ └── wcwidth [required: Any, installed: 0.2.5]
│ └── six [required: Any, installed: 1.10.0]
├── kombu [required: >=5.2.3,<6.0, installed: 5.2.4]
│ ├── amqp [required: >=5.0.9,<6.0.0, installed: 5.1.0]
│ │ └── vine [required: >=5.0.0, installed: 5.0.0]
│ └── vine [required: Any, installed: 5.0.0]
├── pytz [required: >=2021.3, installed: 2021.3]
├── setuptools [required: >=59.1.1,<59.7.0, installed: 59.6.0]
└── vine [required: >=5.0.0,<6.0, installed: 5.0.0]
We’re on some really old versions there, because we’re still porting from CPython 2.7 to CPython 3.10. But if changing those is likely to help, it’ll probably be worth it.
For this part of the system we’re using CPython 3.10, so those versions probably matter far more.
Any suggestions?
I like celery a lot.