[ACCEPTED]-Is Celery as efficient on a local system as python multiprocessing is?-celery

Accepted answer
Score: 33

I just finished a test to decide how much 21 celery adds as overhead over multiprocessing.Pool and shared 20 arrays. The test runs the wiener filter 19 on a (292, 353, 1652) uint16 array. Both 18 versions use the same chunking (roughly:divide 17 the 292,353 dimensions by the square root 16 of the number of available cpu's). Two celery 15 versions were tried: one solution sends 14 pickled data the other opens the underlying 13 data file in every worker.

Result: on my 12 16 core i7 CPU celery takes about 16s, multiprocessing.Pool with 11 shared arrays about 15s. I find this difference 10 surprisingly small.

Increasing granularity 9 increases the difference obviously (celery 8 has to pass more messages): celery takes 7 15 s, multiprocessing.Pool takes 12s.

Take into account that 6 celery workers were already running on the 5 host whereas the pool workers are forked 4 at each run. I am not sure how could I start 3 multiprocessing pool at the beginning since 2 I pass the shared arrays in the initializer:

with closing(Pool(processes=mp.cpu_count(), initializer=poolinit_gen, initargs=(sourcearrays, resarrays))) as p:

and 1 only the resarrays are protected by locking.

Score: 8

I have actually never used Celery, but I 25 have used multiprocessing.

Celery seems to 24 have several ways to pass messages (tasks) around, including 23 ways that you should be able to run workers 22 on different machines. So a downside might 21 be that message passing could be slower 20 than with multiprocessing, but on the other 19 hand you could spread the load to other 18 machines.

You are right that multiprocessing 17 can only run on one machine. But on the 16 other hand, communication between the processes 15 can be very fast, for example by using shared 14 memory. Also if you need to process very 13 large amounts of data, you could easily 12 read and write data from and to the local 11 disk, and just pass filenames between the 10 processes.

I don't know how well Celery would 9 deal with task failures. For example, task 8 might never finish running, or might crash, or 7 you might want to have the ability to kill 6 a task if it did not finish in certain time 5 limit. I don't know how hard it would be 4 to add support for that if it is not there.

multiprocessing 3 does not come with fault tolerance out of 2 the box, but you can build that yourself 1 without too much trouble.

More Related questions