batch processing

[tarekziade]

We need to be able to perform jobs by batches in a new project - that makes it possible to do bulk inserts in mysql etc.

right now retools uses BPOPL so it's one element each time.

here's one way to do batches:

• create one list per worker
• use BRPOPLPUSH and fill the worker-specific list with elements
• wait until the worker-specific list has the size of the batch, then push its content to the function and purge it
• use a timer to treat the list when len < batch_size for too long
• use another list to track all workers lists, so we can ventilate existing elements to a new worker when a worker goes down

What do you think ?

[jayd3e]

What did you end up doing for this, just out of curiosity? Have a few situations where this could be useful.

[bbangert]

I believe tarek has some additional wrappers on top of retools that handle something like this.