开发者

How to shuffle jobs in a Resque queue?

开发者 https://www.devze.com 2023-03-28 11:38 出处:网络
I have a queue named check_integrity and lots of jobs in it. When i run a worker for it it takes jobs first in first out 开发者_如何学编程order. Is it possible to shuffle the jobs in that particular q

I have a queue named check_integrity and lots of jobs in it. When i run a worker for it it takes jobs first in first out 开发者_如何学编程order. Is it possible to shuffle the jobs in that particular queue? I need the worker to take jobs randomly. Please help.

Thanks.


One way to go about this is by popping entries out of the queue, batching them up, shuffling the batch and then re-insert them:

key = "resque:queue:bulk"
total = Redis.current.llen(key)
batch_size = 5_000 # any value that is good enough for you

batch = []
total.times do |i|
  entry = Redis.current.lpop(key)
  batch << entry
  if batch.size == batch_size || i.succ == total
    puts "re-inserting batch..."
    Redis.current.rpush key, batch.shuffle
    batch = []
  end
end

This is really useful when you mistakenly enqueue several jobs that end up racing for shared resources, locks and so on.


Look at this plugin for the Resque. I guess this is exactly what you need.


If you don't mind monkey patching resque then you can use this solution:

module Resque

  # Monkey patch Resque to handle queues as sets instead of lists. This allows
  # use to get jobs randomly rather then sequentially.
  def push(queue, item)
   watch_queue(queue)
   redis.sadd "queue:#{queue}", encode(item)
  end

  def pop(queue)
    decode redis.spop("queue:#{queue}")
  end

  def size(queue)
    redis.scard("queue:#{queue}").to_i
  end
end

If are using rails create a file inside the initializers with that code and you will be set.


you can use Delayed_job

0

精彩评论

暂无评论...
验证码 换一张
取 消