Processing background jobs in Ruby with Sidekiq

Who wants a Sidekiq? #

Sidekiq is a powerful Ruby gem that allows you to easily manage background jobs in your application. It is designed to be simple to use, efficient and reliable. With Sidekiq, you can easily perform tasks in the background, such as sending emails, running reports, processing data and execution of repetitive or scheduled tasks.

Sidekiq also provides a Web UI dashboard that allows you to monitor the status of your jobs and other metric information such as memory usage. You can view the number of jobs that are queued, running, completed or failed as well as view the job history and details, including information about exceptions or errors that appear.

Briefly about the architecture #

Sidekiq uses Redis as an in-memory data structure store and is written in Ruby. It also supports Java clients. It can be used with Resque, another Redis based job scheduler, or more commonly as a standalone product. Sidekiq reads jobs from a Redis queue, using the FIFO model to process jobs. Job processing is asynchronous and allows a web thread to serve requests rather than process heavy tasks.

What’s FIFO? #

FIFO stands for First In First Out. It is a method of organizing and processing data where the first item entered is the first item to be processed or removed. This approach is used as an operating system algorithm, which gives CPU time to every process in the order they arrive. The data structure that FIFO implements is Queue and time complexity of inserting element is O(1).

Redis - vibrant and open source #

Redis is an open source, in-memory data structure store used as a database, cache, and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets, bitmaps, hyperloglogs and geospatial indexes. It is written in ANSI C and works in most POSIX systems. Redis’ architecture is a client-server model, with the server written in C and clients in many languages. It is designed to be highly available, with automatic failover and replication.

How to use Sidekiq? #

Setup with Docker Compose #

1. First of all, we should add sidekiq gem to our Gemfile and run bundle install.

   # Gemfile.rb
   
     gem "sidekiq"

2. Now we should configure our docker-compose.yml file. Let’s add redis and sidekiq services.

# docker-compose.yml

  services:
    redis:
      image: redis
      volumes:
        - redis_data:/data

    sidekiq:
      build: .
      command: bundle exec sidekiq # to idle our Sidekiq when we start the application
      volumes:
        - .:/usr/src/app # mount root folder, not necessary

  # ...

  volumes:
    redis_data:
    # ...

3. We will use an ActiveJob interface to work with Sidekiq. ActiveJob is a wrapper around the most known background job gems providing a base API to work with background jobs. You can find more information about it in the official Rails documentation. If you want to use simple Sidekiq worker without ActiveJob you can skip this step.

So, add the following line inside the application.rb to use the Sidekiq adapter inside of our application.

# config/application.rb

  # ...
  class Application < Rails::Application
      config.active_job.queue_adapter = :sidekiq
      # ...
  end

4. And last but not least is a configuration file.

# config/initializers/sidekiq.rb

  Sidekiq.configure_server do |config|
    config.redis = { url: "redis://redis:6379/0" }
  end

  Sidekiq.configure_client do |config|
    config.redis = { url: "redis://redis:6379/0" }
  end

Now, our app is ready to perform any background task. Let’s give it a shot and try to create our own SuperJob!

Just create a new job/worker and call it in the operation or controller.

If you’re using ActiveJob wrapper:

# app/jobs/my_active_job.rb

  # frozen_string_literal: true

  class MyActiveJob < ApplicationJob
    queue_as :default

    def perform(method_name)
      puts "Hi, I'm #{method_name}."
    end
  end

If you’re using Sidekiq itself:

# app/workers/my_sidekiq_worker.rb

  # frozen_string_literal: true

  class MySidekiqWorker
    include Sidekiq::Worker
    sidekiq_options queue: "default"

    def perform(method_name)
      puts "Hi, I'm #{method_name}."
    end
  end

# app/controllers/some_controller.rb

  class SomeController < ApplicationController
    def index
      # some code
      MyActiveJob.perform_later("perform_later")
      # or
      MySidekiqWorker.perform_async("perform_async")
      # render some partial
    end
  end

There are several different approaches to call the job such as perform_now, perform_later and perform_in. But what’s the difference?

perfom_now #

# active_job method

  MyActiveJob.perform_now

It performs the job immediately. The job is not sent to the queue but directly executed by blocking the execution of others until it’s finished. For example, you can finalize registration of your user with creation of user’s profile and this operation cannot be processed in the background.

perform_later #

# active_job method

  MyActiveJob.perform_later(*args)

# sidekiq method

  MySidekiqWorker.perform_async(*args)

Which method you should use depends on how you have implemented your workers. If you use ActiveJob then you must use perform_later, if you just go with plain Sidekiq workers then use perform_async. Both methods just push new job to its queue. Job will be executed once queue is free.

perform_in #

perform_in method is an alias of perform_at, it is used for delayed execution of required jobs. For example, you can notify your user in an hour after registrations.

# sidekiq method

  MyActiveJob.perform_in(interval, *args)

Interval can be integer or float value in seconds. DateTime or Time formats are also acceptable. Under the hood interval value is coerced into the float value.

Note: if you’re trying to schedule your job to go now or in the past, it will be pushed into the Redis queue as a regular job.

Additionally, if you’d like to use Sidekiq Web UI (dashboards), you should add required path to the routes.rb and mount Sidekiq::Web

# routes.rb

  require 'sidekiq/web'

  Rails.application.routes.draw do
    # Mount Sidekiq Web UI to the project
    mount Sidekiq::Web => '/sidekiq'

    # ...
  end

Real-life examples #

Sidekiq can be a really powerful and efficient instrument to process large amounts of data. As a developer-life example, I can describe two recent cases when background processing was used with this gem.

The first case was connected to uploading and parsing large CSV and XLSX files consisting of thousands of rows with different information for CMS. This information should have been stored in the database after validation. By parallelizing the parsing and processing of this data with the usage of Sidekiq we were able to reduce update speed from several hours to ±30 minutes. Note, that meanwhile data were processed, user was able to continue their normal usage flow of the application. Uploading wasn’t dependent either on their internet connection or their presence in the application.

The second case was pretty unusual and related to the implementation of the real-time updates of the Virtual Machine resources usage with Web Sockets. To do such updates we used two different Sidekiq background jobs. One was used for getting information about currently running Virtual Machines and another - to replace partials with real-time data for those machines.

It’s difficult to imagine both of the cases mentioned above to be implemented without the usage of background processing jobs and Sidekiq especially.