Nomad Batch Queue

What is the Nomad Batch Queue?

Currently, when many batch workloads are queued in Nomad, the one to run next is chosen "at random" amongst other batch jobs of that same priority.

Nomad Batch Queues introduces a builtin mechanism for configuring internal queues for Nomad batch workloads in order to gain better control of which jobs run first when the number of batch workloads is greater than the amount of compute/memory available.

Refer to the ARCHITECTURE.md to learn about how batch queues integrate with Nomad's overall architecture.

In the future there may exist different queue implementations that have different ways of deciding which batch job to run, but currently the only implementation is the Dynamic Priority Queue.

How do I setup a Nomad Batch Queue?

This guide follows the f-batch-job-queue branch.

An example default scheduler config to configure a global cluster queue.

server {
  enabled = true
  ...
  default_scheduler_config {
    batch_queue {
      type         = "dynamic_priority"
      tenant_type  = "metadata"
      metadata_key = "id"
      config {
        calc_interval = "1m"
        max_age       = "12h"
        half_life     = "12h"
        max_size      = 10000
        age_weight    = 10
        usage_weight  = 10
        size_weight   = 10
      }
    }
  }
}

Refer to the examples directory for more server configurations and job examples.

What do these configuration parameters mean?

• type - This configures the queue implementation. The only type currently available is "dynamic_priority"

• tenant_type - Tenancy is a way to group jobs together for making scheduling decisions. The option are currently "namespace" or "metadata". Namespace groups jobs by their namespace, and metadata looks for a key in the top level job.metadata field.

• metadata_key - When tenant_type = "metadata" we need to tell nomad what key to look for in the metadata field.

• config - Different queue implementations may need different configuration. The config block is the way to configure each one.

Dynamic Priority Queue configuration:

• calc_interval - Each job in the queue has it's priority recalculated on a regular basis. This field configures how often that calculation takes place. Recalculating priorities involves locking the queue, so larger queues may want to have a longer calc_interval.

• max_age - Jobs that sit on the queue longer will have their priority adjusted. This field configures the max age at which the job has had it's age priority maximally adjusted and will not age any further.

• max_size - Jobs can have their priority adjusted base on how many resources they are reserving in the cluster. This field configures the max size in which the size priority has been maximally adjusted and will not increase/decrease and further.

• *_weight - The multiple to apply to a specific adjustment.

Calculations - Age/Size/Usage priorities are all clamped between 0..1. For example a job that has been on the queue for 6 hours and has a max_age of 12 will have an age adjustment of .5. If the age_weight is 20, that job will have 10 added to it's priority because .5 * 20 = 10

Usage - The dynamic priority queue keeps track of tenant usage, which is the combined CPU and Memory that batch workloads reserve. The usage adjustment is then calculated as a ratio of that job tenant's usage / total cluster usage. The more cluster usage a tenant has, the more/less priority is given to that job (a positive usage_weight means less priority, a negative weight means more priority).

The overall algorithm for calculating the dynamic priority follows the below calculation:

AdjustedPriority = Base Priority +
  (min(WorkloadAge, MaxAge) / MaxAge * AgeWeight) +
  ((1 - TenantUsage / TotalUsage) * UsageWeight) +
  ((1 – min(JobSize, Maxsize) / MaxSize) * SizeWeight)

Viewing the batch queue

The batch queue can be viewed by using the cli command nomad queue jobs to view the jobs currently queued, along with each job's adjusted priority and scoring for what dimensions are contributing to that adjustment:

Usage is the adjustment made for the amount of resource consumption a given tenant is using. Tenants which have consumed more resources will have jobs that are queued lower in priority if the usage_weight is set.

Age is the adjustment made for the amount of a time a job is waiting in the queue, if the age_weight is set. Jobs that have been waiting longer will be given higher priority.

Size is the adjustment made for the resource size of the job, if size_weight is set.

In addition to jobs, the tenants and their total usage of shared resources can be viewed by running nomad queue tenants