It enables reclamation of resources from queues exceeding their FairShare without waiting for queues to release resources. Preemption allows this imbalance to be adjusted in a predictable manner. Until then, tenant3 will have unmet demand or “starvation” in form of pending containers. ![]() However, application(s) in tenant3 queue will have to wait for applications in tenant1 and/or tenant2 to release resources. Subsequently, if tenant3 also becomes active, Instantaneous FairShare of tenants will change to 25%, 50% and 25% respectively. In the previous example, let’s say only tenant1 and tenant2 queues were active and used all of 33.3% and 66.6% of resources respectively. Starvation and Preemption are related concepts and are best discussed together.ĭue to the elasticity feature in FairScheduler, applications running in a queue can keep other applications (running in the same queue or a different queue) in a starved state. MinShare Starvation is configured as discussed in the following section.MinShare is only enforced when all of the following are true: It is important to note that this does not mean that minimum amount of resources are reserved for this queue. This is also referred to as MinShare of the queue. Minimum ShareĪ queue can specify that it requires a minimum amount of resources. In this post, unless otherwise qualified, FairShare means Instantaneous FairShare. It is important to determine which FairShare is being referred to in any discussion. Note that the term “FairShare” is generally loosely used to refer to either type of FairShare. In the above example, if only tenant1 and tenant2 queues are active, then the Instantaneous FairShare will be computed as follows: The two values are equal when all queues are active. This value differs from Steady FairShare in that inactive queues are not assigned any Instantaneous FairShare. To enforce Instantaneous FairShare, you must enable preemption and configure FairShare Starvation as discussed in the next section. This does not mean that this amount of resources are reserved for this queue. Instantaneous FairShare is the calculated FairShare value for each queue based on active queues only. An active queue is one with at least one running application, whereas an inactive queue is one with no running applications. In order to enable elasticity in a multi-tenant cluster, FairScheduler allows queues to use more resources than their Steady FairShare when a cluster has idle resources. If three tenant queues need to be configured, then Steady FairShare is calculated based on assigned weights as follows: Instantaneous FairShare In other words, it is the theoretical FairShare value of a queue.įor example, let’s say YARN is managing a total of 12 vCores and 24GB memory. This number only provides visibility into resources a queue can expect, and is not used in scheduling decisions, including preemption. The Steady FairShare for a queue is calculated as follows: The Steady FairShare is calculated at queue level and for the root queue it is the equivalent to all the cluster resources. ![]() This amount is known as Steady FairShare. The weight associated with a queue determines the amount of resources a queue deserves in relation to its sibling queues. Queues are sibling queues when they have the same parent. The FairScheduler uses hierarchical queues. Definitionsīefore we begin, it is important to define some key FairScheduler queue properties. We also present a recent overhaul of FairScheduler Preemption in CDH 5.11 which attempts to address a number of issues as documented in YARN-4752. In this post we discuss technical details around how FairScheduler Preemption works and best practices to consider when configuring it. ![]() The multi-part blog post Untangling Apache Hadoop YARN provided an overview of how the YARN scheduler works.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |