Ignore min-ready when at capacity

When it looks like we're at capacity, don't use min-ready nodes to
calculate demand.  Requesting min-ready when over-capacity results in
bad allocation choices; nodes with high min-ready but no real work to
do will be granted their inflated allocations.

Move the provider calculation above the demand calculations so we can
gauge how many available nodes a label will be possibly be allocated.
Clarify the operation in the documentation slightly.

Change-Id: Ifa62dd152ddeaf47aea224f9b7049aeab8b0970d

doc/source/configuration.rst

@@ -134,10 +134,11 @@ providers or images are used to create them).  Example::
         - name: provider1
 
 The `name` and `image` keys are required.  The `providers` list is
-also required if any nodes should actually be created (e.g., the
-label is not currently disabled). The `min-ready` key is optional
-and defaults to 2. If the value is -1 the label is considered
-disabled.
+also required if any nodes should actually be created (e.g., the label
+is not currently disabled). The `min-ready` key is optional and
+defaults to 2. If the value is -1 the label is considered disabled.
+``min-ready`` is best-effort based on available capacity and is not a
+guaranteed allocation.
 
 The `subnodes` key is used to configure multi-node support.  If a
 `subnodes` key is supplied to an image, it indicates that the specified

nodepool/nodepool.py

@@ -1453,22 +1453,6 @@ class NodePool(threading.Thread):
                 count += 1 + len(n.subnodes)
             return count
 
-        # Actual need is demand - (ready + building)
-        for label in self.config.labels.values():
-            start_demand = label_demand.get(label.name, 0)
-            min_demand = start_demand + label.min_ready
-            n_ready = count_nodes(label.name, nodedb.READY)
-            n_building = count_nodes(label.name, nodedb.BUILDING)
-            n_test = count_nodes(label.name, nodedb.TEST)
-            ready = n_ready + n_building + n_test
-            demand = max(min_demand - ready, 0)
-            label_demand[label.name] = demand
-            self.log.debug("  Deficit: %s: %s (start: %s min: %s ready: %s)" %
-                           (label.name, demand, start_demand, min_demand,
-                            ready))
-
-        # Start setting up the allocation system.
-
         # Add a provider for each node provider, along with current
         # capacity
         allocation_providers = {}
@@ -1484,6 +1468,32 @@ class NodePool(threading.Thread):
             ap = allocation.AllocationProvider(provider.name, available)
             allocation_providers[provider.name] = ap
 
+        # calculate demand for labels
+        # Actual need is demand - (ready + building)
+        for label in self.config.labels.values():
+            start_demand = label_demand.get(label.name, 0)
+
+            # ignore min-ready if it doesn't look like we'll have the
+            # capacity to deal with it.
+            capacity = 0
+            for provider in label.providers.values():
+                capacity += allocation_providers[provider.name].available
+            if capacity < label.min_ready:
+                min_demand = start_demand
+            else:
+                min_demand = start_demand + label.min_ready
+
+            n_ready = count_nodes(label.name, nodedb.READY)
+            n_building = count_nodes(label.name, nodedb.BUILDING)
+            n_test = count_nodes(label.name, nodedb.TEST)
+            ready = n_ready + n_building + n_test
+            demand = max(min_demand - ready, 0)
+            label_demand[label.name] = demand
+            self.log.debug("  Deficit: %s: %s "
+                           "(start: %s min: %s ready: %s capacity: %s)" %
+                           (label.name, demand,
+                            start_demand, min_demand, ready, capacity))
+
         # "Target-Label-Provider" -- the triplet of info that identifies
         # the source and location of each node.  The mapping is
         # AllocationGrantTarget -> TargetLabelProvider, because