libstore/build: allow preferLocalBuild = true to bypass max-jobs = 0

When `max-jobs = 0`, our async semaphores have null capacity and
therefore no slot token can be acquired.

In addition, a derivation with `preferLocalBuild = true` with `max-jobs
= 0` which is not a built-in derivation is not considered suitable for
being possible to build.

Our documentation in `max-jobs` writes:

> `0` is useful when using remote builders to prevent any local builds
(except for preferLocalBuild derivation attribute which executes locally
regardless).

A valid interpretation is that `preferLocalBuild` derivations (all of
them) will always execute even if `max-jobs = 0`.

This CL achieves this promise by enabling `preferLocalBuild = true`
derivations to draw in priority capacity from a `preferLocalBuild`-only
pool and then fall back to the build pool (`maxBuildJobs`).

If there's non-zero max-jobs, we make the `preferLocalBuild`-only pool
empty.

If there's zero max-jobs, we make the `preferLocalBuild`-only pool
non-empty (capacity: 1).

Remark 1: if `max-jobs = 0`, no non-`preferLocalBuild = true`
derivations can be built without a remote builder and `preferLocalBuild
= true` derivations can be built one at a time if `extra-local-jobs` is
not overridden.

Remark 2: if `max-jobs != 0`, no `preferLocalBuild = true` derivations
can bypass capacity limits, i.e. `max-jobs` unless `extra-local-jobs` is
overridden.

Fixes #855.

Change-Id: I00a0acac7375a881aec6281ebb13a4530b26e460
Signed-off-by: Raito Bezarius <raito@lix.systems>

lix/libstore/build/local-derivation-goal.cc

@@ -205,10 +205,31 @@ retry:
 
     if (!slotToken.valid()) {
         outputLocks.reset();
-        if (worker.localBuilds.capacity() > 0) {
-            slotToken = co_await worker.localBuilds.acquire();
+
+        // NOTE: we prioritize preferLocalBuilds jobs on the localJobs slot pool.
+        // Once we cannot anymore, we eat towards the builds slot pool.
+        if (parsedDrv->willBuildLocally(worker.store)) {
+            if (auto token = worker.localJobs.tryAcquire()) {
+                slotToken = std::move(*token);
+                co_return co_await tryToBuild();
+            }
+
+            if (auto token = worker.builds.tryAcquire()) {
+                slotToken = std::move(*token);
+                co_return co_await tryToBuild();
+            }
+
+            if (worker.localJobs.capacity() > 0) {
+                slotToken = co_await worker.localJobs.acquire();
+                co_return co_await tryToBuild();
+            }
+        }
+
+        if (worker.builds.capacity() > 0) {
+            slotToken = co_await worker.builds.acquire();
             co_return co_await tryToBuild();
         }
+
         if (getMachines().empty()) {
             throw Error(
                 "unable to start any build; either set '--max-jobs' to a non-zero value or enable "

lix/libstore/build/worker.cc

@@ -35,7 +35,10 @@ Worker::Worker(Store & store, Store & evalStore, AvailableNamespaces namespaces)
     /* Make sure that we are always allowed to run at least one substitution.
        This prevents infinite waiting. */
     , substitutions(std::max<unsigned>(1, settings.maxSubstitutionJobs))
-    , localBuilds(settings.maxBuildJobs)
+    , builds(settings.maxBuildJobs)
+    // We allow local job to progress if there's no build job slot.
+    // See https://git.lix.systems/lix-project/lix/issues/855 for details.
+    , localJobs(settings.extraLocalJobs.get().value_or(settings.maxBuildJobs == 0U ? 1U : 0U))
     , children(errorHandler)
     , namespaces(namespaces)
 {

lix/libstore/build/worker.hh

@@ -188,7 +188,7 @@ public:
 
     Store & store;
     Store & evalStore;
-    AsyncSemaphore substitutions, localBuilds;
+    AsyncSemaphore substitutions, builds, localJobs;
     std::optional<Path> buildDirOverride;
 
     struct PlatformFeatures

lix/libstore/meson.build

@@ -93,6 +93,7 @@ libstore_setting_definitions = files(
   'settings/diff-hook.md',
   'settings/download-speed.md',
   'settings/enable-core-dumps.md',
+  'settings/extra-local-jobs.md',
   'settings/extra-platforms.md',
   'settings/fallback.md',
   'settings/fsync-metadata.md',

lix/libstore/parsed-derivations.cc

@@ -106,10 +106,6 @@ bool ParsedDerivation::canBuildLocally(Store & localStore) const
         && !drv.isBuiltin())
         return false;
 
-    if (settings.maxBuildJobs.get() == 0
-        && !drv.isBuiltin())
-        return false;
-
     for (auto & feature : getRequiredSystemFeatures())
         if (!localStore.config().systemFeatures.get().count(feature)) return false;
 

lix/libstore/settings/extra-local-jobs.md

@@ -0,0 +1,16 @@
+---
+name: extra-local-jobs
+internalName: extraLocalJobs
+type: std::optional<uint32_t>
+defaultText: "1 if `max-jobs = 0` otherwise 0"
+---
+
+This option defines an additional amount of slots for local jobs.
+`preferLocalBuild` derivations will prioritize acquiring from the local job pool first,
+then fall back to a `max-jobs` pool.
+
+By default, `extra-local-jobs` is empty except if `max-jobs = 0`, in which
+case, Lix will automatically set this value to `1` to allow local jobs to
+progress as they are usually trivial.
+
+This option can be overridden using the `--extra-local-jobs` flag.

tests/functional2/build/test_build_jobless.py

@@ -1,5 +1,7 @@
 import pytest
 from testlib.fixtures.nix import Nix
+from testlib.utils import get_global_asset
+from testlib.fixtures.file_helper import with_files, File
 
 
 drv = """
@@ -11,6 +13,17 @@ drv = """
   }
 """
 
+local_drv = """
+  with import ./config.nix; mkDerivation {
+    name = "foo";
+    buildCommand = ''
+      touch $out
+    '';
+    preferLocalBuild = true;
+    requiredSystemFeatures = [ "glitter" ];
+  }
+"""
+
 
 @pytest.fixture(autouse=True)
 def set_features(nix: Nix):
@@ -32,3 +45,20 @@ def test_j0_with_mismatched_remotes_fails(nix: Nix):
         "error: unable to start any build; remote machines may not have all required system features."
         in result.stderr_s
     )
+
+
+@with_files({"config.nix": get_global_asset("config.nix"), "default.nix": File(local_drv)})
+def test_j0_without_local_jobs(nix: Nix):
+    result = nix.nix_build(["-j0", "--extra-local-jobs", "0"]).run().expect(1)
+    assert (
+        "error: unable to start any build; either set '--max-jobs' to a non-zero value or enable remote builds."
+        in result.stderr_s
+    )
+
+
+@with_files({"config.nix": get_global_asset("config.nix"), "default.nix": File(local_drv)})
+def test_j0_with_local_jobs(nix: Nix):
+    nix.nix_build(["-j0"]).run().ok()
+
+
+# TODO(Raito): test -j1 --extra-local-jobs=1 in another file and ensure that preferLocalBuild *AND* another build job can run properly at the same time.