From a527bb251a9b71ac060d10422bbf989135140a90 Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Fri, 30 May 2025 15:55:14 +0200 Subject: [PATCH] libstore/build: cgroup delegation to sandbox We offer full cgroup delegation to our sandbox now, required for running containers inside the sandbox. To run systemd-nspawn or containers managers inside the sandbox, there is a need for one extra ingredient now: control over your own cgroup subtree inside the sandbox. If, in addition, you need multiple UIDs, for e.g. rootless usecases, you need to run with the `uid-range` system feature. Therefore, when the daemon or Nix runs under the right condition, e.g. systemd-style delegation of the cgroup subtree while placing the nix-daemon in a supervisor sub-cgroup, we create a new sub-cgroup for each build based on the build UID and delegate that sub-cgroup to the builder's process. Additionally, `uid-range` always request the `cgroups` feature now, as `uid-range` builds would probably always benefit from having cgroups delegated, but the converse is not true. Inspired from https://github.com/NixOS/nix/pull/11412 with a different design that does not use function-local statics to derive the root cgroup. Co-authored-by: Linus Heckemann Co-authored-by: Parker Hoyes Change-Id: Ic8947c5adaf4b5bbd153386e05fad65a935274fa Signed-off-by: Raito Bezarius --- doc/manual/rl-next/cgroups-delegation.md | 69 +++ .../rl-next/uid-range-requires-cgroups.md | 15 + lix/libstore/build/local-derivation-goal.cc | 2 + lix/libstore/build/local-derivation-goal.hh | 16 + lix/libstore/build/worker.cc | 79 ++++ lix/libstore/build/worker.hh | 10 + lix/libstore/meson.build | 1 + lix/libstore/platform/linux.cc | 61 ++- lix/libstore/settings/use-cgroups.md | 17 + lix/libutil/cgroup.cc | 431 ++++++++++++++---- lix/libutil/cgroup.hh | 189 +++++++- lix/libutil/current-process.cc | 7 +- lix/libutil/experimental-features/cgroups.md | 6 + lix/libutil/meson.build | 1 + misc/systemd/nix-daemon.service.in | 2 + tests/nixos/cgroups/default.nix | 40 ++ tests/nixos/cgroups/hang.nix | 10 + tests/nixos/containers/containers.nix | 44 +- tests/nixos/containers/id-test.nix | 8 + tests/nixos/containers/systemd-nspawn.nix | 80 ++++ tests/nixos/default.nix | 4 + 21 files changed, 965 insertions(+), 127 deletions(-) create mode 100644 doc/manual/rl-next/cgroups-delegation.md create mode 100644 doc/manual/rl-next/uid-range-requires-cgroups.md create mode 100644 lix/libstore/settings/use-cgroups.md create mode 100644 lix/libutil/experimental-features/cgroups.md create mode 100644 tests/nixos/cgroups/default.nix create mode 100644 tests/nixos/cgroups/hang.nix create mode 100644 tests/nixos/containers/id-test.nix create mode 100644 tests/nixos/containers/systemd-nspawn.nix diff --git a/doc/manual/rl-next/cgroups-delegation.md b/doc/manual/rl-next/cgroups-delegation.md new file mode 100644 index 000000000..a6d6418df --- /dev/null +++ b/doc/manual/rl-next/cgroups-delegation.md @@ -0,0 +1,69 @@ +--- +synopsis: New cgroup delegation model +issues: [fj#537, fj#77] +cls: [3230] +category: "Breaking Changes" +credits: [raito, horrors, lheckemann] +--- + +Builds using cgroups (i.e. `use-cgroups = true` and the experimental feature +`cgroups`) now always delegate a cgroup tree to the sandbox. + +Compared to the original C++ Nix project, our delegation includes the +`subtree_control` file as well, which means that the sandbox can disable +certain controllers in its own cgroup tree. + +This is a breaking change because this requires the Nix daemon to run with an +already delegated cgroup tree by the service manager. + +## How to setup the cgroup tree with systemd? + +systemd offers knobs to perform the required setup using: + +``` +[Unit] +Delegate=yes +DelegateSubtree=supervisor +``` + +These directives are now included in our systemd packaging. + +## What about using Nix as root without connecting to the daemon? + +Builds run as `root` without connecting to the daemon relying on the cgroup +feature are now broken, i.e. + +```console +# nix-build --use-cgroups --sandbox ... # will not work +``` + +Consider doing instead: + +```console +# systemd-run --same-dir --wait -p Delegate=yes -p DelegateSubgroup=supervisor nix-build --use-cgroups ... +``` + +If you need to disable cgroups temporarily, remember that you can do +`NIX_CONF='include /etc/nix/nix.conf\nuse-cgroups = false' nix-build ...` or +`nix-build --no-use-cgroups ...`. + +## What about other service managers than systemd? + +systemd has a [documentation](https://systemd.io/CGROUP_DELEGATION/) on how to +handle cgroup delegation from service management perspective. + +If your service manager adheres to systemd semantics, e.g. writing an extended +attribute `user.delegate=1` on the delegated cgroup tree directory and moving +the `nix-daemon` process inside a cgroup tree to respect the inner process +rule, then, the feature will work as well. + +## Why is the cgroup feature still experimental? + +While the cgroup feature unlocks many use cases, its behavior and integration (e.g. user experience), especially at scale on build farms or in multi-tenant environments, are not yet fully matured. There’s also potential for deeper systemd integration (e.g. using slices and scopes) that has not been fully explored. + +To avoid locking in an unstable interface, we’re keeping the experimental flag until we have validated the feature across a broader range of scenarios, including but not limited to: + +* Nix as root +* Hydra-style build farms +* Forgejo CI runners +* Shared remote builders diff --git a/doc/manual/rl-next/uid-range-requires-cgroups.md b/doc/manual/rl-next/uid-range-requires-cgroups.md new file mode 100644 index 000000000..4a883e5a4 --- /dev/null +++ b/doc/manual/rl-next/uid-range-requires-cgroups.md @@ -0,0 +1,15 @@ +--- +synopsis: uid-range depends on cgroups +issues: [] +cls: [3230] +category: "Breaking Changes" +credits: [raito, horrors] +--- + +`uid-range` builds now depends on `cgroups`, an experimental feature. + +`uid-range` builds already depended upon `auto-allocate-uids`, another experimental feature. + +The rationale for doing so is that `uid-range` provides a sandbox with many +UIDs, this is useful for re-mapping them into a nested namespace, e.g. a +container. diff --git a/lix/libstore/build/local-derivation-goal.cc b/lix/libstore/build/local-derivation-goal.cc index c750ca880..d0b975512 100644 --- a/lix/libstore/build/local-derivation-goal.cc +++ b/lix/libstore/build/local-derivation-goal.cc @@ -402,9 +402,11 @@ void LocalDerivationGoal::cleanupPostOutputsRegisteredModeNonCheck() kj::Promise> LocalDerivationGoal::startBuilder() try { +#if !(__linux__) if (buildUser && buildUser->getUIDCount() != 1) { throw Error("cgroups are not supported on this platform"); } +#endif /* Make sure that no other processes are executing under the sandbox uids. This must be done before any chownToBuilder() diff --git a/lix/libstore/build/local-derivation-goal.hh b/lix/libstore/build/local-derivation-goal.hh index 84aece58c..ec874ecea 100644 --- a/lix/libstore/build/local-derivation-goal.hh +++ b/lix/libstore/build/local-derivation-goal.hh @@ -4,9 +4,20 @@ #include "lix/libstore/build/derivation-goal.hh" #include "lix/libstore/local-store.hh" #include "lix/libutil/processes.hh" +#include "lix/libutil/cgroup.hh" namespace nix { +struct BuildContext +{ +#ifdef __linux__ + /** + * Control group for this derivation goal. + */ + std::optional cgroup; +#endif +}; + struct LocalDerivationGoal : public DerivationGoal { LocalStore & getLocalStore(); @@ -16,6 +27,11 @@ struct LocalDerivationGoal : public DerivationGoal */ std::unique_ptr buildUser; + /** + * Build context for this goal. + */ + BuildContext context; + /** * The process ID of the builder. */ diff --git a/lix/libstore/build/worker.cc b/lix/libstore/build/worker.cc index 154fa461c..abe68a103 100644 --- a/lix/libstore/build/worker.cc +++ b/lix/libstore/build/worker.cc @@ -37,6 +37,85 @@ Worker::Worker(Store & store, Store & evalStore) , children(errorHandler) { /* Debugging: prevent recursive workers. */ + +#ifdef __linux__ + + /* When cgroups are used, we need to verify if our context allows + * us to fully use cgroups delegation or ability to kill certain cgroups. + * + * Note that `uid-range` builds implies cgroups, the converse is false. + * A `uid-range` build is defined as `settings.autoAllocateUids && settings.uidCount >= 1` + */ + + if (settings.autoAllocateUids && settings.uidCount > 1 && !settings.useCgroups) { + throw Error( + "Running builds with UID ranges (setting `%s` enabled and `%d` UIDs) requires the " + "setting '%s' to be enabled.", + settings.autoAllocateUids.name, + settings.uidCount, + settings.useCgroups.name + ); + } + + /* Cgroup build absolutely need build user separation. */ + if (!useBuildUsers() && settings.useCgroups) { + throw Error( + "Running all builds with cgroups requires privilege separation for build users but Lix " + "is not configured to use build users." + ); + } + + /* + * At this point, we know that if `settings.useCgroups = true`, then `useBuildUsers() = true`. + * UID ranges may or may not be available. + */ + if (settings.useCgroups) { + if (!hasCgroupFeature( + platformFeatures.availableCgroupFeatures, CgroupAvailableFeatureSet::CGROUPV2 + )) + { + throw Error("Running a build with cgroups requires cgroups v2 support on the system."); + } + + if (!hasCgroupFeature( + platformFeatures.availableCgroupFeatures, CgroupAvailableFeatureSet::CGROUPV2_KILL + )) + { + throw Error( + "Running a build with cgroups requires cgroups v2 kill feature which requires " + "a Linux kernel newer than 5.14." + ); + } + + if (!hasCgroupFeature( + platformFeatures.availableCgroupFeatures, + CgroupAvailableFeatureSet::CGROUPV2_PARENT_DELEGATED + )) + { + if (hasCgroupFeature( + platformFeatures.availableCgroupFeatures, + CgroupAvailableFeatureSet::CGROUPV2_SELF_DELEGATED + )) + { + throw Error( + "Running a build with cgroups requires the parent cgroup tree to be " + "delegated, but only this process' cgroup is delegated.\n" + "If you used systemd with `Delegate=yes`, consider moving the process in a " + "sub-cgroup or use `DelegateSubtree=` to move it automatically.\n" + "See for more information." + ); + } else { + throw Error( + "Running a build with cgroups requires the parent cgroup tree to be " + "delgated.\n" + "If you use systemd, adding `Delegate=yes` and `DelegateSubtree=supervisor` to " + "the [Unit] section will delegate the parent cgroup tree.\n" + "See for more information." + ); + } + } + } +#endif } diff --git a/lix/libstore/build/worker.hh b/lix/libstore/build/worker.hh index ea6fd0f40..b7dcdc881 100644 --- a/lix/libstore/build/worker.hh +++ b/lix/libstore/build/worker.hh @@ -6,6 +6,7 @@ #include "lix/libutil/concepts.hh" #include "lix/libutil/notifying-counter.hh" #include "lix/libutil/result.hh" +#include "lix/libutil/cgroup.hh" #include "lix/libutil/types.hh" #include "lix/libstore/lock.hh" #include "lix/libstore/store-api.hh" @@ -187,6 +188,15 @@ public: Store & evalStore; AsyncSemaphore substitutions, localBuilds; + struct PlatformFeatures + { +#ifdef __linux__ + CgroupAvailableFeatureSet availableCgroupFeatures = detectAvailableCgroupFeatures(); +#endif + }; + + PlatformFeatures platformFeatures; + private: kj::TaskSet children; diff --git a/lix/libstore/meson.build b/lix/libstore/meson.build index e472168b0..e59ae01b5 100644 --- a/lix/libstore/meson.build +++ b/lix/libstore/meson.build @@ -108,6 +108,7 @@ libstore_setting_definitions = files( 'settings/timeout.md', 'settings/trusted-public-keys.md', 'settings/trusted-substituters.md', + 'settings/use-cgroups.md', 'settings/use-sqlite-wal.md', 'settings/use-xdg-base-directories.md', # keep-sorted end diff --git a/lix/libstore/platform/linux.cc b/lix/libstore/platform/linux.cc index 98b08f89a..229136a93 100644 --- a/lix/libstore/platform/linux.cc +++ b/lix/libstore/platform/linux.cc @@ -1,4 +1,5 @@ #include "lix/libstore/build/worker.hh" +#include "lix/libutil/cgroup.hh" #include "lix/libutil/finally.hh" #include "lix/libstore/gc-store.hh" #include "lix/libutil/signals.hh" @@ -780,12 +781,14 @@ void LinuxLocalDerivationGoal::prepareSandbox() nobody account. The latter is kind of a hack to support Samba-in-QEMU. */ createDirs(chrootRootDir + "/etc"); - if (parsedDrv->useUidRange()) - chownToBuilder(chrootRootDir + "/etc"); if (parsedDrv->useUidRange() && (!buildUser || buildUser->getUIDCount() < 65536)) throw Error("feature 'uid-range' requires the setting '%s' to be enabled", settings.autoAllocateUids.name); + if (parsedDrv->useUidRange()) { + chownToBuilder(chrootRootDir + "/etc"); + } + /* Create /etc/hosts with localhost entry. */ if (derivationType->isSandboxed()) writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n::1 localhost\n"); @@ -818,6 +821,42 @@ void LinuxLocalDerivationGoal::prepareSandbox() for (auto & i : drv->outputsAndPaths(worker.store)) { pathsInChroot.erase(worker.store.printStorePath(i.second.second)); } + + if (buildUser && (buildUser->getUIDCount() != 1 || settings.useCgroups)) { + context.cgroup.emplace( + settings.nixStateDir + "/cgroups", + fmt("nix-build-uid-%d", buildUser->getUID()), + buildUser->getUID(), + buildUser->getGID() + ); + + debug("using cgroup '%s' for build", context.cgroup->name()); + + /* TODO(raito): it would be very nice if we could propagate system features + * based on which cgroup controllers are available in `context.cgroup` + * so that we would re-schedule any derivation that actually has + * anti-affinity or pro-affinity with certain cgroup controllers, e.g. + * a derivation that is very sensitive to the memory cgroup controller + * for performance reason. + * + * Unfortunately, the current design of system features prevent mutation + * and worse, we are too late for rescheduling this derivation. + * + * Therefore, we decide to always copy all the available controllers + * to the delegated cgroup. + */ + debug( + "available cgroup controllers for cgroup '%s': '%s'", + context.cgroup->name(), + concatStringsSep(",", context.cgroup->controllers()) + ); + } + + if (parsedDrv->useUidRange() && !context.cgroup) { + throw Error( + "feature 'uid-range' requires the setting '%s' to be enabled", settings.useCgroups.name + ); + } } Pid LinuxLocalDerivationGoal::startChild(std::function openSlave) @@ -957,6 +996,11 @@ Pid LinuxLocalDerivationGoal::startChild(std::function openSlave) "nixbld:!:%1%:\n" "nogroup:x:65534:\n", sandboxGid())); + /* Migrate the child inside the available control group. */ + if (context.cgroup) { + context.cgroup->adoptProcess(pid.get()); + } + /* Signal the builder that we've updated its user namespace. */ writeFull(userNamespaceSync.writeSide.get(), "1"); @@ -965,7 +1009,18 @@ Pid LinuxLocalDerivationGoal::startChild(std::function openSlave) void LinuxLocalDerivationGoal::killSandbox(bool getStats) { - if (!useChroot) { + if (context.cgroup) { + context.cgroup->kill(); + if (getStats) { + auto stats = context.cgroup->getStatistics(); + buildResult.cpuUser = stats.cpuUser; + buildResult.cpuSystem = stats.cpuSystem; + } + /* It may be desireable to destroy the cgroup here + * but we may be calling this at the start of the build + * to ensure that no leftover process are running under sandbox UIDs. + * With control groups, that's already impossible. */ + } else if (!useChroot) { /* Linux sandboxes use PID namespaces, which ensure that processes cannot escape from a build. Therefore, we don't need to kill all processes belonging to the build user. This avoids processes unrelated to the build being killed, thus avoiding: https://git.lix.systems/lix-project/lix/issues/667 */ diff --git a/lix/libstore/settings/use-cgroups.md b/lix/libstore/settings/use-cgroups.md new file mode 100644 index 000000000..6ed460ace --- /dev/null +++ b/lix/libstore/settings/use-cgroups.md @@ -0,0 +1,17 @@ +--- +name: use-cgroups +internalName: useCgroups +platforms: [linux] +type: bool +default: false +experimentalFeature: cgroups +--- +Whether to execute builds inside cgroups. + +Cgroups are required and enabled automatically for derivations +that require the `uid-range` system feature. + +Cgroups requires a cgroup delegation according to , i.e. +the Nix process (`nix-daemon` or any single user command) should run in a cgroup tree of a parent cgroup which possess the `user.delegate=1` extended attribute. + +In this scenario, Nix will run builds in a sibling cgroup named `nix-build-uid-`. diff --git a/lix/libutil/cgroup.cc b/lix/libutil/cgroup.cc index 2aa8e4d76..a4d2c7518 100644 --- a/lix/libutil/cgroup.cc +++ b/lix/libutil/cgroup.cc @@ -1,7 +1,8 @@ -#include "lix/libutil/logging.hh" -#include "regex.hh" +#include "logging.hh" #if __linux__ +#include "regex.hh" + #include "lix/libutil/cgroup.hh" #include "lix/libutil/file-system.hh" #include "lix/libutil/finally.hh" @@ -10,32 +11,29 @@ #include #include #include -#include -#include -#include #include #include +#include namespace nix { -std::optional getCgroupFS() +static bool isCgroupDelegated(const Path & path) { - static auto res = [&]() -> std::optional { - auto fp = fopen("/proc/mounts", "r"); - if (!fp) return std::nullopt; - Finally delFP = [&]() { fclose(fp); }; - while (auto ent = getmntent(fp)) - if (std::string_view(ent->mnt_type) == "cgroup2") - return ent->mnt_dir; + char delegate_xattr; - return std::nullopt; - }(); - return res; + if (getxattr(path.c_str(), "user.delegate", &delegate_xattr, sizeof(delegate_xattr)) >= 1) { + if (delegate_xattr != '1') { + throw Error("Unexpected `user.delegate` xattr: '%c'", delegate_xattr); + } + + return true; + } + + return false; } -// FIXME: obsolete, check for cgroup2 -std::map getCgroups(const Path & cgroupFile) +static std::map getCgroups(const Path & cgroupFile) { std::map cgroups; @@ -52,101 +50,342 @@ std::map getCgroups(const Path & cgroupFile) return cgroups; } -static CgroupStats destroyCgroup(const Path & cgroup, bool returnStats) +static CgroupStats readStatistics(const std::filesystem::path & cgroup) { - if (!pathExists(cgroup)) return {}; - - auto procsFile = cgroup + "/cgroup.procs"; - - if (!pathExists(procsFile)) - throw Error("'%s' is not a cgroup", cgroup); - - /* Use the fast way to kill every process in a cgroup, if - available. */ - auto killFile = cgroup + "/cgroup.kill"; - if (pathExists(killFile)) - writeFile(killFile, "1"); - - /* Otherwise, manually kill every process in the subcgroups and - this cgroup. */ - for (auto & entry : readDirectory(cgroup)) { - if (entry.type != DT_DIR) continue; - destroyCgroup(cgroup + "/" + entry.name, false); - } - - int round = 1; - - std::unordered_set pidsShown; - - while (true) { - auto pids = tokenizeString>(readFile(procsFile)); - - if (pids.empty()) break; - - if (round > 20) - throw Error("cannot kill cgroup '%s'", cgroup); - - for (auto & pid_s : pids) { - pid_t pid; - if (auto o = string2Int(pid_s)) - pid = *o; - else - throw Error("invalid pid '%s'", pid); - if (pidsShown.insert(pid).second) { - try { - auto cmdline = readFile(fmt("/proc/%d/cmdline", pid)); - using namespace std::string_literals; - warn("killing stray builder process %d (%s)...", - pid, trim(replaceStrings(cmdline, "\0"s, " "))); - } catch (SysError &) { - } - } - // FIXME: pid wraparound - if (kill(pid, SIGKILL) == -1 && errno != ESRCH) - throw SysError("killing member %d of cgroup '%s'", pid, cgroup); - } - - auto sleep = std::chrono::milliseconds((int) std::pow(2.0, std::min(round, 10))); - if (sleep.count() > 100) - printError("waiting for %d ms for cgroup '%s' to become empty", sleep.count(), cgroup); - std::this_thread::sleep_for(sleep); - round++; - } - CgroupStats stats; - if (returnStats) { - auto cpustatPath = cgroup + "/cpu.stat"; + auto cpustatPath = cgroup / "cpu.stat"; - if (pathExists(cpustatPath)) { - for (auto & line : tokenizeString>(readFile(cpustatPath), "\n")) { - std::string_view userPrefix = "user_usec "; - if (line.starts_with(userPrefix)) { - auto n = string2Int(line.substr(userPrefix.size())); - if (n) stats.cpuUser = std::chrono::microseconds(*n); + if (pathExists(cpustatPath)) { + for (auto & line : tokenizeString>(readFile(cpustatPath), "\n")) { + std::string_view userPrefix = "user_usec "; + if (line.starts_with(userPrefix)) { + auto n = string2Int(line.substr(userPrefix.size())); + if (n) { + stats.cpuUser = std::chrono::microseconds(*n); } + } - std::string_view systemPrefix = "system_usec "; - if (line.starts_with(systemPrefix)) { - auto n = string2Int(line.substr(systemPrefix.size())); - if (n) stats.cpuSystem = std::chrono::microseconds(*n); + std::string_view systemPrefix = "system_usec "; + if (line.starts_with(systemPrefix)) { + auto n = string2Int(line.substr(systemPrefix.size())); + if (n) { + stats.cpuSystem = std::chrono::microseconds(*n); } } } - } - if (rmdir(cgroup.c_str()) == -1) - throw SysError("deleting cgroup '%s'", cgroup); - return stats; } -CgroupStats destroyCgroup(const Path & cgroup) +static void killCgroup(const std::string & name, const std::filesystem::path & cgroup) { - return destroyCgroup(cgroup, true); + auto killFile = cgroup / "cgroup.kill"; + if (pathExists(killFile)) + writeFile(killFile, "1"); + else { + throw SysError( + "cgroup '%s' at '%s' does not possess `cgroup.kill` ; are you running Lix on a kernel " + "older than 5.14 with cgroups?", + name, + cgroup + ); + } } +static std::optional +destroyCgroup(const std::string & name, const std::filesystem::path & aliveCgroup) +{ + debug("destroying cgroup '%s' at '%s'", name, aliveCgroup); + if (!pathExists(aliveCgroup)) { + debug("destroying cgroup '%s' already destroyed", name); + return {}; + } + + auto procsFile = aliveCgroup / "cgroup.procs"; + + if (!pathExists(procsFile)) { + throw SysError( + "cgroup '%s' at '%s' has an invalid cgroup hierarchy (missing `cgroup.procs`)", + name, + aliveCgroup + ); + } + + killCgroup(name, aliveCgroup); + + CgroupStats stats = readStatistics(aliveCgroup); + + if (rmdir(aliveCgroup.c_str()) == -1) { + throw SysError("deleting cgroup '%s' at '%s'", name, aliveCgroup); + } + + debug("cgroup '%s' destroyed", name); + + return stats; +} + +CgroupHierarchy getLocalHierarchy(const std::filesystem::path & cgroupFilesystem) +{ + CgroupHierarchy hierarchy; + + auto ourCgroups = getCgroups("/proc/self/cgroup"); + auto ourCgroup = ourCgroups[""]; + + if (ourCgroup == "") { + throw Error("cannot determine cgroup name from '/proc/self/cgroup'"); + } + + if (ourCgroup[0] == '/') { + ourCgroup.erase(0, 1); + } + + auto ourCgroupPath = (cgroupFilesystem / ourCgroup).lexically_normal(); + + if (!pathExists(ourCgroupPath)) { + throw Error("expected cgroup directory '%s'", ourCgroupPath); + } + + hierarchy.ourCgroupPath = ourCgroupPath; + + return hierarchy; +} + +CgroupAvailableFeatureSet operator|(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs) +{ + return static_cast( + static_cast::type>(lhs) + | static_cast::type>(rhs) + ); +} + +CgroupAvailableFeatureSet & +operator|=(CgroupAvailableFeatureSet & lhs, CgroupAvailableFeatureSet rhs) +{ + return lhs = lhs | rhs; +} + +CgroupAvailableFeatureSet operator&(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs) +{ + return static_cast( + static_cast::type>(lhs) + & static_cast::type>(rhs) + ); +} + +bool hasCgroupFeature(CgroupAvailableFeatureSet featureSet, CgroupAvailableFeatureSet testedFeature) +{ + return static_cast::type>( + featureSet & testedFeature + ) + != 0; +} + +CgroupAvailableFeatureSet detectAvailableCgroupFeatures() +{ + CgroupAvailableFeatureSet features = {}; + + auto fs = getCgroupFS(); + + if (fs && !fs->empty()) { + features |= CgroupAvailableFeatureSet::CGROUPV2; + + auto localHierarchy = getLocalHierarchy(*fs); + if (pathExists(localHierarchy.ourCgroupPath / "cgroup.kill")) { + features |= CgroupAvailableFeatureSet::CGROUPV2_KILL; + } + + if (isCgroupDelegated(localHierarchy.ourCgroupPath)) { + features |= CgroupAvailableFeatureSet::CGROUPV2_SELF_DELEGATED; + } + + auto parentCgroupPath = localHierarchy.parentCgroupPath(); + if (parentCgroupPath && isCgroupDelegated(*parentCgroupPath)) { + features |= CgroupAvailableFeatureSet::CGROUPV2_PARENT_DELEGATED; + } + } + + return features; +} + +static std::vector readControllers(const std::filesystem::path & cgroupPath) +{ + return tokenizeString>( + readFile(cgroupPath / "cgroup.controllers"), " " + ); +} + +AutoDestroyCgroup::AutoDestroyCgroup( + const std::filesystem::path & cgroupRecordsDir, std::string const & name +) + : name_(name) +{ + auto cgroupFilesystem = getCgroupFS(); + if (!cgroupFilesystem || cgroupFilesystem->empty()) { + throw Error("cannot determine the path to the cgroupv2 filesystem"); + } + + auto hierarchy = getLocalHierarchy(*cgroupFilesystem); + auto parentCgroupPath = hierarchy.parentCgroupPath(); + assert(parentCgroupPath && "AutoDestroyCgroup cannot be used on the root cgroup"); + /* We assert that the parent cgroup is delegated at this point. + * This is a responsibility of the caller. */ + assert(isCgroupDelegated(*parentCgroupPath) && "parent cgroup was supposed to be delegated"); + + /* All available controllers on the parent cgroup path will be delegated. + * TODO(raito): implementing a filtering mechanism is for the future. */ + controllers_ = readControllers(*parentCgroupPath); + + /* Enable all the controllers */ + writeFile( + *parentCgroupPath / "cgroup.subtree_control", + concatMapStringsSep( + " ", + controllers_, + [](const std::string & controller) -> std::string { return fmt("+%s", controller); } + ) + ); + + cgroup_ = *parentCgroupPath / name; + + /* + * In case we get interrupted without cleaning up the cgroup we just created, + * we look at Nix's state directory where we record all cgroups being used + * and destroy it before reusing it. */ + cleansePreviousInstancesAndRecordOurself(cgroupRecordsDir); +} + +AutoDestroyCgroup::AutoDestroyCgroup( + const std::filesystem::path & cgroupRecordsDir, std::string const & name, uid_t uid, gid_t gid +) + : AutoDestroyCgroup(cgroupRecordsDir, name) +{ + auto path = std::get(cgroup_); + + if (mkdir(path.c_str(), 0755) == -1) { + throw SysError( + "cannot create the top-level directory at '%s' for cgroup '%s'", path, name_ + ); + } + + if (chown(path.c_str(), uid, gid) == -1) { + throw SysError( + "cannot delegate the top-level directory '%s' from cgroup '%s' to user uid=%d,gid=%d", + path, + name_, + uid, + gid + ); + } + + AutoCloseFD cgroupFd{open(path.c_str(), O_PATH | O_NOFOLLOW)}; + for (auto node : {"procs", "threads", "subtree_control"}) { + if (fchownat(cgroupFd.get(), fmt("cgroup.%s", node).c_str(), uid, gid, 0) == -1) { + throw SysError( + "cannot delegate '%s' from cgroup '%s' to user uid=%d,gid=%d", node, name_, uid, gid + ); + } + } + + delegation_ = {.uid = uid, .gid = gid}; +} + +AutoDestroyCgroup::~AutoDestroyCgroup() +{ + try { + std::visit( + overloaded{ + [&, this](const Path & aliveCgroup) { + auto maybeStats = destroyCgroup(name_, aliveCgroup); + if (!maybeStats) { + warn( + "cgroup '%s' was destroyed unexpectedly (something else removed the " + "cgroup).", + aliveCgroup + ); + } + }, + [&](const CgroupStats & stats) {} + }, + cgroup_ + ); + } catch (...) { + ignoreExceptionInDestructor(); + } +} + +void AutoDestroyCgroup::cleansePreviousInstancesAndRecordOurself( + const std::filesystem::path & cgroupRecordsDir +) +{ + createDirs(cgroupRecordsDir); + + auto cgroupFile = cgroupRecordsDir / name_; + + if (pathExists(cgroupFile)) { + auto prevCgroup = readFile(cgroupFile); + warn("destroying past cgroup '%s' found in the state directory", name_); + destroyCgroup(fmt("past %s", name_), prevCgroup); + } + + writeFile(cgroupFile, std::get(cgroup_).string()); + stateRecord = AutoDelete(cgroupFile, false); +} + +void AutoDestroyCgroup::adoptProcess(int pid) +{ + auto path = std::get_if(&cgroup_); + if (!path) { + throw SysError("cgroup '%s' went away while adopting process '%d'", name_, pid); + } + + writeFile(*path / "cgroup.procs", fmt("%d", pid)); +} + +void AutoDestroyCgroup::kill() +{ + auto path = std::get_if(&cgroup_); + if (!path) { + throw SysError("killing cgroup '%s' but it went away", name_); + } + + killCgroup(name_, *path); +} + +CgroupStats AutoDestroyCgroup::getStatistics() const +{ + /* Either: + * - we need to pull statistics from an alive cgroup. + * - we need to get historical statistics from a dead cgroup. + */ + return std::visit( + nix::overloaded{ + [&](const Path & aliveCgroup) { return readStatistics(aliveCgroup); }, + [&](const CgroupStats & stats) { return stats; } + }, + cgroup_ + ); +} + +std::optional getCgroupFS() +{ + static auto res = [&]() -> std::optional { + auto fp = fopen("/proc/mounts", "r"); + if (!fp) { + return {}; + } + Finally delFP = [&]() { fclose(fp); }; + while (auto ent = getmntent(fp)) { + if (std::string_view(ent->mnt_type) == "cgroup2") { + return {ent->mnt_dir}; + } + } + + return {}; + }(); + return res; +} } #endif diff --git a/lix/libutil/cgroup.hh b/lix/libutil/cgroup.hh index ed9d40e40..89d6bc916 100644 --- a/lix/libutil/cgroup.hh +++ b/lix/libutil/cgroup.hh @@ -1,18 +1,99 @@ #pragma once ///@file +#include "file-system.hh" #if __linux__ #include #include +#include +#include #include "lix/libutil/types.hh" namespace nix { -std::optional getCgroupFS(); +/* This represent a cgroup hierarchy. + * When doing cgroup delegation, it is necessary to ensure that all processes + * lives in the leaves of the cgroup tree. + * + * When Nix runs a build, the cgroup tree should be arranged so that: + * + * Nix parent process cgroup (e.g. `system.slice` on systemd) + * | + * | + * nix-daemon.service cgroup + * / \ + * / \ + * / \ + * / \ + * / \ + * / \ + * nix-build-uid-418238 cgroup supervisor cgroup + * | | + * derivation build processes | + * nix-daemon subdaemons + * + * Notice that if you put the nix-daemon subdaemons directly inside the `nix-daemon.service` + * cgroup, then there would be processes in _inner cgroups_, which is a violation of cgroup best + * practices (it opens up the path for children to compete for cgroup control). + * + * If you run systemd with `Delegate` and `DelegateSubgroup` options, the hierarchy is setup + * like the above automatically. + * + * If you are not in these conditions, you will need to write code to move the various daemon-like + * processes inside a sibling cgroup. + * + */ +struct CgroupHierarchy +{ + std::filesystem::path ourCgroupPath; + std::optional parentCgroupPath() const + { + if (ourCgroupPath.has_parent_path()) { + return ourCgroupPath.parent_path(); + } else { + return {}; + } + } +}; -std::map getCgroups(const Path & cgroupFile); +/* Return the current process's view of the cgroup hierarchy, i.e. + * the parent process's cgroup and this current process's cgroup. + * */ +CgroupHierarchy getLocalHierarchy(std::filesystem::path const & cgroupFilesystem); + +/* Return a path to the cgroupv2 filesystem path, if it exist */ +std::optional getCgroupFS(); + +/* Help detect the list of feature sets available + * for the running kernel. */ +enum class CgroupAvailableFeatureSet : uint8_t { + /* cgroupvs2 were detected. */ + CGROUPV2 = (1 << 0), + /* cgroupsv2 kill capability detected. The absence of this capability is unsupported. + * It appeared in kernel 5.14. */ + CGROUPV2_KILL = (1 << 1), + /* Current process is delegated à la systemd-style, e.g. + * `user.delegate=1` was written as an xattr of the cgroup directory. */ + CGROUPV2_SELF_DELEGATED = (1 << 2), + /* Current process' parent cgroup was delegated à la systemd-style. */ + CGROUPV2_PARENT_DELEGATED = (1 << 3), +}; + +CgroupAvailableFeatureSet operator|(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs); +CgroupAvailableFeatureSet & +operator|=(CgroupAvailableFeatureSet & lhs, CgroupAvailableFeatureSet rhs); +CgroupAvailableFeatureSet operator&(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs); + +/* Whether the `featureSet` given do possess the `testedFeature` in the set? */ +bool hasCgroupFeature( + CgroupAvailableFeatureSet featureSet, CgroupAvailableFeatureSet testedFeature +); + +/* Lix cgroup support relies on certain modern features to be available to avoid implementing many + * legacy code paths. */ +CgroupAvailableFeatureSet detectAvailableCgroupFeatures(); struct CgroupStats { @@ -20,13 +101,107 @@ struct CgroupStats }; /** - * Destroy the cgroup denoted by 'path'. The postcondition is that - * 'path' does not exist, and thus any processes in the cgroup have - * been killed. Also return statistics from the cgroup just before - * destruction. + * RAII class to hold an owned cgroup + * which will kill all processes under its hierarchy at destruction time. */ -CgroupStats destroyCgroup(const Path & cgroup); +class AutoDestroyCgroup +{ +private: + struct Delegation + { + /* + * Delegatee UID + */ + uid_t uid; + /* + * Delegatee GID + */ + gid_t gid; + }; + /* Friendly name of this cgroup */ + std::string name_; + + /* Enabled controllers of this cgroup */ + std::vector controllers_; + + /* A cgroup can be delegated to a pair of UID/GID */ + std::optional delegation_; + + /* + * Either the cgroup exist and is alive, + * either the cgroup was destroyed and we collected its statistics. + */ + std::variant cgroup_; + + /* + * Path to the state record of this cgroup's existence. + * This is used when the deletion process is interrupted + * for the next run. + */ + AutoDelete stateRecord; + + /* Kill all processes under its hierarchy and tear down the cgroup */ + void destroy(); + + /* + * Cleanse all previous instances of this cgroup where the deletion process + * might have been interrupted and record ourself in stead. + * If this cgroup deletion process does not run, on the next run, this will be + * reaped when the same name will be reused. + */ + void cleansePreviousInstancesAndRecordOurself(const std::filesystem::path & cgroupRecordsDir); +public: + KJ_DISALLOW_COPY(AutoDestroyCgroup); + + explicit AutoDestroyCgroup( + const std::filesystem::path & cgroupRecordsDir, std::string const & name + ); + + /* Delegate this cgroup to the given UID and GID. */ + explicit AutoDestroyCgroup( + const std::filesystem::path & cgroupRecordsDir, + std::string const & name, + uid_t uid, + gid_t gid + ); + ~AutoDestroyCgroup(); + + std::optional path() const + { + return std::visit( + nix::overloaded{ + [&](const Path & path) -> std::optional { return {path}; }, + [&](const CgroupStats &) -> std::optional { return {}; } + }, + cgroup_ + ); + } + + const std::string & name() const + { + return name_; + } + + const std::vector & controllers() const + { + return controllers_; + } + + std::optional delegation() const + { + return delegation_; + } + + /* Adopt a process in this cgroup. */ + void adoptProcess(int pid); + + /* Kill all processes under the control group. */ + void kill(); + + /* Return all statistics inside of this cgroup. */ + CgroupStats getStatistics() const; +}; } #endif diff --git a/lix/libutil/current-process.cc b/lix/libutil/current-process.cc index f8a850710..f3d77cfa0 100644 --- a/lix/libutil/current-process.cc +++ b/lix/libutil/current-process.cc @@ -32,11 +32,8 @@ unsigned int getMaxCPU() auto cgroupFS = getCgroupFS(); if (!cgroupFS) return 0; - auto cgroups = getCgroups("/proc/self/cgroup"); - auto cgroup = cgroups[""]; - if (cgroup == "") return 0; - - auto cpuFile = *cgroupFS + "/" + cgroup + "/cpu.max"; + auto localHierarchy = getLocalHierarchy(*cgroupFS); + auto cpuFile = localHierarchy.ourCgroupPath / "cpu.max"; auto cpuMax = readFile(cpuFile); auto cpuMaxParts = tokenizeString>(cpuMax, " \n"); diff --git a/lix/libutil/experimental-features/cgroups.md b/lix/libutil/experimental-features/cgroups.md new file mode 100644 index 000000000..2a00b7c8c --- /dev/null +++ b/lix/libutil/experimental-features/cgroups.md @@ -0,0 +1,6 @@ +--- +name: cgroups +internalName: Cgroups +--- +Allows Nix to execute builds inside cgroups. See +the [`use-cgroups`](../command-ref/conf-file.md#conf-use-cgroups) setting for details. diff --git a/lix/libutil/meson.build b/lix/libutil/meson.build index 7c40e5c45..ea8c3b051 100644 --- a/lix/libutil/meson.build +++ b/lix/libutil/meson.build @@ -142,6 +142,7 @@ libutil_headers = files( experimental_feature_definitions = files( # keep-sorted start 'experimental-features/auto-allocate-uids.md', + 'experimental-features/cgroups.md', 'experimental-features/coerce-integers.md', 'experimental-features/daemon-trust-override.md', 'experimental-features/fetch-closure.md', diff --git a/misc/systemd/nix-daemon.service.in b/misc/systemd/nix-daemon.service.in index 65356594c..88d60ea7d 100644 --- a/misc/systemd/nix-daemon.service.in +++ b/misc/systemd/nix-daemon.service.in @@ -11,6 +11,8 @@ ExecStart=@@bindir@/nix-daemon nix-daemon --daemon KillMode=process LimitNOFILE=1048576 TasksMax=1048576 +Delegate=yes +DelegateSubgroup=supervisor [Install] WantedBy=multi-user.target diff --git a/tests/nixos/cgroups/default.nix b/tests/nixos/cgroups/default.nix new file mode 100644 index 000000000..a8700ae4f --- /dev/null +++ b/tests/nixos/cgroups/default.nix @@ -0,0 +1,40 @@ +{ nixpkgs, ... }: + +{ + name = "cgroups"; + + nodes = + { + host = + { config, pkgs, ... }: + { virtualisation.additionalPaths = [ pkgs.stdenvNoCC ]; + nix.extraOptions = + '' + extra-experimental-features = nix-command auto-allocate-uids cgroups + extra-system-features = uid-range + ''; + nix.settings.use-cgroups = true; + nix.nixPath = [ "nixpkgs=${nixpkgs}" ]; + }; + }; + + testScript = { nodes }: '' + start_all() + + host.wait_for_unit("multi-user.target") + + # Start build in background + host.execute("NIX_REMOTE=daemon nix build --use-cgroups --auto-allocate-uids --file ${./hang.nix} >&2 &") + service = "/sys/fs/cgroup/system.slice/nix-daemon.service" + + # Wait for cgroups to be created + host.succeed(f"until [ -e {service}/supervisor ]; do sleep 1; done", timeout=30) + host.succeed(f"until [ -e {service}/nix-build-uid-* ]; do sleep 1; done", timeout=30) + + # Check that there aren't processes where there shouldn't be, and that there are where there should be + host.succeed(f'[ -z "$(cat {service}/cgroup.procs)" ]') + host.succeed(f'[ -n "$(cat {service}/supervisor/cgroup.procs)" ]') + host.succeed(f'[ -n "$(cat {service}/nix-build-uid-*/cgroup.procs)" ]') + ''; + +} diff --git a/tests/nixos/cgroups/hang.nix b/tests/nixos/cgroups/hang.nix new file mode 100644 index 000000000..cefe2d031 --- /dev/null +++ b/tests/nixos/cgroups/hang.nix @@ -0,0 +1,10 @@ +{ }: + +with import {}; + +runCommand "hang" + { requiredSystemFeatures = "uid-range"; + } + '' + sleep infinity + '' diff --git a/tests/nixos/containers/containers.nix b/tests/nixos/containers/containers.nix index 462ad9be8..b66974e3a 100644 --- a/tests/nixos/containers/containers.nix +++ b/tests/nixos/containers/containers.nix @@ -15,10 +15,13 @@ (import ./systemd-nspawn.nix { inherit nixpkgs; }).toplevel ]; virtualisation.memorySize = 4096; - nix.settings.substituters = lib.mkForce [ ]; + nix.settings = { + substituters = lib.mkForce [ ]; + use-cgroups = true; + }; nix.extraOptions = '' - extra-experimental-features = nix-command auto-allocate-uids + extra-experimental-features = nix-command auto-allocate-uids cgroups extra-system-features = uid-range ''; nix.nixPath = [ "nixpkgs=${nixpkgs}" ]; @@ -33,30 +36,39 @@ # Test that 'id' gives the expected result in various configurations. # Existing UIDs, sandbox. - host.succeed("nix build -v --no-auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-1") + host.succeed("nix build -v --no-auto-allocate-uids --no-use-cgroups --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-1") host.succeed("[[ $(cat ./result) = 'uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)' ]]") # Existing UIDs, no sandbox. - host.succeed("nix build -v --no-auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-2") + host.succeed("nix build -v --no-auto-allocate-uids --no-use-cgroups --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-2") host.succeed("[[ $(cat ./result) = 'uid=30001(nixbld1) gid=30000(nixbld) groups=30000(nixbld)' ]]") - # Auto-allocated UIDs, sandbox. - host.succeed("nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-3") - host.succeed("[[ $(cat ./result) = 'uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)' ]]") + # Auto-allocated UIDs, sandbox but no cgroups. + host.fail("nix build -v --auto-allocate-uids --no-use-cgroups --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-3") - # Auto-allocated UIDs, no sandbox. - host.succeed("nix build -v --auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-4") - host.succeed("[[ $(cat ./result) = 'uid=872415232 gid=30000(nixbld) groups=30000(nixbld)' ]]") + # Auto-allocated UIDs, no sandbox but no cgroups. + host.fail("nix build -v --auto-allocate-uids --no-use-cgroups --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-4") - # Auto-allocated UIDs, UID range, sandbox. - host.succeed("nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-5 --arg uidRange true") + # Auto-allocated UIDs, UID range, sandbox, via daemon. + host.succeed("NIX_REMOTE=daemon nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-5 --arg uidRange true") host.succeed("[[ $(cat ./result) = 'uid=0(root) gid=0(root) groups=0(root)' ]]") - # Auto-allocated UIDs, UID range, no sandbox. - host.fail("nix build -v --auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-6 --arg uidRange true") + # Auto-allocated UIDs, UID range, no sandbox, with and without daemon. + host.fail("NIX_REMOTE=daemon nix build -v --auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-6 --arg uidRange true") + host.fail("nix build -v --auto-allocate-uids --no-use-cgroups --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-7 --arg uidRange true") - # Run systemd-nspawn in a Nix build. - host.succeed("nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./systemd-nspawn.nix} --argstr nixpkgs ${nixpkgs}") + # Run systemd-nspawn in a Nix build, via daemon. + host.succeed("NIX_REMOTE=daemon nix build -vv --auto-allocate-uids --sandbox -L --offline --impure --file ${./systemd-nspawn.nix} --argstr nixpkgs ${nixpkgs}") + host.succeed("[[ $(cat ./result/msg) = 'Hello World' ]]") + + # Auto-allocated UIDs, UID range, sandbox, WITHOUT daemon (so-called: Nix as root), IN `systemd-run` transient scope. + # `systemd-run` is CRITICAL to run this successfully. + host.succeed("systemd-run --same-dir --wait -p Delegate=yes -p DelegateSubgroup=supervisor nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-8 --arg uidRange true -I nixpkgs=${nixpkgs}") + host.succeed("[[ $(cat ./result) = 'uid=0(root) gid=0(root) groups=0(root)' ]]") + + # Run systemd-nspawn in a Nix build, WITHOUT daemon (so-called: Nix as root) IN `systemd-run`. + # `systemd-run` is CRITICAL to run this successfully. + host.succeed("systemd-run --same-dir --wait -p Delegate=yes -p DelegateSubgroup=supervisor nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./systemd-nspawn.nix} --argstr nixpkgs ${nixpkgs} -I nixpkgs=${nixpkgs}") host.succeed("[[ $(cat ./result/msg) = 'Hello World' ]]") ''; diff --git a/tests/nixos/containers/id-test.nix b/tests/nixos/containers/id-test.nix new file mode 100644 index 000000000..8eb9d38f9 --- /dev/null +++ b/tests/nixos/containers/id-test.nix @@ -0,0 +1,8 @@ +{ name, uidRange ? false }: + +with import {}; + +runCommand name + { requiredSystemFeatures = if uidRange then ["uid-range"] else []; + } + "id; id > $out" diff --git a/tests/nixos/containers/systemd-nspawn.nix b/tests/nixos/containers/systemd-nspawn.nix new file mode 100644 index 000000000..1dad4ebd7 --- /dev/null +++ b/tests/nixos/containers/systemd-nspawn.nix @@ -0,0 +1,80 @@ +{ nixpkgs }: + +let + + machine = { config, pkgs, ... }: + { + system.stateVersion = "22.05"; + boot.isContainer = true; + systemd.services.console-getty.enable = false; + networking.dhcpcd.enable = false; + + services.httpd = { + enable = true; + adminAddr = "nixos@example.org"; + }; + + systemd.services.test = { + wantedBy = [ "multi-user.target" ]; + after = [ "httpd.service" ]; + script = '' + source /.env + echo "Hello World" > $out/msg + ls -lR /dev > $out/dev + ${pkgs.curl}/bin/curl -sS --fail http://localhost/ > $out/page.html + ''; + unitConfig = { + FailureAction = "exit-force"; + FailureActionExitStatus = 42; + SuccessAction = "exit-force"; + }; + }; + }; + + cfg = (import (nixpkgs + "/nixos/lib/eval-config.nix") { + modules = [ machine ]; + system = "x86_64-linux"; + }); + + config = cfg.config; + +in + +with cfg._module.args.pkgs; + +runCommand "test" + { buildInputs = [ config.system.path ]; + requiredSystemFeatures = [ "uid-range" ]; + toplevel = config.system.build.toplevel; + } + '' + root=$(pwd)/root + mkdir -p $root $root/etc + + export > $root/.env + + # Make /run a tmpfs to shut up a systemd warning. + mkdir /run + mount -t tmpfs none /run + + mount -t cgroup2 none /sys/fs/cgroup + + mkdir -p $out + + chmod +w /etc + touch /etc/os-release + echo a5ea3f98dedc0278b6f3cc8c37eeaeac > /etc/machine-id + + SYSTEMD_NSPAWN_UNIFIED_HIERARCHY=1 \ + ${config.systemd.package}/bin/systemd-nspawn \ + --keep-unit \ + -M ${config.networking.hostName} -D "$root" \ + --register=no \ + --resolv-conf=off \ + --bind-ro=/nix/store \ + --bind=$out \ + --bind=/proc:/run/host/proc \ + --bind=/sys:/run/host/sys \ + --private-network \ + $toplevel/init + '' diff --git a/tests/nixos/default.nix b/tests/nixos/default.nix index 986c15493..c9a2d1055 100644 --- a/tests/nixos/default.nix +++ b/tests/nixos/default.nix @@ -150,6 +150,10 @@ in tarballFlakes = runNixOSTestFor "x86_64-linux" ./tarball-flakes.nix; + containers = runNixOSTestFor "x86_64-linux" ./containers/containers.nix; + + cgroups = runNixOSTestFor "x86_64-linux" ./cgroups; + setuid = lib.genAttrs ["i686-linux" "x86_64-linux"] (system: runNixOSTestFor system ./setuid/setuid.nix);