libstore/build: cgroup delegation to sandbox
We offer full cgroup delegation to our sandbox now, required for running containers inside the sandbox. To run systemd-nspawn or containers managers inside the sandbox, there is a need for one extra ingredient now: control over your own cgroup subtree inside the sandbox. If, in addition, you need multiple UIDs, for e.g. rootless usecases, you need to run with the `uid-range` system feature. Therefore, when the daemon or Nix runs under the right condition, e.g. systemd-style delegation of the cgroup subtree while placing the nix-daemon in a supervisor sub-cgroup, we create a new sub-cgroup for each build based on the build UID and delegate that sub-cgroup to the builder's process. Additionally, `uid-range` always request the `cgroups` feature now, as `uid-range` builds would probably always benefit from having cgroups delegated, but the converse is not true. Inspired from https://github.com/NixOS/nix/pull/11412 with a different design that does not use function-local statics to derive the root cgroup. Co-authored-by: Linus Heckemann <git@sphalerite.org> Co-authored-by: Parker Hoyes <contact@parkerhoyes.com> Change-Id: Ic8947c5adaf4b5bbd153386e05fad65a935274fa Signed-off-by: Raito Bezarius <raito@lix.systems>
This commit is contained in:
Raito Bezarius
co-authored by
Linus Heckemann
Parker Hoyes
Linus Heckemann
Parker Hoyes
parent
1783d5b348
commit
a527bb251a
@@ -0,0 +1,69 @@
|
||||
---
|
||||
synopsis: New cgroup delegation model
|
||||
issues: [fj#537, fj#77]
|
||||
cls: [3230]
|
||||
category: "Breaking Changes"
|
||||
credits: [raito, horrors, lheckemann]
|
||||
---
|
||||
|
||||
Builds using cgroups (i.e. `use-cgroups = true` and the experimental feature
|
||||
`cgroups`) now always delegate a cgroup tree to the sandbox.
|
||||
|
||||
Compared to the original C++ Nix project, our delegation includes the
|
||||
`subtree_control` file as well, which means that the sandbox can disable
|
||||
certain controllers in its own cgroup tree.
|
||||
|
||||
This is a breaking change because this requires the Nix daemon to run with an
|
||||
already delegated cgroup tree by the service manager.
|
||||
|
||||
## How to setup the cgroup tree with systemd?
|
||||
|
||||
systemd offers knobs to perform the required setup using:
|
||||
|
||||
```
|
||||
[Unit]
|
||||
Delegate=yes
|
||||
DelegateSubtree=supervisor
|
||||
```
|
||||
|
||||
These directives are now included in our systemd packaging.
|
||||
|
||||
## What about using Nix as root without connecting to the daemon?
|
||||
|
||||
Builds run as `root` without connecting to the daemon relying on the cgroup
|
||||
feature are now broken, i.e.
|
||||
|
||||
```console
|
||||
# nix-build --use-cgroups --sandbox ... # will not work
|
||||
```
|
||||
|
||||
Consider doing instead:
|
||||
|
||||
```console
|
||||
# systemd-run --same-dir --wait -p Delegate=yes -p DelegateSubgroup=supervisor nix-build --use-cgroups ...
|
||||
```
|
||||
|
||||
If you need to disable cgroups temporarily, remember that you can do
|
||||
`NIX_CONF='include /etc/nix/nix.conf\nuse-cgroups = false' nix-build ...` or
|
||||
`nix-build --no-use-cgroups ...`.
|
||||
|
||||
## What about other service managers than systemd?
|
||||
|
||||
systemd has a [documentation](https://systemd.io/CGROUP_DELEGATION/) on how to
|
||||
handle cgroup delegation from service management perspective.
|
||||
|
||||
If your service manager adheres to systemd semantics, e.g. writing an extended
|
||||
attribute `user.delegate=1` on the delegated cgroup tree directory and moving
|
||||
the `nix-daemon` process inside a cgroup tree to respect the inner process
|
||||
rule, then, the feature will work as well.
|
||||
|
||||
## Why is the cgroup feature still experimental?
|
||||
|
||||
While the cgroup feature unlocks many use cases, its behavior and integration (e.g. user experience), especially at scale on build farms or in multi-tenant environments, are not yet fully matured. There’s also potential for deeper systemd integration (e.g. using slices and scopes) that has not been fully explored.
|
||||
|
||||
To avoid locking in an unstable interface, we’re keeping the experimental flag until we have validated the feature across a broader range of scenarios, including but not limited to:
|
||||
|
||||
* Nix as root
|
||||
* Hydra-style build farms
|
||||
* Forgejo CI runners
|
||||
* Shared remote builders
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
synopsis: uid-range depends on cgroups
|
||||
issues: []
|
||||
cls: [3230]
|
||||
category: "Breaking Changes"
|
||||
credits: [raito, horrors]
|
||||
---
|
||||
|
||||
`uid-range` builds now depends on `cgroups`, an experimental feature.
|
||||
|
||||
`uid-range` builds already depended upon `auto-allocate-uids`, another experimental feature.
|
||||
|
||||
The rationale for doing so is that `uid-range` provides a sandbox with many
|
||||
UIDs, this is useful for re-mapping them into a nested namespace, e.g. a
|
||||
container.
|
||||
@@ -402,9 +402,11 @@ void LocalDerivationGoal::cleanupPostOutputsRegisteredModeNonCheck()
|
||||
|
||||
kj::Promise<Result<void>> LocalDerivationGoal::startBuilder()
|
||||
try {
|
||||
#if !(__linux__)
|
||||
if (buildUser && buildUser->getUIDCount() != 1) {
|
||||
throw Error("cgroups are not supported on this platform");
|
||||
}
|
||||
#endif
|
||||
|
||||
/* Make sure that no other processes are executing under the
|
||||
sandbox uids. This must be done before any chownToBuilder()
|
||||
|
||||
@@ -4,9 +4,20 @@
|
||||
#include "lix/libstore/build/derivation-goal.hh"
|
||||
#include "lix/libstore/local-store.hh"
|
||||
#include "lix/libutil/processes.hh"
|
||||
#include "lix/libutil/cgroup.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
struct BuildContext
|
||||
{
|
||||
#ifdef __linux__
|
||||
/**
|
||||
* Control group for this derivation goal.
|
||||
*/
|
||||
std::optional<AutoDestroyCgroup> cgroup;
|
||||
#endif
|
||||
};
|
||||
|
||||
struct LocalDerivationGoal : public DerivationGoal
|
||||
{
|
||||
LocalStore & getLocalStore();
|
||||
@@ -16,6 +27,11 @@ struct LocalDerivationGoal : public DerivationGoal
|
||||
*/
|
||||
std::unique_ptr<UserLock> buildUser;
|
||||
|
||||
/**
|
||||
* Build context for this goal.
|
||||
*/
|
||||
BuildContext context;
|
||||
|
||||
/**
|
||||
* The process ID of the builder.
|
||||
*/
|
||||
|
||||
@@ -37,6 +37,85 @@ Worker::Worker(Store & store, Store & evalStore)
|
||||
, children(errorHandler)
|
||||
{
|
||||
/* Debugging: prevent recursive workers. */
|
||||
|
||||
#ifdef __linux__
|
||||
|
||||
/* When cgroups are used, we need to verify if our context allows
|
||||
* us to fully use cgroups delegation or ability to kill certain cgroups.
|
||||
*
|
||||
* Note that `uid-range` builds implies cgroups, the converse is false.
|
||||
* A `uid-range` build is defined as `settings.autoAllocateUids && settings.uidCount >= 1`
|
||||
*/
|
||||
|
||||
if (settings.autoAllocateUids && settings.uidCount > 1 && !settings.useCgroups) {
|
||||
throw Error(
|
||||
"Running builds with UID ranges (setting `%s` enabled and `%d` UIDs) requires the "
|
||||
"setting '%s' to be enabled.",
|
||||
settings.autoAllocateUids.name,
|
||||
settings.uidCount,
|
||||
settings.useCgroups.name
|
||||
);
|
||||
}
|
||||
|
||||
/* Cgroup build absolutely need build user separation. */
|
||||
if (!useBuildUsers() && settings.useCgroups) {
|
||||
throw Error(
|
||||
"Running all builds with cgroups requires privilege separation for build users but Lix "
|
||||
"is not configured to use build users."
|
||||
);
|
||||
}
|
||||
|
||||
/*
|
||||
* At this point, we know that if `settings.useCgroups = true`, then `useBuildUsers() = true`.
|
||||
* UID ranges may or may not be available.
|
||||
*/
|
||||
if (settings.useCgroups) {
|
||||
if (!hasCgroupFeature(
|
||||
platformFeatures.availableCgroupFeatures, CgroupAvailableFeatureSet::CGROUPV2
|
||||
))
|
||||
{
|
||||
throw Error("Running a build with cgroups requires cgroups v2 support on the system.");
|
||||
}
|
||||
|
||||
if (!hasCgroupFeature(
|
||||
platformFeatures.availableCgroupFeatures, CgroupAvailableFeatureSet::CGROUPV2_KILL
|
||||
))
|
||||
{
|
||||
throw Error(
|
||||
"Running a build with cgroups requires cgroups v2 kill feature which requires "
|
||||
"a Linux kernel newer than 5.14."
|
||||
);
|
||||
}
|
||||
|
||||
if (!hasCgroupFeature(
|
||||
platformFeatures.availableCgroupFeatures,
|
||||
CgroupAvailableFeatureSet::CGROUPV2_PARENT_DELEGATED
|
||||
))
|
||||
{
|
||||
if (hasCgroupFeature(
|
||||
platformFeatures.availableCgroupFeatures,
|
||||
CgroupAvailableFeatureSet::CGROUPV2_SELF_DELEGATED
|
||||
))
|
||||
{
|
||||
throw Error(
|
||||
"Running a build with cgroups requires the parent cgroup tree to be "
|
||||
"delegated, but only this process' cgroup is delegated.\n"
|
||||
"If you used systemd with `Delegate=yes`, consider moving the process in a "
|
||||
"sub-cgroup or use `DelegateSubtree=` to move it automatically.\n"
|
||||
"See <https://systemd.io/CGROUP_DELEGATION/> for more information."
|
||||
);
|
||||
} else {
|
||||
throw Error(
|
||||
"Running a build with cgroups requires the parent cgroup tree to be "
|
||||
"delgated.\n"
|
||||
"If you use systemd, adding `Delegate=yes` and `DelegateSubtree=supervisor` to "
|
||||
"the [Unit] section will delegate the parent cgroup tree.\n"
|
||||
"See <https://systemd.io/CGROUP_DELEGATION/> for more information."
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
#include "lix/libutil/concepts.hh"
|
||||
#include "lix/libutil/notifying-counter.hh"
|
||||
#include "lix/libutil/result.hh"
|
||||
#include "lix/libutil/cgroup.hh"
|
||||
#include "lix/libutil/types.hh"
|
||||
#include "lix/libstore/lock.hh"
|
||||
#include "lix/libstore/store-api.hh"
|
||||
@@ -187,6 +188,15 @@ public:
|
||||
Store & evalStore;
|
||||
AsyncSemaphore substitutions, localBuilds;
|
||||
|
||||
struct PlatformFeatures
|
||||
{
|
||||
#ifdef __linux__
|
||||
CgroupAvailableFeatureSet availableCgroupFeatures = detectAvailableCgroupFeatures();
|
||||
#endif
|
||||
};
|
||||
|
||||
PlatformFeatures platformFeatures;
|
||||
|
||||
private:
|
||||
kj::TaskSet children;
|
||||
|
||||
|
||||
@@ -108,6 +108,7 @@ libstore_setting_definitions = files(
|
||||
'settings/timeout.md',
|
||||
'settings/trusted-public-keys.md',
|
||||
'settings/trusted-substituters.md',
|
||||
'settings/use-cgroups.md',
|
||||
'settings/use-sqlite-wal.md',
|
||||
'settings/use-xdg-base-directories.md',
|
||||
# keep-sorted end
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
#include "lix/libstore/build/worker.hh"
|
||||
#include "lix/libutil/cgroup.hh"
|
||||
#include "lix/libutil/finally.hh"
|
||||
#include "lix/libstore/gc-store.hh"
|
||||
#include "lix/libutil/signals.hh"
|
||||
@@ -780,12 +781,14 @@ void LinuxLocalDerivationGoal::prepareSandbox()
|
||||
nobody account. The latter is kind of a hack to support
|
||||
Samba-in-QEMU. */
|
||||
createDirs(chrootRootDir + "/etc");
|
||||
if (parsedDrv->useUidRange())
|
||||
chownToBuilder(chrootRootDir + "/etc");
|
||||
|
||||
if (parsedDrv->useUidRange() && (!buildUser || buildUser->getUIDCount() < 65536))
|
||||
throw Error("feature 'uid-range' requires the setting '%s' to be enabled", settings.autoAllocateUids.name);
|
||||
|
||||
if (parsedDrv->useUidRange()) {
|
||||
chownToBuilder(chrootRootDir + "/etc");
|
||||
}
|
||||
|
||||
/* Create /etc/hosts with localhost entry. */
|
||||
if (derivationType->isSandboxed())
|
||||
writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n::1 localhost\n");
|
||||
@@ -818,6 +821,42 @@ void LinuxLocalDerivationGoal::prepareSandbox()
|
||||
for (auto & i : drv->outputsAndPaths(worker.store)) {
|
||||
pathsInChroot.erase(worker.store.printStorePath(i.second.second));
|
||||
}
|
||||
|
||||
if (buildUser && (buildUser->getUIDCount() != 1 || settings.useCgroups)) {
|
||||
context.cgroup.emplace(
|
||||
settings.nixStateDir + "/cgroups",
|
||||
fmt("nix-build-uid-%d", buildUser->getUID()),
|
||||
buildUser->getUID(),
|
||||
buildUser->getGID()
|
||||
);
|
||||
|
||||
debug("using cgroup '%s' for build", context.cgroup->name());
|
||||
|
||||
/* TODO(raito): it would be very nice if we could propagate system features
|
||||
* based on which cgroup controllers are available in `context.cgroup`
|
||||
* so that we would re-schedule any derivation that actually has
|
||||
* anti-affinity or pro-affinity with certain cgroup controllers, e.g.
|
||||
* a derivation that is very sensitive to the memory cgroup controller
|
||||
* for performance reason.
|
||||
*
|
||||
* Unfortunately, the current design of system features prevent mutation
|
||||
* and worse, we are too late for rescheduling this derivation.
|
||||
*
|
||||
* Therefore, we decide to always copy all the available controllers
|
||||
* to the delegated cgroup.
|
||||
*/
|
||||
debug(
|
||||
"available cgroup controllers for cgroup '%s': '%s'",
|
||||
context.cgroup->name(),
|
||||
concatStringsSep(",", context.cgroup->controllers())
|
||||
);
|
||||
}
|
||||
|
||||
if (parsedDrv->useUidRange() && !context.cgroup) {
|
||||
throw Error(
|
||||
"feature 'uid-range' requires the setting '%s' to be enabled", settings.useCgroups.name
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Pid LinuxLocalDerivationGoal::startChild(std::function<void()> openSlave)
|
||||
@@ -957,6 +996,11 @@ Pid LinuxLocalDerivationGoal::startChild(std::function<void()> openSlave)
|
||||
"nixbld:!:%1%:\n"
|
||||
"nogroup:x:65534:\n", sandboxGid()));
|
||||
|
||||
/* Migrate the child inside the available control group. */
|
||||
if (context.cgroup) {
|
||||
context.cgroup->adoptProcess(pid.get());
|
||||
}
|
||||
|
||||
/* Signal the builder that we've updated its user namespace. */
|
||||
writeFull(userNamespaceSync.writeSide.get(), "1");
|
||||
|
||||
@@ -965,7 +1009,18 @@ Pid LinuxLocalDerivationGoal::startChild(std::function<void()> openSlave)
|
||||
|
||||
void LinuxLocalDerivationGoal::killSandbox(bool getStats)
|
||||
{
|
||||
if (!useChroot) {
|
||||
if (context.cgroup) {
|
||||
context.cgroup->kill();
|
||||
if (getStats) {
|
||||
auto stats = context.cgroup->getStatistics();
|
||||
buildResult.cpuUser = stats.cpuUser;
|
||||
buildResult.cpuSystem = stats.cpuSystem;
|
||||
}
|
||||
/* It may be desireable to destroy the cgroup here
|
||||
* but we may be calling this at the start of the build
|
||||
* to ensure that no leftover process are running under sandbox UIDs.
|
||||
* With control groups, that's already impossible. */
|
||||
} else if (!useChroot) {
|
||||
/* Linux sandboxes use PID namespaces, which ensure that processes cannot escape from a build.
|
||||
Therefore, we don't need to kill all processes belonging to the build user.
|
||||
This avoids processes unrelated to the build being killed, thus avoiding: https://git.lix.systems/lix-project/lix/issues/667 */
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
name: use-cgroups
|
||||
internalName: useCgroups
|
||||
platforms: [linux]
|
||||
type: bool
|
||||
default: false
|
||||
experimentalFeature: cgroups
|
||||
---
|
||||
Whether to execute builds inside cgroups.
|
||||
|
||||
Cgroups are required and enabled automatically for derivations
|
||||
that require the `uid-range` system feature.
|
||||
|
||||
Cgroups requires a cgroup delegation according to <https://systemd.io/CGROUP_DELEGATION/>, i.e.
|
||||
the Nix process (`nix-daemon` or any single user command) should run in a cgroup tree of a parent cgroup which possess the `user.delegate=1` extended attribute.
|
||||
|
||||
In this scenario, Nix will run builds in a sibling cgroup named `nix-build-uid-<build user uid>`.
|
||||
+335
-96
@@ -1,7 +1,8 @@
|
||||
#include "lix/libutil/logging.hh"
|
||||
#include "regex.hh"
|
||||
#include "logging.hh"
|
||||
#if __linux__
|
||||
|
||||
#include "regex.hh"
|
||||
|
||||
#include "lix/libutil/cgroup.hh"
|
||||
#include "lix/libutil/file-system.hh"
|
||||
#include "lix/libutil/finally.hh"
|
||||
@@ -10,32 +11,29 @@
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <regex>
|
||||
#include <unordered_set>
|
||||
#include <thread>
|
||||
#include <signal.h>
|
||||
|
||||
#include <dirent.h>
|
||||
#include <mntent.h>
|
||||
#include <sys/xattr.h>
|
||||
|
||||
namespace nix {
|
||||
|
||||
std::optional<Path> getCgroupFS()
|
||||
static bool isCgroupDelegated(const Path & path)
|
||||
{
|
||||
static auto res = [&]() -> std::optional<Path> {
|
||||
auto fp = fopen("/proc/mounts", "r");
|
||||
if (!fp) return std::nullopt;
|
||||
Finally delFP = [&]() { fclose(fp); };
|
||||
while (auto ent = getmntent(fp))
|
||||
if (std::string_view(ent->mnt_type) == "cgroup2")
|
||||
return ent->mnt_dir;
|
||||
char delegate_xattr;
|
||||
|
||||
return std::nullopt;
|
||||
}();
|
||||
return res;
|
||||
if (getxattr(path.c_str(), "user.delegate", &delegate_xattr, sizeof(delegate_xattr)) >= 1) {
|
||||
if (delegate_xattr != '1') {
|
||||
throw Error("Unexpected `user.delegate` xattr: '%c'", delegate_xattr);
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
// FIXME: obsolete, check for cgroup2
|
||||
std::map<std::string, std::string> getCgroups(const Path & cgroupFile)
|
||||
static std::map<std::string, std::string> getCgroups(const Path & cgroupFile)
|
||||
{
|
||||
std::map<std::string, std::string> cgroups;
|
||||
|
||||
@@ -52,101 +50,342 @@ std::map<std::string, std::string> getCgroups(const Path & cgroupFile)
|
||||
return cgroups;
|
||||
}
|
||||
|
||||
static CgroupStats destroyCgroup(const Path & cgroup, bool returnStats)
|
||||
static CgroupStats readStatistics(const std::filesystem::path & cgroup)
|
||||
{
|
||||
if (!pathExists(cgroup)) return {};
|
||||
|
||||
auto procsFile = cgroup + "/cgroup.procs";
|
||||
|
||||
if (!pathExists(procsFile))
|
||||
throw Error("'%s' is not a cgroup", cgroup);
|
||||
|
||||
/* Use the fast way to kill every process in a cgroup, if
|
||||
available. */
|
||||
auto killFile = cgroup + "/cgroup.kill";
|
||||
if (pathExists(killFile))
|
||||
writeFile(killFile, "1");
|
||||
|
||||
/* Otherwise, manually kill every process in the subcgroups and
|
||||
this cgroup. */
|
||||
for (auto & entry : readDirectory(cgroup)) {
|
||||
if (entry.type != DT_DIR) continue;
|
||||
destroyCgroup(cgroup + "/" + entry.name, false);
|
||||
}
|
||||
|
||||
int round = 1;
|
||||
|
||||
std::unordered_set<pid_t> pidsShown;
|
||||
|
||||
while (true) {
|
||||
auto pids = tokenizeString<std::vector<std::string>>(readFile(procsFile));
|
||||
|
||||
if (pids.empty()) break;
|
||||
|
||||
if (round > 20)
|
||||
throw Error("cannot kill cgroup '%s'", cgroup);
|
||||
|
||||
for (auto & pid_s : pids) {
|
||||
pid_t pid;
|
||||
if (auto o = string2Int<pid_t>(pid_s))
|
||||
pid = *o;
|
||||
else
|
||||
throw Error("invalid pid '%s'", pid);
|
||||
if (pidsShown.insert(pid).second) {
|
||||
try {
|
||||
auto cmdline = readFile(fmt("/proc/%d/cmdline", pid));
|
||||
using namespace std::string_literals;
|
||||
warn("killing stray builder process %d (%s)...",
|
||||
pid, trim(replaceStrings(cmdline, "\0"s, " ")));
|
||||
} catch (SysError &) {
|
||||
}
|
||||
}
|
||||
// FIXME: pid wraparound
|
||||
if (kill(pid, SIGKILL) == -1 && errno != ESRCH)
|
||||
throw SysError("killing member %d of cgroup '%s'", pid, cgroup);
|
||||
}
|
||||
|
||||
auto sleep = std::chrono::milliseconds((int) std::pow(2.0, std::min(round, 10)));
|
||||
if (sleep.count() > 100)
|
||||
printError("waiting for %d ms for cgroup '%s' to become empty", sleep.count(), cgroup);
|
||||
std::this_thread::sleep_for(sleep);
|
||||
round++;
|
||||
}
|
||||
|
||||
CgroupStats stats;
|
||||
|
||||
if (returnStats) {
|
||||
auto cpustatPath = cgroup + "/cpu.stat";
|
||||
auto cpustatPath = cgroup / "cpu.stat";
|
||||
|
||||
if (pathExists(cpustatPath)) {
|
||||
for (auto & line : tokenizeString<std::vector<std::string>>(readFile(cpustatPath), "\n")) {
|
||||
std::string_view userPrefix = "user_usec ";
|
||||
if (line.starts_with(userPrefix)) {
|
||||
auto n = string2Int<uint64_t>(line.substr(userPrefix.size()));
|
||||
if (n) stats.cpuUser = std::chrono::microseconds(*n);
|
||||
if (pathExists(cpustatPath)) {
|
||||
for (auto & line : tokenizeString<std::vector<std::string>>(readFile(cpustatPath), "\n")) {
|
||||
std::string_view userPrefix = "user_usec ";
|
||||
if (line.starts_with(userPrefix)) {
|
||||
auto n = string2Int<uint64_t>(line.substr(userPrefix.size()));
|
||||
if (n) {
|
||||
stats.cpuUser = std::chrono::microseconds(*n);
|
||||
}
|
||||
}
|
||||
|
||||
std::string_view systemPrefix = "system_usec ";
|
||||
if (line.starts_with(systemPrefix)) {
|
||||
auto n = string2Int<uint64_t>(line.substr(systemPrefix.size()));
|
||||
if (n) stats.cpuSystem = std::chrono::microseconds(*n);
|
||||
std::string_view systemPrefix = "system_usec ";
|
||||
if (line.starts_with(systemPrefix)) {
|
||||
auto n = string2Int<uint64_t>(line.substr(systemPrefix.size()));
|
||||
if (n) {
|
||||
stats.cpuSystem = std::chrono::microseconds(*n);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
if (rmdir(cgroup.c_str()) == -1)
|
||||
throw SysError("deleting cgroup '%s'", cgroup);
|
||||
|
||||
return stats;
|
||||
}
|
||||
|
||||
CgroupStats destroyCgroup(const Path & cgroup)
|
||||
static void killCgroup(const std::string & name, const std::filesystem::path & cgroup)
|
||||
{
|
||||
return destroyCgroup(cgroup, true);
|
||||
auto killFile = cgroup / "cgroup.kill";
|
||||
if (pathExists(killFile))
|
||||
writeFile(killFile, "1");
|
||||
else {
|
||||
throw SysError(
|
||||
"cgroup '%s' at '%s' does not possess `cgroup.kill` ; are you running Lix on a kernel "
|
||||
"older than 5.14 with cgroups?",
|
||||
name,
|
||||
cgroup
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
static std::optional<CgroupStats>
|
||||
destroyCgroup(const std::string & name, const std::filesystem::path & aliveCgroup)
|
||||
{
|
||||
debug("destroying cgroup '%s' at '%s'", name, aliveCgroup);
|
||||
if (!pathExists(aliveCgroup)) {
|
||||
debug("destroying cgroup '%s' already destroyed", name);
|
||||
return {};
|
||||
}
|
||||
|
||||
auto procsFile = aliveCgroup / "cgroup.procs";
|
||||
|
||||
if (!pathExists(procsFile)) {
|
||||
throw SysError(
|
||||
"cgroup '%s' at '%s' has an invalid cgroup hierarchy (missing `cgroup.procs`)",
|
||||
name,
|
||||
aliveCgroup
|
||||
);
|
||||
}
|
||||
|
||||
killCgroup(name, aliveCgroup);
|
||||
|
||||
CgroupStats stats = readStatistics(aliveCgroup);
|
||||
|
||||
if (rmdir(aliveCgroup.c_str()) == -1) {
|
||||
throw SysError("deleting cgroup '%s' at '%s'", name, aliveCgroup);
|
||||
}
|
||||
|
||||
debug("cgroup '%s' destroyed", name);
|
||||
|
||||
return stats;
|
||||
}
|
||||
|
||||
CgroupHierarchy getLocalHierarchy(const std::filesystem::path & cgroupFilesystem)
|
||||
{
|
||||
CgroupHierarchy hierarchy;
|
||||
|
||||
auto ourCgroups = getCgroups("/proc/self/cgroup");
|
||||
auto ourCgroup = ourCgroups[""];
|
||||
|
||||
if (ourCgroup == "") {
|
||||
throw Error("cannot determine cgroup name from '/proc/self/cgroup'");
|
||||
}
|
||||
|
||||
if (ourCgroup[0] == '/') {
|
||||
ourCgroup.erase(0, 1);
|
||||
}
|
||||
|
||||
auto ourCgroupPath = (cgroupFilesystem / ourCgroup).lexically_normal();
|
||||
|
||||
if (!pathExists(ourCgroupPath)) {
|
||||
throw Error("expected cgroup directory '%s'", ourCgroupPath);
|
||||
}
|
||||
|
||||
hierarchy.ourCgroupPath = ourCgroupPath;
|
||||
|
||||
return hierarchy;
|
||||
}
|
||||
|
||||
CgroupAvailableFeatureSet operator|(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs)
|
||||
{
|
||||
return static_cast<CgroupAvailableFeatureSet>(
|
||||
static_cast<std::underlying_type<CgroupAvailableFeatureSet>::type>(lhs)
|
||||
| static_cast<std::underlying_type<CgroupAvailableFeatureSet>::type>(rhs)
|
||||
);
|
||||
}
|
||||
|
||||
CgroupAvailableFeatureSet &
|
||||
operator|=(CgroupAvailableFeatureSet & lhs, CgroupAvailableFeatureSet rhs)
|
||||
{
|
||||
return lhs = lhs | rhs;
|
||||
}
|
||||
|
||||
CgroupAvailableFeatureSet operator&(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs)
|
||||
{
|
||||
return static_cast<CgroupAvailableFeatureSet>(
|
||||
static_cast<std::underlying_type<CgroupAvailableFeatureSet>::type>(lhs)
|
||||
& static_cast<std::underlying_type<CgroupAvailableFeatureSet>::type>(rhs)
|
||||
);
|
||||
}
|
||||
|
||||
bool hasCgroupFeature(CgroupAvailableFeatureSet featureSet, CgroupAvailableFeatureSet testedFeature)
|
||||
{
|
||||
return static_cast<std::underlying_type<CgroupAvailableFeatureSet>::type>(
|
||||
featureSet & testedFeature
|
||||
)
|
||||
!= 0;
|
||||
}
|
||||
|
||||
CgroupAvailableFeatureSet detectAvailableCgroupFeatures()
|
||||
{
|
||||
CgroupAvailableFeatureSet features = {};
|
||||
|
||||
auto fs = getCgroupFS();
|
||||
|
||||
if (fs && !fs->empty()) {
|
||||
features |= CgroupAvailableFeatureSet::CGROUPV2;
|
||||
|
||||
auto localHierarchy = getLocalHierarchy(*fs);
|
||||
if (pathExists(localHierarchy.ourCgroupPath / "cgroup.kill")) {
|
||||
features |= CgroupAvailableFeatureSet::CGROUPV2_KILL;
|
||||
}
|
||||
|
||||
if (isCgroupDelegated(localHierarchy.ourCgroupPath)) {
|
||||
features |= CgroupAvailableFeatureSet::CGROUPV2_SELF_DELEGATED;
|
||||
}
|
||||
|
||||
auto parentCgroupPath = localHierarchy.parentCgroupPath();
|
||||
if (parentCgroupPath && isCgroupDelegated(*parentCgroupPath)) {
|
||||
features |= CgroupAvailableFeatureSet::CGROUPV2_PARENT_DELEGATED;
|
||||
}
|
||||
}
|
||||
|
||||
return features;
|
||||
}
|
||||
|
||||
static std::vector<std::string> readControllers(const std::filesystem::path & cgroupPath)
|
||||
{
|
||||
return tokenizeString<std::vector<std::string>>(
|
||||
readFile(cgroupPath / "cgroup.controllers"), " "
|
||||
);
|
||||
}
|
||||
|
||||
AutoDestroyCgroup::AutoDestroyCgroup(
|
||||
const std::filesystem::path & cgroupRecordsDir, std::string const & name
|
||||
)
|
||||
: name_(name)
|
||||
{
|
||||
auto cgroupFilesystem = getCgroupFS();
|
||||
if (!cgroupFilesystem || cgroupFilesystem->empty()) {
|
||||
throw Error("cannot determine the path to the cgroupv2 filesystem");
|
||||
}
|
||||
|
||||
auto hierarchy = getLocalHierarchy(*cgroupFilesystem);
|
||||
auto parentCgroupPath = hierarchy.parentCgroupPath();
|
||||
assert(parentCgroupPath && "AutoDestroyCgroup cannot be used on the root cgroup");
|
||||
/* We assert that the parent cgroup is delegated at this point.
|
||||
* This is a responsibility of the caller. */
|
||||
assert(isCgroupDelegated(*parentCgroupPath) && "parent cgroup was supposed to be delegated");
|
||||
|
||||
/* All available controllers on the parent cgroup path will be delegated.
|
||||
* TODO(raito): implementing a filtering mechanism is for the future. */
|
||||
controllers_ = readControllers(*parentCgroupPath);
|
||||
|
||||
/* Enable all the controllers */
|
||||
writeFile(
|
||||
*parentCgroupPath / "cgroup.subtree_control",
|
||||
concatMapStringsSep(
|
||||
" ",
|
||||
controllers_,
|
||||
[](const std::string & controller) -> std::string { return fmt("+%s", controller); }
|
||||
)
|
||||
);
|
||||
|
||||
cgroup_ = *parentCgroupPath / name;
|
||||
|
||||
/*
|
||||
* In case we get interrupted without cleaning up the cgroup we just created,
|
||||
* we look at Nix's state directory where we record all cgroups being used
|
||||
* and destroy it before reusing it. */
|
||||
cleansePreviousInstancesAndRecordOurself(cgroupRecordsDir);
|
||||
}
|
||||
|
||||
AutoDestroyCgroup::AutoDestroyCgroup(
|
||||
const std::filesystem::path & cgroupRecordsDir, std::string const & name, uid_t uid, gid_t gid
|
||||
)
|
||||
: AutoDestroyCgroup(cgroupRecordsDir, name)
|
||||
{
|
||||
auto path = std::get<std::filesystem::path>(cgroup_);
|
||||
|
||||
if (mkdir(path.c_str(), 0755) == -1) {
|
||||
throw SysError(
|
||||
"cannot create the top-level directory at '%s' for cgroup '%s'", path, name_
|
||||
);
|
||||
}
|
||||
|
||||
if (chown(path.c_str(), uid, gid) == -1) {
|
||||
throw SysError(
|
||||
"cannot delegate the top-level directory '%s' from cgroup '%s' to user uid=%d,gid=%d",
|
||||
path,
|
||||
name_,
|
||||
uid,
|
||||
gid
|
||||
);
|
||||
}
|
||||
|
||||
AutoCloseFD cgroupFd{open(path.c_str(), O_PATH | O_NOFOLLOW)};
|
||||
for (auto node : {"procs", "threads", "subtree_control"}) {
|
||||
if (fchownat(cgroupFd.get(), fmt("cgroup.%s", node).c_str(), uid, gid, 0) == -1) {
|
||||
throw SysError(
|
||||
"cannot delegate '%s' from cgroup '%s' to user uid=%d,gid=%d", node, name_, uid, gid
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
delegation_ = {.uid = uid, .gid = gid};
|
||||
}
|
||||
|
||||
AutoDestroyCgroup::~AutoDestroyCgroup()
|
||||
{
|
||||
try {
|
||||
std::visit(
|
||||
overloaded{
|
||||
[&, this](const Path & aliveCgroup) {
|
||||
auto maybeStats = destroyCgroup(name_, aliveCgroup);
|
||||
if (!maybeStats) {
|
||||
warn(
|
||||
"cgroup '%s' was destroyed unexpectedly (something else removed the "
|
||||
"cgroup).",
|
||||
aliveCgroup
|
||||
);
|
||||
}
|
||||
},
|
||||
[&](const CgroupStats & stats) {}
|
||||
},
|
||||
cgroup_
|
||||
);
|
||||
} catch (...) {
|
||||
ignoreExceptionInDestructor();
|
||||
}
|
||||
}
|
||||
|
||||
void AutoDestroyCgroup::cleansePreviousInstancesAndRecordOurself(
|
||||
const std::filesystem::path & cgroupRecordsDir
|
||||
)
|
||||
{
|
||||
createDirs(cgroupRecordsDir);
|
||||
|
||||
auto cgroupFile = cgroupRecordsDir / name_;
|
||||
|
||||
if (pathExists(cgroupFile)) {
|
||||
auto prevCgroup = readFile(cgroupFile);
|
||||
warn("destroying past cgroup '%s' found in the state directory", name_);
|
||||
destroyCgroup(fmt("past %s", name_), prevCgroup);
|
||||
}
|
||||
|
||||
writeFile(cgroupFile, std::get<std::filesystem::path>(cgroup_).string());
|
||||
stateRecord = AutoDelete(cgroupFile, false);
|
||||
}
|
||||
|
||||
void AutoDestroyCgroup::adoptProcess(int pid)
|
||||
{
|
||||
auto path = std::get_if<std::filesystem::path>(&cgroup_);
|
||||
if (!path) {
|
||||
throw SysError("cgroup '%s' went away while adopting process '%d'", name_, pid);
|
||||
}
|
||||
|
||||
writeFile(*path / "cgroup.procs", fmt("%d", pid));
|
||||
}
|
||||
|
||||
void AutoDestroyCgroup::kill()
|
||||
{
|
||||
auto path = std::get_if<std::filesystem::path>(&cgroup_);
|
||||
if (!path) {
|
||||
throw SysError("killing cgroup '%s' but it went away", name_);
|
||||
}
|
||||
|
||||
killCgroup(name_, *path);
|
||||
}
|
||||
|
||||
CgroupStats AutoDestroyCgroup::getStatistics() const
|
||||
{
|
||||
/* Either:
|
||||
* - we need to pull statistics from an alive cgroup.
|
||||
* - we need to get historical statistics from a dead cgroup.
|
||||
*/
|
||||
return std::visit(
|
||||
nix::overloaded{
|
||||
[&](const Path & aliveCgroup) { return readStatistics(aliveCgroup); },
|
||||
[&](const CgroupStats & stats) { return stats; }
|
||||
},
|
||||
cgroup_
|
||||
);
|
||||
}
|
||||
|
||||
std::optional<std::filesystem::path> getCgroupFS()
|
||||
{
|
||||
static auto res = [&]() -> std::optional<std::filesystem::path> {
|
||||
auto fp = fopen("/proc/mounts", "r");
|
||||
if (!fp) {
|
||||
return {};
|
||||
}
|
||||
Finally delFP = [&]() { fclose(fp); };
|
||||
while (auto ent = getmntent(fp)) {
|
||||
if (std::string_view(ent->mnt_type) == "cgroup2") {
|
||||
return {ent->mnt_dir};
|
||||
}
|
||||
}
|
||||
|
||||
return {};
|
||||
}();
|
||||
return res;
|
||||
}
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
+182
-7
@@ -1,18 +1,99 @@
|
||||
#pragma once
|
||||
///@file
|
||||
|
||||
#include "file-system.hh"
|
||||
#if __linux__
|
||||
|
||||
#include <chrono>
|
||||
#include <optional>
|
||||
#include <filesystem>
|
||||
#include <kj/common.h>
|
||||
|
||||
#include "lix/libutil/types.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
std::optional<Path> getCgroupFS();
|
||||
/* This represent a cgroup hierarchy.
|
||||
* When doing cgroup delegation, it is necessary to ensure that all processes
|
||||
* lives in the leaves of the cgroup tree.
|
||||
*
|
||||
* When Nix runs a build, the cgroup tree should be arranged so that:
|
||||
*
|
||||
* Nix parent process cgroup (e.g. `system.slice` on systemd)
|
||||
* |
|
||||
* |
|
||||
* nix-daemon.service cgroup
|
||||
* / \
|
||||
* / \
|
||||
* / \
|
||||
* / \
|
||||
* / \
|
||||
* / \
|
||||
* nix-build-uid-418238 cgroup supervisor cgroup
|
||||
* | |
|
||||
* derivation build processes |
|
||||
* nix-daemon subdaemons
|
||||
*
|
||||
* Notice that if you put the nix-daemon subdaemons directly inside the `nix-daemon.service`
|
||||
* cgroup, then there would be processes in _inner cgroups_, which is a violation of cgroup best
|
||||
* practices (it opens up the path for children to compete for cgroup control).
|
||||
*
|
||||
* If you run systemd with `Delegate` and `DelegateSubgroup` options, the hierarchy is setup
|
||||
* like the above automatically.
|
||||
*
|
||||
* If you are not in these conditions, you will need to write code to move the various daemon-like
|
||||
* processes inside a sibling cgroup.
|
||||
*
|
||||
*/
|
||||
struct CgroupHierarchy
|
||||
{
|
||||
std::filesystem::path ourCgroupPath;
|
||||
std::optional<std::filesystem::path> parentCgroupPath() const
|
||||
{
|
||||
if (ourCgroupPath.has_parent_path()) {
|
||||
return ourCgroupPath.parent_path();
|
||||
} else {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
std::map<std::string, std::string> getCgroups(const Path & cgroupFile);
|
||||
/* Return the current process's view of the cgroup hierarchy, i.e.
|
||||
* the parent process's cgroup and this current process's cgroup.
|
||||
* */
|
||||
CgroupHierarchy getLocalHierarchy(std::filesystem::path const & cgroupFilesystem);
|
||||
|
||||
/* Return a path to the cgroupv2 filesystem path, if it exist */
|
||||
std::optional<std::filesystem::path> getCgroupFS();
|
||||
|
||||
/* Help detect the list of feature sets available
|
||||
* for the running kernel. */
|
||||
enum class CgroupAvailableFeatureSet : uint8_t {
|
||||
/* cgroupvs2 were detected. */
|
||||
CGROUPV2 = (1 << 0),
|
||||
/* cgroupsv2 kill capability detected. The absence of this capability is unsupported.
|
||||
* It appeared in kernel 5.14. */
|
||||
CGROUPV2_KILL = (1 << 1),
|
||||
/* Current process is delegated à la systemd-style, e.g.
|
||||
* `user.delegate=1` was written as an xattr of the cgroup directory. */
|
||||
CGROUPV2_SELF_DELEGATED = (1 << 2),
|
||||
/* Current process' parent cgroup was delegated à la systemd-style. */
|
||||
CGROUPV2_PARENT_DELEGATED = (1 << 3),
|
||||
};
|
||||
|
||||
CgroupAvailableFeatureSet operator|(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs);
|
||||
CgroupAvailableFeatureSet &
|
||||
operator|=(CgroupAvailableFeatureSet & lhs, CgroupAvailableFeatureSet rhs);
|
||||
CgroupAvailableFeatureSet operator&(CgroupAvailableFeatureSet lhs, CgroupAvailableFeatureSet rhs);
|
||||
|
||||
/* Whether the `featureSet` given do possess the `testedFeature` in the set? */
|
||||
bool hasCgroupFeature(
|
||||
CgroupAvailableFeatureSet featureSet, CgroupAvailableFeatureSet testedFeature
|
||||
);
|
||||
|
||||
/* Lix cgroup support relies on certain modern features to be available to avoid implementing many
|
||||
* legacy code paths. */
|
||||
CgroupAvailableFeatureSet detectAvailableCgroupFeatures();
|
||||
|
||||
struct CgroupStats
|
||||
{
|
||||
@@ -20,13 +101,107 @@ struct CgroupStats
|
||||
};
|
||||
|
||||
/**
|
||||
* Destroy the cgroup denoted by 'path'. The postcondition is that
|
||||
* 'path' does not exist, and thus any processes in the cgroup have
|
||||
* been killed. Also return statistics from the cgroup just before
|
||||
* destruction.
|
||||
* RAII class to hold an owned cgroup
|
||||
* which will kill all processes under its hierarchy at destruction time.
|
||||
*/
|
||||
CgroupStats destroyCgroup(const Path & cgroup);
|
||||
class AutoDestroyCgroup
|
||||
{
|
||||
private:
|
||||
struct Delegation
|
||||
{
|
||||
/*
|
||||
* Delegatee UID
|
||||
*/
|
||||
uid_t uid;
|
||||
/*
|
||||
* Delegatee GID
|
||||
*/
|
||||
gid_t gid;
|
||||
};
|
||||
|
||||
/* Friendly name of this cgroup */
|
||||
std::string name_;
|
||||
|
||||
/* Enabled controllers of this cgroup */
|
||||
std::vector<std::string> controllers_;
|
||||
|
||||
/* A cgroup can be delegated to a pair of UID/GID */
|
||||
std::optional<Delegation> delegation_;
|
||||
|
||||
/*
|
||||
* Either the cgroup exist and is alive,
|
||||
* either the cgroup was destroyed and we collected its statistics.
|
||||
*/
|
||||
std::variant<std::filesystem::path, CgroupStats> cgroup_;
|
||||
|
||||
/*
|
||||
* Path to the state record of this cgroup's existence.
|
||||
* This is used when the deletion process is interrupted
|
||||
* for the next run.
|
||||
*/
|
||||
AutoDelete stateRecord;
|
||||
|
||||
/* Kill all processes under its hierarchy and tear down the cgroup */
|
||||
void destroy();
|
||||
|
||||
/*
|
||||
* Cleanse all previous instances of this cgroup where the deletion process
|
||||
* might have been interrupted and record ourself in stead.
|
||||
* If this cgroup deletion process does not run, on the next run, this will be
|
||||
* reaped when the same name will be reused.
|
||||
*/
|
||||
void cleansePreviousInstancesAndRecordOurself(const std::filesystem::path & cgroupRecordsDir);
|
||||
public:
|
||||
KJ_DISALLOW_COPY(AutoDestroyCgroup);
|
||||
|
||||
explicit AutoDestroyCgroup(
|
||||
const std::filesystem::path & cgroupRecordsDir, std::string const & name
|
||||
);
|
||||
|
||||
/* Delegate this cgroup to the given UID and GID. */
|
||||
explicit AutoDestroyCgroup(
|
||||
const std::filesystem::path & cgroupRecordsDir,
|
||||
std::string const & name,
|
||||
uid_t uid,
|
||||
gid_t gid
|
||||
);
|
||||
~AutoDestroyCgroup();
|
||||
|
||||
std::optional<Path> path() const
|
||||
{
|
||||
return std::visit(
|
||||
nix::overloaded{
|
||||
[&](const Path & path) -> std::optional<Path> { return {path}; },
|
||||
[&](const CgroupStats &) -> std::optional<Path> { return {}; }
|
||||
},
|
||||
cgroup_
|
||||
);
|
||||
}
|
||||
|
||||
const std::string & name() const
|
||||
{
|
||||
return name_;
|
||||
}
|
||||
|
||||
const std::vector<std::string> & controllers() const
|
||||
{
|
||||
return controllers_;
|
||||
}
|
||||
|
||||
std::optional<Delegation> delegation() const
|
||||
{
|
||||
return delegation_;
|
||||
}
|
||||
|
||||
/* Adopt a process in this cgroup. */
|
||||
void adoptProcess(int pid);
|
||||
|
||||
/* Kill all processes under the control group. */
|
||||
void kill();
|
||||
|
||||
/* Return all statistics inside of this cgroup. */
|
||||
CgroupStats getStatistics() const;
|
||||
};
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
@@ -32,11 +32,8 @@ unsigned int getMaxCPU()
|
||||
auto cgroupFS = getCgroupFS();
|
||||
if (!cgroupFS) return 0;
|
||||
|
||||
auto cgroups = getCgroups("/proc/self/cgroup");
|
||||
auto cgroup = cgroups[""];
|
||||
if (cgroup == "") return 0;
|
||||
|
||||
auto cpuFile = *cgroupFS + "/" + cgroup + "/cpu.max";
|
||||
auto localHierarchy = getLocalHierarchy(*cgroupFS);
|
||||
auto cpuFile = localHierarchy.ourCgroupPath / "cpu.max";
|
||||
|
||||
auto cpuMax = readFile(cpuFile);
|
||||
auto cpuMaxParts = tokenizeString<std::vector<std::string>>(cpuMax, " \n");
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: cgroups
|
||||
internalName: Cgroups
|
||||
---
|
||||
Allows Nix to execute builds inside cgroups. See
|
||||
the [`use-cgroups`](../command-ref/conf-file.md#conf-use-cgroups) setting for details.
|
||||
@@ -142,6 +142,7 @@ libutil_headers = files(
|
||||
experimental_feature_definitions = files(
|
||||
# keep-sorted start
|
||||
'experimental-features/auto-allocate-uids.md',
|
||||
'experimental-features/cgroups.md',
|
||||
'experimental-features/coerce-integers.md',
|
||||
'experimental-features/daemon-trust-override.md',
|
||||
'experimental-features/fetch-closure.md',
|
||||
|
||||
@@ -11,6 +11,8 @@ ExecStart=@@bindir@/nix-daemon nix-daemon --daemon
|
||||
KillMode=process
|
||||
LimitNOFILE=1048576
|
||||
TasksMax=1048576
|
||||
Delegate=yes
|
||||
DelegateSubgroup=supervisor
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
{ nixpkgs, ... }:
|
||||
|
||||
{
|
||||
name = "cgroups";
|
||||
|
||||
nodes =
|
||||
{
|
||||
host =
|
||||
{ config, pkgs, ... }:
|
||||
{ virtualisation.additionalPaths = [ pkgs.stdenvNoCC ];
|
||||
nix.extraOptions =
|
||||
''
|
||||
extra-experimental-features = nix-command auto-allocate-uids cgroups
|
||||
extra-system-features = uid-range
|
||||
'';
|
||||
nix.settings.use-cgroups = true;
|
||||
nix.nixPath = [ "nixpkgs=${nixpkgs}" ];
|
||||
};
|
||||
};
|
||||
|
||||
testScript = { nodes }: ''
|
||||
start_all()
|
||||
|
||||
host.wait_for_unit("multi-user.target")
|
||||
|
||||
# Start build in background
|
||||
host.execute("NIX_REMOTE=daemon nix build --use-cgroups --auto-allocate-uids --file ${./hang.nix} >&2 &")
|
||||
service = "/sys/fs/cgroup/system.slice/nix-daemon.service"
|
||||
|
||||
# Wait for cgroups to be created
|
||||
host.succeed(f"until [ -e {service}/supervisor ]; do sleep 1; done", timeout=30)
|
||||
host.succeed(f"until [ -e {service}/nix-build-uid-* ]; do sleep 1; done", timeout=30)
|
||||
|
||||
# Check that there aren't processes where there shouldn't be, and that there are where there should be
|
||||
host.succeed(f'[ -z "$(cat {service}/cgroup.procs)" ]')
|
||||
host.succeed(f'[ -n "$(cat {service}/supervisor/cgroup.procs)" ]')
|
||||
host.succeed(f'[ -n "$(cat {service}/nix-build-uid-*/cgroup.procs)" ]')
|
||||
'';
|
||||
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{ }:
|
||||
|
||||
with import <nixpkgs> {};
|
||||
|
||||
runCommand "hang"
|
||||
{ requiredSystemFeatures = "uid-range";
|
||||
}
|
||||
''
|
||||
sleep infinity
|
||||
''
|
||||
@@ -15,10 +15,13 @@
|
||||
(import ./systemd-nspawn.nix { inherit nixpkgs; }).toplevel
|
||||
];
|
||||
virtualisation.memorySize = 4096;
|
||||
nix.settings.substituters = lib.mkForce [ ];
|
||||
nix.settings = {
|
||||
substituters = lib.mkForce [ ];
|
||||
use-cgroups = true;
|
||||
};
|
||||
nix.extraOptions =
|
||||
''
|
||||
extra-experimental-features = nix-command auto-allocate-uids
|
||||
extra-experimental-features = nix-command auto-allocate-uids cgroups
|
||||
extra-system-features = uid-range
|
||||
'';
|
||||
nix.nixPath = [ "nixpkgs=${nixpkgs}" ];
|
||||
@@ -33,30 +36,39 @@
|
||||
# Test that 'id' gives the expected result in various configurations.
|
||||
|
||||
# Existing UIDs, sandbox.
|
||||
host.succeed("nix build -v --no-auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-1")
|
||||
host.succeed("nix build -v --no-auto-allocate-uids --no-use-cgroups --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-1")
|
||||
host.succeed("[[ $(cat ./result) = 'uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)' ]]")
|
||||
|
||||
# Existing UIDs, no sandbox.
|
||||
host.succeed("nix build -v --no-auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-2")
|
||||
host.succeed("nix build -v --no-auto-allocate-uids --no-use-cgroups --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-2")
|
||||
host.succeed("[[ $(cat ./result) = 'uid=30001(nixbld1) gid=30000(nixbld) groups=30000(nixbld)' ]]")
|
||||
|
||||
# Auto-allocated UIDs, sandbox.
|
||||
host.succeed("nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-3")
|
||||
host.succeed("[[ $(cat ./result) = 'uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)' ]]")
|
||||
# Auto-allocated UIDs, sandbox but no cgroups.
|
||||
host.fail("nix build -v --auto-allocate-uids --no-use-cgroups --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-3")
|
||||
|
||||
# Auto-allocated UIDs, no sandbox.
|
||||
host.succeed("nix build -v --auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-4")
|
||||
host.succeed("[[ $(cat ./result) = 'uid=872415232 gid=30000(nixbld) groups=30000(nixbld)' ]]")
|
||||
# Auto-allocated UIDs, no sandbox but no cgroups.
|
||||
host.fail("nix build -v --auto-allocate-uids --no-use-cgroups --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-4")
|
||||
|
||||
# Auto-allocated UIDs, UID range, sandbox.
|
||||
host.succeed("nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-5 --arg uidRange true")
|
||||
# Auto-allocated UIDs, UID range, sandbox, via daemon.
|
||||
host.succeed("NIX_REMOTE=daemon nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-5 --arg uidRange true")
|
||||
host.succeed("[[ $(cat ./result) = 'uid=0(root) gid=0(root) groups=0(root)' ]]")
|
||||
|
||||
# Auto-allocated UIDs, UID range, no sandbox.
|
||||
host.fail("nix build -v --auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-6 --arg uidRange true")
|
||||
# Auto-allocated UIDs, UID range, no sandbox, with and without daemon.
|
||||
host.fail("NIX_REMOTE=daemon nix build -v --auto-allocate-uids --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-6 --arg uidRange true")
|
||||
host.fail("nix build -v --auto-allocate-uids --no-use-cgroups --no-sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-7 --arg uidRange true")
|
||||
|
||||
# Run systemd-nspawn in a Nix build.
|
||||
host.succeed("nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./systemd-nspawn.nix} --argstr nixpkgs ${nixpkgs}")
|
||||
# Run systemd-nspawn in a Nix build, via daemon.
|
||||
host.succeed("NIX_REMOTE=daemon nix build -vv --auto-allocate-uids --sandbox -L --offline --impure --file ${./systemd-nspawn.nix} --argstr nixpkgs ${nixpkgs}")
|
||||
host.succeed("[[ $(cat ./result/msg) = 'Hello World' ]]")
|
||||
|
||||
# Auto-allocated UIDs, UID range, sandbox, WITHOUT daemon (so-called: Nix as root), IN `systemd-run` transient scope.
|
||||
# `systemd-run` is CRITICAL to run this successfully.
|
||||
host.succeed("systemd-run --same-dir --wait -p Delegate=yes -p DelegateSubgroup=supervisor nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./id-test.nix} --argstr name id-test-8 --arg uidRange true -I nixpkgs=${nixpkgs}")
|
||||
host.succeed("[[ $(cat ./result) = 'uid=0(root) gid=0(root) groups=0(root)' ]]")
|
||||
|
||||
# Run systemd-nspawn in a Nix build, WITHOUT daemon (so-called: Nix as root) IN `systemd-run`.
|
||||
# `systemd-run` is CRITICAL to run this successfully.
|
||||
host.succeed("systemd-run --same-dir --wait -p Delegate=yes -p DelegateSubgroup=supervisor nix build -v --auto-allocate-uids --sandbox -L --offline --impure --file ${./systemd-nspawn.nix} --argstr nixpkgs ${nixpkgs} -I nixpkgs=${nixpkgs}")
|
||||
host.succeed("[[ $(cat ./result/msg) = 'Hello World' ]]")
|
||||
'';
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
{ name, uidRange ? false }:
|
||||
|
||||
with import <nixpkgs> {};
|
||||
|
||||
runCommand name
|
||||
{ requiredSystemFeatures = if uidRange then ["uid-range"] else [];
|
||||
}
|
||||
"id; id > $out"
|
||||
@@ -0,0 +1,80 @@
|
||||
{ nixpkgs }:
|
||||
|
||||
let
|
||||
|
||||
machine = { config, pkgs, ... }:
|
||||
{
|
||||
system.stateVersion = "22.05";
|
||||
boot.isContainer = true;
|
||||
systemd.services.console-getty.enable = false;
|
||||
networking.dhcpcd.enable = false;
|
||||
|
||||
services.httpd = {
|
||||
enable = true;
|
||||
adminAddr = "nixos@example.org";
|
||||
};
|
||||
|
||||
systemd.services.test = {
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "httpd.service" ];
|
||||
script = ''
|
||||
source /.env
|
||||
echo "Hello World" > $out/msg
|
||||
ls -lR /dev > $out/dev
|
||||
${pkgs.curl}/bin/curl -sS --fail http://localhost/ > $out/page.html
|
||||
'';
|
||||
unitConfig = {
|
||||
FailureAction = "exit-force";
|
||||
FailureActionExitStatus = 42;
|
||||
SuccessAction = "exit-force";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
cfg = (import (nixpkgs + "/nixos/lib/eval-config.nix") {
|
||||
modules = [ machine ];
|
||||
system = "x86_64-linux";
|
||||
});
|
||||
|
||||
config = cfg.config;
|
||||
|
||||
in
|
||||
|
||||
with cfg._module.args.pkgs;
|
||||
|
||||
runCommand "test"
|
||||
{ buildInputs = [ config.system.path ];
|
||||
requiredSystemFeatures = [ "uid-range" ];
|
||||
toplevel = config.system.build.toplevel;
|
||||
}
|
||||
''
|
||||
root=$(pwd)/root
|
||||
mkdir -p $root $root/etc
|
||||
|
||||
export > $root/.env
|
||||
|
||||
# Make /run a tmpfs to shut up a systemd warning.
|
||||
mkdir /run
|
||||
mount -t tmpfs none /run
|
||||
|
||||
mount -t cgroup2 none /sys/fs/cgroup
|
||||
|
||||
mkdir -p $out
|
||||
|
||||
chmod +w /etc
|
||||
touch /etc/os-release
|
||||
echo a5ea3f98dedc0278b6f3cc8c37eeaeac > /etc/machine-id
|
||||
|
||||
SYSTEMD_NSPAWN_UNIFIED_HIERARCHY=1 \
|
||||
${config.systemd.package}/bin/systemd-nspawn \
|
||||
--keep-unit \
|
||||
-M ${config.networking.hostName} -D "$root" \
|
||||
--register=no \
|
||||
--resolv-conf=off \
|
||||
--bind-ro=/nix/store \
|
||||
--bind=$out \
|
||||
--bind=/proc:/run/host/proc \
|
||||
--bind=/sys:/run/host/sys \
|
||||
--private-network \
|
||||
$toplevel/init
|
||||
''
|
||||
@@ -150,6 +150,10 @@ in
|
||||
|
||||
tarballFlakes = runNixOSTestFor "x86_64-linux" ./tarball-flakes.nix;
|
||||
|
||||
containers = runNixOSTestFor "x86_64-linux" ./containers/containers.nix;
|
||||
|
||||
cgroups = runNixOSTestFor "x86_64-linux" ./cgroups;
|
||||
|
||||
setuid = lib.genAttrs
|
||||
["i686-linux" "x86_64-linux"]
|
||||
(system: runNixOSTestFor system ./setuid/setuid.nix);
|
||||
|
||||
Reference in New Issue
Block a user