Commit Graph
544 Commits
Author SHA1 Message Date
Eelco Dolstra d4dcffd643 Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Will Dietz 9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Will Dietz e4d43f1f94 logging.cc: add missing 'override' 2017-12-11 12:08:40 -06:00
Eelco Dolstra 6cdaa858d0 Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra d0b88db441 Cleanup 2017-11-20 17:32:58 +01:00
Eelco DolstraandGitHub 72804dc0bd Merge pull request #1645 from twhitehead/stat-race
Fix (highly unlikely) race condition in readLink
2017-11-20 17:32:27 +01:00
Eelco Dolstra e026bc3b05 fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra 72cd52c3cd builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Tyson Whitehead 07d2c6d213 Fix (highly unlikely) race condition in readLink
Used to determine symlink size with stat and value with readlink.
This could technically result in garbage if symlink changed between
calls.  Also gets around the broken stat implementation in our
network filesystem (returns size + 1 giving a byte of garbage).
2017-10-30 11:49:55 -04:00
Eelco Dolstra 2d5b1b24bf Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra 96051dd057 More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra fe9d2f974d Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra 0d59f1ca49 nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Eelco Dolstra 37fbfffd8e Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Eelco Dolstra fda7b95cb0 Fix a hang in ThreadPool
The worker threads could exit prematurely if they finished processing
all items while the main thread was still adding items. In particular,
this caused hanging nix-store --serve processes in the build farm.

Also, process items from the main thread.
2017-10-09 15:41:09 +02:00
Eelco Dolstra e350671737 Doh 2017-09-25 13:25:55 +02:00
Eelco Dolstra 89dc62c174 RemoteStore: Add option to drop old connections from the pool
This is a hack to make hydra-queue-runner free its temproots
periodically, thereby ensuring that garbage collection of the
corresponding paths is not blocked until the queue runner is
restarted.

It would be better if temproots could be released earlier than at
process exit. I started working on a RAII object returned by functions
like addToStore() that releases temproots. However, this would be a
pretty massive change so I gave up on it for now.
2017-09-14 18:10:38 +02:00
Eelco Dolstra 970366266b Add "nix add-to-store" command 2017-09-14 13:29:33 +02:00
Eelco Dolstra b7376edf06 ThreadPool: On exception, interrupt the other worker threads 2017-09-08 15:31:24 +02:00
Eelco Dolstra 8f6b347abd ThreadPool: Improve exception handling
In particular, process() won't return as long as there are active
items. This prevents work item lambdas from referring to stack frames
that no longer exist.
2017-09-08 14:42:09 +02:00
Eelco Dolstra dff440aab3 nix build: Add --out-link and --no-link options 2017-09-06 16:20:34 +02:00
Eelco Dolstra fd73c1e20a Add an activity for binary cache queries 2017-08-31 15:25:58 +02:00
Eelco Dolstra 9f47eac92b Fix mismatched tag warning
https://hydra.nixos.org/build/59649086
2017-08-31 11:04:52 +02:00
Eelco Dolstra 5cc8609e30 nix run: Allow passing a command to execute
E.g.

  nix run nixpkgs.hello -c hello --greeting Hallo

Note that unlike "nix-shell --command", no quoting of arguments is
necessary.

"-c" (short for "--command") cannot be combined with "--" because they
both consume all remaining arguments. But since installables shouldn't
start with a dash, this is unlikely to cause problems.
2017-08-29 14:42:48 +02:00
Eelco Dolstra 05c45f301d readLink(): Handle symlinks in /proc
Symlinks like /proc/self/exe report a stat() size of 0, so use a
buffer of at least PATH_MAX instead.
2017-08-29 11:52:34 +02:00
Eelco Dolstra 2cc345b95f Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra fe34b91289 Tunnel progress messages from the daemon to the client
This makes the progress bar work for non-root users.
2017-08-28 18:49:42 +02:00
Eelco Dolstra e681b1f064 Simplify 2017-08-28 14:30:35 +02:00
Eelco Dolstra 94a0548dc4 Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra 0ac35b67b8 Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra c137c0a5eb Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra db1d45037c Handle SIGWINCH 2017-08-25 15:59:03 +02:00
Eelco Dolstra a3015db6c3 Typo 2017-08-25 15:59:03 +02:00
Eelco Dolstra 0e9ddcc306 Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra 9400cb36b7 Disallow accidental copy construction 2017-08-21 12:18:42 +02:00
Andy Wingo 6bb4e3e8fe Remove unused decodeOctalEscaped
Besides being unused, this function has a bug that it will incorrectly
decode the path component Ubuntu\04016.04.2\040LTS\040amd64 as
"Ubuntu.04.2 LTS amd64" instead of "Ubuntu 16.04.2 LTS amd64".
2017-08-18 11:07:00 +02:00
Eelco Dolstra c2cab20732 nix verify: Restore the progress indicator 2017-08-16 20:56:03 +02:00
Eelco Dolstra b4ed97e3a3 nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra 23b8b7e096 nix optimise-store: Add
This replaces "nix-store --optimise". Main difference is that it has a
progress indicator.
2017-08-16 20:56:03 +02:00
Eelco Dolstra 40bffe0a43 Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra dff12b38f9 Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra bf1f123b09 Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra 0e0dcf2c7e Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra c36467ad2e Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra b29b6feaba nix copy: Improve progress indicator
It now shows the amount of data copied:

  [8/1038 copied, 160.4/1590.9 MiB copied] copying path '...'
2017-08-16 20:56:02 +02:00
Eelco Dolstra c5e4404580 nix copy: Revive progress bar 2017-08-16 20:56:02 +02:00
Eelco Dolstra af765a8eab Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Brian McKenna b39cc4fc81 Include missing <cstdlib> for abort()
This is needed to get Nix compiled using Android NDK.
2017-08-03 07:03:22 +10:00
Eelco Dolstra a2778988f2 Merge branch 'macOS' of https://github.com/davidak/nix 2017-07-31 10:31:51 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00