Commit Graph
6357 Commits
Author SHA1 Message Date
Eelco Dolstra fa4def3d46 Require libbrotli 2018-08-06 14:06:54 +02:00
Eelco Dolstra 4361a4331f Fix reporting of HTTP body size when a result callback is used 2018-08-06 11:31:14 +02:00
Eelco Dolstra 2825e05d21 Make adding paths via nix-store --serve run in constant memory
It adds a new operation, cmdAddToStoreNar, that does the same thing as
the corresponding nix-daemon operation, i.e. call addToStore(). This
replaces cmdImportPaths, which has the major issue that it sends the
NAR first and the store path second, thus requiring us to store the
incoming NAR either in memory or on disk until we decide what to do
with it.

For example, this reduces the memory usage of

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79

from 267 MiB to 12 MiB.

Probably fixes #1988.
2018-08-03 21:21:16 +02:00
Eelco Dolstra 34c17fdae5 Add a test for LegacySSHStore that doesn't require a VM 2018-08-03 21:20:38 +02:00
Eelco Dolstra eeebe4cdc5 cmdDumpStorePath: Support chroot stores 2018-08-03 21:20:38 +02:00
Eelco Dolstra 848a9375c3 Support escaping in store URIs 2018-08-03 21:20:38 +02:00
Eelco Dolstra 4e7d5f660c SSHMaster: Bypass SSH when connecting to localhost
This is primarily useful for testing since it removes the need to have
SSH working.
2018-08-03 21:20:38 +02:00
Eelco Dolstra e268bbc054 LegacySSHStore: Add remote-store option
This is primarily useful for testing, e.g.

  $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 21:20:38 +02:00
Eelco DolstraandGitHub 25da1c64f9 Merge pull request #2327 from NixOS/grahamc-patch-1
2.1 release notes: Add note about s3-compatible stores
2018-08-03 17:33:34 +02:00
Graham ChristensenandGitHub 06080e4abc 2.1 release notes: Add note about s3-compatible stores 2018-08-03 11:09:31 -04:00
Eelco Dolstra 87356cc8a4 Add Nix 2.1 release notes 2018-08-03 17:05:17 +02:00
Eelco DolstraandGitHub bc65e02d96 Merge pull request #2326 from aszlig/fix-symlink-leak
Fix symlink leak in restricted eval mode
2018-08-03 17:01:34 +02:00
Eelco DolstraandGitHub 122e1a61f8 Merge pull request #2323 from samueldr/feature/selective-impurity
Allows selectively adding environment variables to pure shells.
2018-08-03 16:58:18 +02:00
aszlig 43e28a1b75 Fix symlink leak in restricted eval mode
In EvalState::checkSourcePath, the path is checked against the list of
allowed paths first and later it's checked again *after* resolving
symlinks.

The resolving of the symlinks is done via canonPath, which also strips
out "../" and "./". However after the canonicalisation the error message
pointing out that the path is not allowed prints the symlink target in
the error message.

Even if we'd suppress the message, symlink targets could still be leaked
if the symlink target doesn't exist (in this case the error is thrown in
canonPath).

So instead, we now do canonPath() without symlink resolving first before
even checking against the list of allowed paths and then later do the
symlink resolving and checking the allowed paths again.

The first call to canonPath() should get rid of all the "../" and "./",
so in theory the only way to leak a symlink if the attacker is able to
put a symlink in one of the paths allowed by restricted evaluation mode.

For the latter I don't think this is part of the threat model, because
if the attacker can write to that path, the attack vector is even
larger.

Signed-off-by: aszlig <aszlig@nix.build>
2018-08-03 06:46:43 +02:00
Samuel Dionne-Riel 438e02529d Allows selectively adding environment variables to pure shells.
Includes documentation and test.
2018-08-02 08:21:30 -04:00
Graham Christensen 49a53c1d3f s3 binary cache: support specifying an endpoint
Works for uploading and not downloading.
2018-08-02 08:19:50 -04:00
Eelco DolstraandGitHub 3193f5ff3e Merge pull request #2322 from samueldr/fix/removes-unused-variable
Removes unused variable from `nix-build/nix-shell`
2018-08-02 11:17:51 +02:00
Samuel Dionne-Riel fb6907e3d6 Removes unused variable from nix-build/nix-shell
This particular `shell` variable wasn't used, since a new one was
declared in the only side of the `if` branch that used a `shell`
variable.

It could realistically confuse developers thinking it could use `$SHELL`
under some situations.
2018-08-01 18:30:03 -04:00
Graham Christensen b27431b7cb Document s3 substitutions 2018-08-01 11:07:41 -04:00
Eelco Dolstra 45bcf5416a Merge branch 'prim_mapAttr-fix' of https://github.com/volth/nix 2018-07-31 20:05:07 +02:00
Eelco Dolstra 94b2e4e1be Disable rpm_fedora27x86_64
Fedora 27 provides an incompatible version of Boost (1.64.0).
2018-07-31 17:34:53 +02:00
Eelco Dolstra 4b4adbc93d Fix race downloading into a sink
This fixes 'error 10 while decompressing xz file'.

https://hydra.nixos.org/build/78308551
2018-07-31 17:26:04 +02:00
Eelco Dolstra d9e8ab0ff2 Add a check for broken Boost versions
In some Boost versions, coroutines don't propagate exceptions
properly, causing Nix to fail with the exception 'coroutine has
finished'.
2018-07-31 16:35:05 +02:00
Eelco Dolstra 607e75083a Attempt to fix the RPM build 2018-07-31 14:03:19 +02:00
Eelco Dolstra bd19959d90 Fix RPM build
https://hydra.nixos.org/build/73991153
2018-07-31 11:53:53 +02:00
Eelco DolstraandGitHub f602ff264b Merge pull request #2305 from lheckemann/copy-keep-going
copyPathsToStore: honour keep-going
2018-07-24 17:23:13 +02:00
Linus Heckemann 9ac1a79882 copyPathsToStore: honour keep-going 2018-07-24 17:05:06 +02:00
Eelco DolstraandGitHub a7fb7d3cde Merge pull request #2303 from volth/patch-4
parser.y: fix assoc of -> and < > <= >=
2018-07-23 11:38:15 +02:00
volthandGitHub deaa6e9a34 parser.y: right-associativity of -> 2018-07-23 07:28:48 +00:00
Eelco DolstraandGitHub 925053e974 Merge pull request #2302 from volth/patch-3
prim_foldlStrict: call forceValue() before value is copied
2018-07-23 09:19:05 +02:00
volthandGitHub 85fe4a819c parser.y: fix assoc of -> and < > <= >=
The parser allowed senseless `a > b > c` but disallowed `a -> b -> c` which seems valid
It might be a typo
2018-07-21 15:24:51 +00:00
volthandGitHub e2b114cfe1 prim_foldlStrict: call forceValue() before value is copied
forceValue() were called after a value is copied effectively forcing only one of the copies keeping another copy not evaluated.
This resulted in its evaluation of the same lazy value more than once (the number of hits is not big though)
2018-07-21 06:44:42 +00:00
Eelco Dolstra 1b34b69b45 nix-prefetch-url: Download file in constant memory
Before:

  $ command time nix-prefetch-url https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.6.tar.xz
  1.19user 1.02system 0:41.96elapsed 5%CPU (0avgtext+0avgdata 182720maxresident)k

After:

  1.38user 1.05system 0:39.73elapsed 6%CPU (0avgtext+0avgdata 16204maxresident)k

Note however that addToStore() can still take a lot of memory
(e.g. RemoteStore::addToStore() is constant space, but
LocalStore::addToStore() isn't; that's fixed by
https://github.com/edolstra/nix/commit/c94b4fc7ee0c7b322a5f3c7ee784063b47a11d98
though).

Fixes #1400.
2018-07-12 18:48:41 +02:00
Eelco Dolstra c2de2ff385 Fix build on 32-bit systems and macOS
Apparently, on macOS, 'long' != 'int64_t'.

https://hydra.nixos.org/build/77100756
2018-07-11 21:12:18 +02:00
Eelco Dolstra efd04888ca Shup up a warning 2018-07-11 21:05:09 +02:00
Eelco Dolstra 875cd9da2b Remove unused function printStats2()
Closes #2282.
2018-07-11 20:29:18 +02:00
volthandGitHub c03d73c1cf prim_map, prim_genList: no need to force function argument which will be stored in mkApp 2018-07-07 23:30:17 +00:00
volth 627e28ba33 prim_mapAttrs: `f' must be evaluated lazily to avoid infinite recursion 2018-07-06 21:52:54 +00:00
Eelco DolstraandGitHub aa64e95bc8 Merge pull request #2250 from dtzWill/feature/repl-progress-bar-builds
repl: use `nix build` for building instead of `nix-store -r`
2018-07-06 21:36:37 +02:00
Eelco DolstraandGitHub f1f4c257f4 Merge pull request #2275 from volth/issue-1776
[bugfix] lib.concatMap and lib.mapAttrs to be builtins
2018-07-05 17:42:06 +02:00
volth 1515c65616 prim_concatMap: no need to force value 2018-07-05 15:33:33 +00:00
volth e6bf1a79d7 prim_mapAttrs: must be lazy to avoid infinite recursion 2018-07-05 15:33:12 +00:00
Eelco DolstraandGitHub ddc9b87df1 Merge pull request #2273 from volth/issue-1776
[wip] lib.concatMap and lib.mapAttrs to be builtins
2018-07-05 16:50:20 +02:00
volthandGitHub 841747b0e6 prim_concatMap: allocate intermediate list on stack 2018-07-05 12:37:37 +00:00
volthandGitHub ee218f99ca primops.cc: fix comment 2018-07-05 11:58:15 +00:00
volth 403a76a18f lib.concatMap and lib.mapAttrs to be builtins 2018-07-05 02:54:09 +00:00
Eelco DolstraandGitHub 3defdccf4b Merge pull request #2268 from dtzWill/fix/search-ux-test-break
tests/search.sh: minor fix to unbreak tests after search ux merge
2018-07-04 10:55:43 +02:00
Will Dietz cb9ef85852 tests/search.sh: minor fix to unbreak tests after search ux merge 2018-07-03 15:37:06 -05:00
Eelco Dolstra e388739098 nix-daemon: Respect --store
For example, this allows you to do run nix-daemon as a non-privileged
user:

  eelco$ NIX_STATE_DIR=~/my-nix/nix/var nix-daemon --store ~/my-nix/

The NIX_STATE_DIR is still needed because settings.nixDaemonSocketFile
is not derived from settings.storeUri (and we can't derive it from the
store's state directory because we don't want to open the store in the
parent process).
2018-07-03 19:52:32 +02:00
Eelco Dolstra ea3c9dab5f Include cpptoml for build simplicity 2018-07-03 18:39:36 +02:00