Update release notes for Nix 1.5.3

Before calling dumpPath(), we have to make sure the files are owned by
the build user.  Otherwise, the build could contain a hard link to
(say) /etc/shadow, which would then be read by the daemon and
rewritten as a world-readable file.

This only affects systems that don't have hard link restrictions
enabled.

doc/manual/conf-file.xml

@@ -325,7 +325,7 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
 
     <listitem><para>A list of URLs of binary caches, separated by
     whitespace.  The default is
-    <literal>http://nixos.org/binary-cache</literal>.</para></listitem>
+    <literal>http://cache.nixos.org</literal>.</para></listitem>
 
   </varlistentry>
 

doc/manual/installation.xml

@@ -538,7 +538,7 @@ a symbolic link to the current <emphasis>user environment</emphasis>
 installed packages).  The simplest way to set the required environment
 variables is to include the file
 <filename><replaceable>prefix</replaceable>/etc/profile.d/nix.sh</filename>
-in your <filename>~/.bashrc</filename> (or similar), like this:</para>
+in your <filename>~/.profile</filename> (or similar), like this:</para>
 
 <screen>
 source <replaceable>prefix</replaceable>/etc/profile.d/nix.sh</screen>

doc/manual/nix-push.xml

@@ -252,7 +252,7 @@ The properties that are currently supported are:
     <listitem><para>Each binary cache has a priority (defaulting to
     50).  Binary caches are checked for binaries in order of ascending
     priority; thus a higher number denotes a lower priority.  The
-    binary cache <uri>http://nixos.org/binary-cache</uri> has priority
+    binary cache <uri>http://cache.nixos.org</uri> has priority
     40.</para></listitem>
 
   </varlistentry>
@@ -270,14 +270,14 @@ URL <replaceable>url</replaceable> has a binary for
 <replaceable>p</replaceable>, Nix fetches
 <replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
 is the hash part of <replaceable>p</replaceable>.  Thus, if we have a
-cache <uri>http://nixos.org/binary-cache</uri> and we want to obtain
+cache <uri>http://cache.nixos.org</uri> and we want to obtain
 the store path
 <screen>
 /nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
 </screen>
 then Nix will attempt to fetch
 <screen>
-http://nixos.org/binary-cache/a8922c0h87iilxzzvwn2hmv8x210aqb9.narinfo
+http://cache.nixos.org/a8922c0h87iilxzzvwn2hmv8x210aqb9.narinfo
 </screen>
 (Commands such as <command>nix-env -qas</command> will issue an HTTP
 HEAD request, since it only needs to know if the
@@ -381,7 +381,7 @@ The fields are as follows:
 references exist (e.g.,
 <filename>/nix/store/2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16</filename>),
 Nix will fetch <screen>
-http://nixos.org/binary-cache/nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
+http://cache.nixos.org/nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
 </screen> and decompress and unpack it to
 <filename>/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7</filename>.</para>
 

doc/manual/nix-store.xml

@@ -852,7 +852,7 @@ in Nix itself.</para>
 
 <refsection><title>Description</title>
             
-<para>The operation <option>--verify-paths</option> compares the
+<para>The operation <option>--verify-path</option> compares the
 contents of the given store paths to their cryptographic hashes stored
 in Nix’s database.  For every changed path, it prints a warning
 message.  The exit status is 0 if no path has changed, and 1

doc/manual/quick-start.xml

@@ -36,7 +36,7 @@ changed using
 
 <listitem><para>You should add
 <filename><replaceable>prefix</replaceable>/etc/profile.d/nix.sh</filename>
-to your <filename>~/.bashrc</filename> (or some other login
+to your <filename>~/.profile</filename> (or some other login
 file).</para></listitem>
 
 <listitem><para>Subscribe to the Nix Packages channel.

doc/manual/release-notes.xml

@@ -6,6 +6,54 @@
 
 
 
+<!--==================================================================-->
+
+<section xml:id="ssec-relnotes-1.5.3"><title>Release 1.5.3 (June 17, 2013)</title>
+
+<para>This is primarily a bug fix release.  The following changes are
+noteworthy:</para>
+
+<itemizedlist>
+
+  <listitem>
+    <para>Yet another security bug involving hard links to files
+    outside the store was fixed.  This bug only affected multi-user
+    installations that do not have hard link restrictions
+    enabled.  (NixOS is thus not vulnerable.)</para>
+  </listitem>
+
+  <listitem>
+    <para>The default binary cache URL has changed from
+    <uri>http://nixos.org/binary-cache</uri> to
+    <uri>http://cache.nixos.org</uri>.  The latter is hosted on Amazon
+    CloudFront (courtesy of <link
+    xlink:href="http://www.logicblox.com/">LogicBlox</link>) and
+    should provide better performance for users in both Europe and
+    North America.</para>
+  </listitem>
+
+  <listitem>
+    <para>The binary cache substituter now prints a warning message if
+    fetching information from the cache takes more than five seconds.
+    Thus network or server problems no longer make Nix appear to just
+    hang.</para>
+  </listitem>
+
+  <listitem>
+    <para>Stack traces now show function names, e.g.
+<screen>
+while evaluating `concatMapStrings' at `<replaceable>...</replaceable>/nixpkgs/pkgs/lib/strings.nix:18:25':
+</screen>
+      Also, if a function is called with an unexpected argument, Nix
+      now shows the name of the argument.
+    </para>
+  </listitem>
+
+</itemizedlist>
+
+</section>
+
+
 <!--==================================================================-->
 
 <section xml:id="ssec-relnotes-1.5.2"><title>Release 1.5.2 (May 13, 2013)</title>
@@ -107,10 +155,10 @@ Pernsteiner.</para>
     configuration setting <option>binary-caches</option> contains a
     list of URLs of binary caches.  For instance, doing
 <screen>
-$ nix-env -i thunderbird --option binary-caches http://nixos.org/binary-cache
+$ nix-env -i thunderbird --option binary-caches http://cache.nixos.org
 </screen>
     will install Thunderbird and its dependencies, using the available
-    pre-built binaries in <uri>http://nixos.org/binary-cache</uri>.
+    pre-built binaries in <uri>http://cache.nixos.org</uri>.
     The main advantage over the old “manifest”-based method of getting
     pre-built binaries is that you don’t have to worry about your
     manifest being in sync with the Nix expressions you’re installing
@@ -123,7 +171,7 @@ $ nix-env -i thunderbird --option binary-caches http://nixos.org/binary-cache
     used automatically if you subscribe to that channel.  If you use
     the Nixpkgs or NixOS channels
     (<uri>http://nixos.org/channels</uri>) you automatically get the
-    cache <uri>http://nixos.org/binary-cache</uri>.</para>
+    cache <uri>http://cache.nixos.org</uri>.</para>
 
     <para>Binary caches are created using <command>nix-push</command>.
     For details on the operation and format of binary caches, see the

scripts/build-remote.pl.in

@@ -197,10 +197,11 @@ REQ: while (1) {
         $hostName = $machine->{hostName};
         if (openSSHConnection($hostName)) {
             last REQ if system("ssh $hostName @sshOpts nix-builds-inhibited < /dev/null > /dev/null 2>&1") != 0;
+            warn "machine `$hostName' is refusing builds, trying other available machines...\n";
             closeSSHConnection;
+        } else {
+            warn "unable to open SSH connection to `$hostName', trying other available machines...\n";
         }
-
-        warn "unable to open SSH connection to $hostName, trying other available machines...\n";
         $machine->{enabled} = 0;
     }
 }
@@ -288,13 +289,11 @@ if (system("exec ssh $hostName @sshOpts '(read; kill -INT -\$\$) <&0 & exec nix-
 
 
 # Copy the output from the build machine.
-foreach my $output (@outputs) {
-    my $maybeSignRemote = "";
-    $maybeSignRemote = "--sign" if $UID != 0;
-    next if isValidPath($output);
-    system("exec ssh $hostName @sshOpts 'nix-store --export $maybeSignRemote $output'" .
-           "| NIX_HELD_LOCKS=$output @bindir@/nix-store --import > /dev/null") == 0
-        or die "cannot copy $output from $hostName: $?";
+my @outputs2 = grep { !isValidPath($_) } @outputs;
+if (scalar @outputs2 > 0) {
+    system("exec ssh $hostName @sshOpts 'nix-store --export @outputs2'" .
+           "| NIX_HELD_LOCKS='@outputs2' @bindir@/nix-store --import > /dev/null") == 0
+        or die("cannot copy paths " . join(", ", @outputs) . " from `$hostName': $?");
 }
 
 

scripts/download-from-binary-cache.pl.in

@@ -24,8 +24,9 @@ my $ttlNegative = 24 * 3600; # when to purge negative lookups from the database
 my $ttlNegativeUse = 3600; # how long negative lookups are valid for non-"have" lookups
 my $didExpiration = 0;
 
-my $debug = ($ENV{"NIX_DEBUG_SUBST"} // "") eq 1;
-open(STDERR, ">>/dev/tty") if $debug;
+my $showAfter = 5; # show that we're waiting for a request after this many seconds
+
+my $debug = ($Nix::Config::config{"debug-subst"} // "") eq 1 || ($Nix::Config::config{"untrusted-debug-subst"} // "") eq 1;
 
 my $cacheFileURLs = ($ENV{"_NIX_CACHE_FILE_URLS"} // "") eq 1; # for testing
 
@@ -46,7 +47,8 @@ sub addRequest {
 
     my $curl = WWW::Curl::Easy->new;
     my $curlId = $curlIdCount++;
-    $requests{$curlId} = { storePath => $storePath, url => $url, handle => $curl, content => "", type => $head ? "HEAD" : "GET" };
+    $requests{$curlId} = { storePath => $storePath, url => $url, handle => $curl, content => "", type => $head ? "HEAD" : "GET"
+                         , shown => 0, started => time() };
 
     $curl->setopt(CURLOPT_PRIVATE, $curlId);
     $curl->setopt(CURLOPT_URL, $url);
@@ -57,6 +59,7 @@ sub addRequest {
     $curl->setopt(CURLOPT_USERAGENT, "Nix/$Nix::Config::version");
     $curl->setopt(CURLOPT_NOBODY, 1) if $head;
     $curl->setopt(CURLOPT_FAILONERROR, 1);
+    $curl->setopt(CURLOPT_TIMEOUT, int($ENV{"NIX_CONNECT_TIMEOUT"} // 0));
 
     if ($activeRequests >= $maxParallelRequests) {
         $scheduled{$curlId} = 1;
@@ -76,7 +79,7 @@ sub processRequests {
 
         # Sleep until we can read or write some data.
         if (scalar @{$rfds} + scalar @{$wfds} + scalar @{$efds} > 0) {
-            IO::Select->select(IO::Select->new(@{$rfds}), IO::Select->new(@{$wfds}), IO::Select->new(@{$efds}), 0.1);
+            IO::Select->select(IO::Select->new(@{$rfds}), IO::Select->new(@{$wfds}), IO::Select->new(@{$efds}), 1.0);
         }
 
         if ($curlm->perform() != $activeRequests) {
@@ -101,6 +104,16 @@ sub processRequests {
                 }
             }
         }
+
+        my $time = time();
+        while (my ($key, $request) = each %requests) {
+            next unless defined $request->{handle};
+            next if $request->{shown};
+            if ($time > $request->{started} + $showAfter) {
+                print STDERR "still waiting for ‘$request->{url}’ after $showAfter seconds...\n";
+                $request->{shown} = 1;
+            }
+        }
     }
 }
 
@@ -196,7 +209,7 @@ sub getAvailableCaches {
     }
 
     my @urls = strToList($Nix::Config::config{"binary-caches"} //
-        ($Nix::Config::storeDir eq "/nix/store" ? "http://nixos.org/binary-cache" : ""));
+        ($Nix::Config::storeDir eq "/nix/store" ? "http://cache.nixos.org" : ""));
 
     my $urlsFiles = $Nix::Config::config{"binary-cache-files"}
         // "$Nix::Config::stateDir/profiles/per-user/$userName/channels/binary-caches/*";
@@ -225,7 +238,7 @@ sub getAvailableCaches {
         }
     }
 
-    my @untrustedUrls = strToList $Nix::Config::config{"untrusted-extra-binary-caches"};
+    my @untrustedUrls = strToList $Nix::Config::config{"untrusted-extra-binary-caches"} // "";
     foreach my $url (@untrustedUrls) {
         next unless scalar(grep { $url eq $_ } @trustedUrls) > 0;
         push @urls, $url;
@@ -265,11 +278,12 @@ sub getAvailableCaches {
             elsif ($1 eq "Priority") { $priority = int($2); }
         }
 
-        $dbh->do("insert into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?, ?, ?, ?, ?)",
+        $dbh->do("insert or replace into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?, ?, ?, ?, ?)",
                  {}, $url, time(), $storeDir, $wantMassQuery, $priority);
-        my $id = $dbh->last_insert_id("", "", "", "");
+        $queryCache->execute($url);
+        $res = $queryCache->fetchrow_hashref() or die;
         next if $storeDir ne $Nix::Config::storeDir;
-        push @caches, { id => $id, url => $url, wantMassQuery => $wantMassQuery, priority => $priority };
+        push @caches, { id => $res->{id}, url => $url, wantMassQuery => $wantMassQuery, priority => $priority };
     }
 
     @caches = sort { $a->{priority} <=> $b->{priority} } @caches;
@@ -288,7 +302,7 @@ sub processNARInfo {
     my ($storePath, $cache, $request) = @_;
 
     if ($request->{result} != 0) {
-        if ($request->{result} != 37 && $request->{httpStatus} != 404) {
+        if ($request->{result} != 37 && $request->{httpStatus} != 404 && $request->{httpStatus} != 403) {
             print STDERR "could not download ‘$request->{url}’ (" .
                 ($request->{result} != 0 ? "Curl error $request->{result}" : "HTTP status $request->{httpStatus}") . ")\n";
         } else {
@@ -467,7 +481,7 @@ sub printSubstitutablePaths {
 
         foreach my $request (values %requests) {
             if ($request->{result} != 0) {
-                if ($request->{result} != 37 && $request->{httpStatus} != 404) {
+                if ($request->{result} != 37 && $request->{httpStatus} != 404 && $request->{httpStatus} != 403) {
                     print STDERR "could not check ‘$request->{url}’ (" .
                         ($request->{result} != 0 ? "Curl error $request->{result}" : "HTTP status $request->{httpStatus}") . ")\n";
                 } else {

src/libexpr/eval.cc

@@ -141,10 +141,10 @@ EvalState::EvalState()
     , sOutputs(symbols.create("outputs"))
     , sOutputName(symbols.create("outputName"))
     , sIgnoreNulls(symbols.create("__ignoreNulls"))
+    , repair(false)
     , baseEnv(allocEnv(128))
     , baseEnvDispl(0)
     , staticBaseEnv(false, 0)
-    , repair(false)
 {
     nrEnvs = nrValuesInEnvs = nrValues = nrListElems = 0;
     nrAttrsets = nrOpUpdates = nrOpUpdateValuesCopied = 0;
@@ -247,6 +247,11 @@ LocalNoInlineNoReturn(void throwTypeError(const char * s, const Pos & pos, const
     throw TypeError(format(s) % pos % s2);
 }
 
+LocalNoInlineNoReturn(void throwTypeError(const char * s, const string & s1, const string & s2))
+{
+    throw TypeError(format(s) % s1 % s2);
+}
+
 LocalNoInlineNoReturn(void throwTypeError(const char * s, const Pos & pos))
 {
     throw TypeError(format(s) % pos);
@@ -755,8 +760,8 @@ void EvalState::callFunction(Value & fun, Value & arg, Value & v)
         foreach (Formals::Formals_::iterator, i, fun.lambda.fun->formals->formals) {
             Bindings::iterator j = arg.attrs->find(i->name);
             if (j == arg.attrs->end()) {
-                if (!i->def) throwTypeError("function at %1% called without required argument `%2%'",
-                    fun.lambda.fun->pos, i->name);
+                if (!i->def) throwTypeError("%1% called without required argument `%2%'",
+                    fun.lambda.fun->showNamePos(), i->name);
                 env2.values[displ++] = i->def->maybeThunk(*this, env2);
             } else {
                 attrsUsed++;
@@ -765,11 +770,15 @@ void EvalState::callFunction(Value & fun, Value & arg, Value & v)
         }
 
         /* Check that each actual argument is listed as a formal
-           argument (unless the attribute match specifies a `...').
-           TODO: show the names of the expected/unexpected
-           arguments. */
-        if (!fun.lambda.fun->formals->ellipsis && attrsUsed != arg.attrs->size()) 
-            throwTypeError("function at %1% called with unexpected argument", fun.lambda.fun->pos);
+           argument (unless the attribute match specifies a `...'). */
+        if (!fun.lambda.fun->formals->ellipsis && attrsUsed != arg.attrs->size()) {
+            /* Nope, so show the first unexpected argument to the
+               user. */
+            foreach (Bindings::iterator, i, *arg.attrs)
+                if (fun.lambda.fun->formals->argNames.find(i->name) == fun.lambda.fun->formals->argNames.end())
+                    throwTypeError("%1% called with unexpected argument `%2%'", fun.lambda.fun->showNamePos(), i->name);
+            abort(); // can't happen
+        }
     }
 
     nrFunctionCalls++;
@@ -778,7 +787,7 @@ void EvalState::callFunction(Value & fun, Value & arg, Value & v)
     try {
         fun.lambda.fun->body->eval(*this, env2, v);
     } catch (Error & e) {
-        addErrorPrefix(e, "while evaluating the function at %1%:\n", fun.lambda.fun->pos);
+        addErrorPrefix(e, "while evaluating %1%:\n", fun.lambda.fun->showNamePos());
         throw;
     }
 }

src/libexpr/nixexpr.cc

@@ -324,4 +324,24 @@ void ExprConcatStrings::bindVars(const StaticEnv & env)
 }
 
 
+/* Storing function names. */
+
+void Expr::setName(Symbol & name)
+{
+}
+
+
+void ExprLambda::setName(Symbol & name)
+{
+    this->name = name;
+    body->setName(name);
+}
+
+
+string ExprLambda::showNamePos()
+{
+    return (format("%1% at %2%") % (name.set() ? "`" + (string) name + "'" : "an anonymous function") % pos).str();
+}
+
+
 }

src/libexpr/nixexpr.hh

@@ -63,6 +63,7 @@ struct Expr
     virtual void bindVars(const StaticEnv & env);
     virtual void eval(EvalState & state, Env & env, Value & v);
     virtual Value * maybeThunk(EvalState & state, Env & env);
+    virtual void setName(Symbol & name);
 };
 
 std::ostream & operator << (std::ostream & str, Expr & e);
@@ -197,6 +198,7 @@ struct Formals
 struct ExprLambda : Expr
 {
     Pos pos;
+    Symbol name;
     Symbol arg;
     bool matchAttrs;
     Formals * formals;
@@ -208,6 +210,8 @@ struct ExprLambda : Expr
             throw ParseError(format("duplicate formal function argument `%1%' at %2%")
                 % arg % pos);
     };
+    void setName(Symbol & name);
+    string showNamePos();
     COMMON_METHODS
 };
 

src/libexpr/parser.y

@@ -104,6 +104,7 @@ static void addAttr(ExprAttrs * attrs, AttrPath & attrPath,
             }
         }
     }
+    e->setName(attrPath.back());
 }
 
 

src/libexpr/symbol-table.hh

@@ -28,17 +28,17 @@ private:
 
 public:
     Symbol() : s(0) { };
-    
+
     bool operator == (const Symbol & s2) const
     {
         return s == s2.s;
     }
-    
+
     bool operator != (const Symbol & s2) const
     {
         return s != s2.s;
     }
-    
+
     bool operator < (const Symbol & s2) const
     {
         return s < s2.s;
@@ -49,6 +49,11 @@ public:
         return *s;
     }
 
+    bool set() const
+    {
+        return s;
+    }
+
     bool empty() const
     {
         return s->empty();
@@ -66,7 +71,7 @@ inline std::ostream & operator << (std::ostream & str, const Symbol & sym)
 class SymbolTable
 {
 private:
-#if HAVE_TR1_UNORDERED_SET 
+#if HAVE_TR1_UNORDERED_SET
     typedef std::tr1::unordered_set<string> Symbols;
 #else
     typedef std::set<string> Symbols;

src/libstore/build.cc

@@ -786,6 +786,12 @@ private:
     /* Outputs that are already valid. */
     PathSet validPaths;
 
+    /* Outputs that are corrupt or not valid. */
+    PathSet missingPaths;
+
+    /* Paths that have been subject to hash rewriting. */
+    PathSet rewrittenPaths;
+
     /* User selected for running the builder. */
     UserLock buildUser;
 
@@ -838,6 +844,11 @@ private:
     bool repair;
     map<Path, Path> redirectedBadOutputs;
 
+    /* Set of inodes seen during calls to canonicalisePathMetaData()
+       for this build's outputs.  This needs to be shared between
+       outputs to allow hard links between outputs. */
+    InodesSeen inodesSeen;
+
     /* Magic exit code denoting that setting up the child environment
        failed.  (It's possible that the child actually returns the
        exit code, but ah well.) */
@@ -1305,6 +1316,9 @@ void DerivationGoal::tryToBuild()
         return;
     }
 
+    missingPaths = outputPaths(drv.outputs);
+    foreach (PathSet::iterator, i, validPaths) missingPaths.erase(*i);
+
     /* If any of the outputs already exist but are not valid, delete
        them. */
     foreach (DerivationOutputs::iterator, i, drv.outputs) {
@@ -1442,8 +1456,8 @@ void DerivationGoal::buildDone()
         /* Some cleanup per path.  We do this here and not in
            computeClosure() for convenience when the build has
            failed. */
-        foreach (DerivationOutputs::iterator, i, drv.outputs) {
-            Path path = i->second.path;
+        foreach (PathSet::iterator, i, missingPaths) {
+            Path path = *i;
 
             /* If the output was already valid, just skip (discard) it. */
             if (validPaths.find(path) != validPaths.end()) continue;
@@ -1480,6 +1494,12 @@ void DerivationGoal::buildDone()
             /* Apply hash rewriting if necessary. */
             if (!rewritesFromTmp.empty()) {
                 printMsg(lvlError, format("warning: rewriting hashes in `%1%'; cross fingers") % path);
+
+                /* Canonicalise first.  This ensures that the path
+                   we're rewriting doesn't contain a hard link to
+                   /etc/shadow or something like that. */
+                canonicalisePathMetaData(path, buildUser.enabled() ? buildUser.getUID() : -1, inodesSeen);
+
                 /* FIXME: this is in-memory. */
                 StringSink sink;
                 dumpPath(path, sink);
@@ -1487,6 +1507,8 @@ void DerivationGoal::buildDone()
                 sink.s = rewriteHashes(sink.s, rewritesFromTmp);
                 StringSource source(sink.s);
                 restorePath(path, source);
+
+                rewrittenPaths.insert(path);
             }
 
             /* Gain ownership of the build result using the setuid
@@ -1671,9 +1693,7 @@ int childEntry(void * arg)
 
 void DerivationGoal::startBuilder()
 {
-    PathSet missing = outputPaths(drv.outputs);
-    foreach (PathSet::iterator, i, validPaths) missing.erase(*i);
-    startNest(nest, lvlInfo, format("building path(s) %1%") % showPaths(missing));
+    startNest(nest, lvlInfo, format(repair ? "repairing path(s) %1%" : "building path(s) %1%") % showPaths(missingPaths));
 
     /* Right platform? */
     if (!canBuildLocally(drv.platform))
@@ -1964,7 +1984,7 @@ void DerivationGoal::startBuilder()
         /* If we're repairing, then we don't want to delete the
            corrupt outputs in advance.  So rewrite them as well. */
         if (repair)
-            foreach (PathSet::iterator, i, missing)
+            foreach (PathSet::iterator, i, missingPaths)
                 if (worker.store.isValidPath(*i) && pathExists(*i))
                     redirectedBadOutputs[*i] = addHashRewrite(*i);
     }
@@ -2249,6 +2269,8 @@ void DerivationGoal::computeClosure()
        output paths read-only. */
     foreach (DerivationOutputs::iterator, i, drv.outputs) {
         Path path = i->second.path;
+        if (missingPaths.find(path) == missingPaths.end()) continue;
+
         if (!pathExists(path)) {
             throw BuildError(
                 format("builder for `%1%' failed to produce output path `%2%'")
@@ -2287,8 +2309,10 @@ void DerivationGoal::computeClosure()
                     % path % i->second.hashAlgo % printHash16or32(h) % printHash16or32(h2));
         }
 
-        /* Get rid of all weird permissions. */
-        canonicalisePathMetaData(path, buildUser.enabled() ? buildUser.getUID() : -1);
+        /* Get rid of all weird permissions.  This also checks that
+           all files are owned by the build user, if applicable. */
+        canonicalisePathMetaData(path,
+            buildUser.enabled() && rewrittenPaths.find(path) == rewrittenPaths.end() ? buildUser.getUID() : -1, inodesSeen);
 
         /* For this output path, find the references to other paths
            contained in it.  Compute the SHA-256 NAR hash at the same
@@ -2330,12 +2354,12 @@ void DerivationGoal::computeClosure()
        paths referenced by each of them.  If there are cycles in the
        outputs, this will fail. */
     ValidPathInfos infos;
-    foreach (DerivationOutputs::iterator, i, drv.outputs) {
+    foreach (PathSet::iterator, i, missingPaths) {
         ValidPathInfo info;
-        info.path = i->second.path;
-        info.hash = contentHashes[i->second.path].first;
-        info.narSize = contentHashes[i->second.path].second;
-        info.references = allReferences[i->second.path];
+        info.path = *i;
+        info.hash = contentHashes[*i].first;
+        info.narSize = contentHashes[*i].second;
+        info.references = allReferences[*i];
         info.deriver = drvPath;
         infos.push_back(info);
     }

src/libstore/local-store.cc

@@ -500,10 +500,6 @@ void canonicaliseTimestampAndPermissions(const Path & path)
 }
 
 
-typedef std::pair<dev_t, ino_t> Inode;
-typedef set<Inode> InodesSeen;
-
-
 static void canonicalisePathMetaData_(const Path & path, uid_t fromUid, InodesSeen & inodesSeen)
 {
     checkInterrupt();
@@ -561,10 +557,8 @@ static void canonicalisePathMetaData_(const Path & path, uid_t fromUid, InodesSe
 }
 
 
-void canonicalisePathMetaData(const Path & path, uid_t fromUid)
+void canonicalisePathMetaData(const Path & path, uid_t fromUid, InodesSeen & inodesSeen)
 {
-    InodesSeen inodesSeen;
-
     canonicalisePathMetaData_(path, fromUid, inodesSeen);
 
     /* On platforms that don't have lchown(), the top-level path can't
@@ -580,6 +574,13 @@ void canonicalisePathMetaData(const Path & path, uid_t fromUid)
 }
 
 
+void canonicalisePathMetaData(const Path & path, uid_t fromUid)
+{
+    InodesSeen inodesSeen;
+    canonicalisePathMetaData(path, fromUid, inodesSeen);
+}
+
+
 void LocalStore::checkDerivationOutputs(const Path & drvPath, const Derivation & drv)
 {
     string drvName = storePathToName(drvPath);
@@ -1007,10 +1008,6 @@ void LocalStore::startSubstituter(const Path & substituter, RunningSubstituter &
     fromPipe.create();
     errorPipe.create();
 
-    /* Hack: prevent substituters that write too much to stderr from
-       deadlocking our read() from stdout. */
-    fcntl(errorPipe.writeSide, F_SETFL, O_NONBLOCK);
-
     setSubstituterEnv();
 
     run.pid = maybeVfork();
@@ -1038,15 +1035,72 @@ void LocalStore::startSubstituter(const Path & substituter, RunningSubstituter &
 
     /* Parent. */
 
+    run.program = baseNameOf(substituter);
     run.to = toPipe.writeSide.borrow();
-    run.from = fromPipe.readSide.borrow();
+    run.from = run.fromBuf.fd = fromPipe.readSide.borrow();
     run.error = errorPipe.readSide.borrow();
 }
 
 
-template<class T> T getIntLine(int fd)
+/* Read a line from the substituter's stdout, while also processing
+   its stderr. */
+string LocalStore::getLineFromSubstituter(RunningSubstituter & run)
 {
-    string s = readLine(fd);
+    string res, err;
+
+    /* We might have stdout data left over from the last time. */
+    if (run.fromBuf.hasData()) goto haveData;
+
+    while (1) {
+        fd_set fds;
+        FD_ZERO(&fds);
+        FD_SET(run.from, &fds);
+        FD_SET(run.error, &fds);
+
+        /* Wait for data to appear on the substituter's stdout or
+           stderr. */
+        if (select(run.from > run.error ? run.from + 1 : run.error + 1, &fds, 0, 0, 0) == -1) {
+            if (errno == EINTR) continue;
+            throw SysError("waiting for input from the substituter");
+        }
+
+        /* Completely drain stderr before dealing with stdout. */
+        if (FD_ISSET(run.error, &fds)) {
+            char buf[4096];
+            ssize_t n = read(run.error, (unsigned char *) buf, sizeof(buf));
+            if (n == -1) {
+                if (errno == EINTR) continue;
+                throw SysError("reading from substituter's stderr");
+            }
+            if (n == 0) throw Error(format("substituter `%1%' died unexpectedly") % run.program);
+            err.append(buf, n);
+            string::size_type p;
+            while ((p = err.find('\n')) != string::npos) {
+                printMsg(lvlError, run.program + ": " + string(err, 0, p));
+                err = string(err, p + 1);
+            }
+        }
+
+        /* Read from stdout until we get a newline or the buffer is empty. */
+        else if (run.fromBuf.hasData() || FD_ISSET(run.from, &fds)) {
+        haveData:
+            do {
+                unsigned char c;
+                run.fromBuf(&c, 1);
+                if (c == '\n') {
+                    if (!err.empty()) printMsg(lvlError, run.program + ": " + err);
+                    return res;
+                }
+                res += c;
+            } while (run.fromBuf.hasData());
+        }
+    }
+}
+
+
+template<class T> T LocalStore::getIntLineFromSubstituter(RunningSubstituter & run)
+{
+    string s = getLineFromSubstituter(run);
     T res;
     if (!string2Int(s, res)) throw Error("integer expected from stream");
     return res;
@@ -1069,13 +1123,9 @@ PathSet LocalStore::querySubstitutablePaths(const PathSet & paths)
                substituters should only write (short) messages to
                stderr when they fail.  I.e. they shouldn't write debug
                output. */
-            try {
-                Path path = readLine(run.from);
-                if (path == "") break;
-                res.insert(path);
-            } catch (EndOfFile e) {
-                throw Error(format("substituter `%1%' failed: %2%") % *i % chomp(drainFD(run.error)));
-            }
+            Path path = getLineFromSubstituter(run);
+            if (path == "") break;
+            res.insert(path);
         }
     }
     return res;
@@ -1094,26 +1144,22 @@ void LocalStore::querySubstitutablePathInfos(const Path & substituter,
     writeLine(run.to, s);
 
     while (true) {
-        try {
-            Path path = readLine(run.from);
-            if (path == "") break;
-            if (paths.find(path) == paths.end())
-                throw Error(format("got unexpected path `%1%' from substituter") % path);
-            paths.erase(path);
-            SubstitutablePathInfo & info(infos[path]);
-            info.deriver = readLine(run.from);
-            if (info.deriver != "") assertStorePath(info.deriver);
-            int nrRefs = getIntLine<int>(run.from);
-            while (nrRefs--) {
-                Path p = readLine(run.from);
-                assertStorePath(p);
-                info.references.insert(p);
-            }
-            info.downloadSize = getIntLine<long long>(run.from);
-            info.narSize = getIntLine<long long>(run.from);
-        } catch (EndOfFile e) {
-            throw Error(format("substituter `%1%' failed: %2%") % substituter % chomp(drainFD(run.error)));
+        Path path = getLineFromSubstituter(run);
+        if (path == "") break;
+        if (paths.find(path) == paths.end())
+            throw Error(format("got unexpected path `%1%' from substituter") % path);
+        paths.erase(path);
+        SubstitutablePathInfo & info(infos[path]);
+        info.deriver = getLineFromSubstituter(run);
+        if (info.deriver != "") assertStorePath(info.deriver);
+        int nrRefs = getIntLineFromSubstituter<int>(run);
+        while (nrRefs--) {
+            Path p = getLineFromSubstituter(run);
+            assertStorePath(p);
+            info.references.insert(p);
         }
+        info.downloadSize = getIntLineFromSubstituter<long long>(run);
+        info.narSize = getIntLineFromSubstituter<long long>(run);
     }
 }
 
@@ -1157,9 +1203,10 @@ void LocalStore::registerValidPaths(const ValidPathInfos & infos)
 
             foreach (ValidPathInfos::const_iterator, i, infos) {
                 assert(i->hash.type == htSHA256);
-                /* !!! Maybe the registration info should be updated if the
-                   path is already valid. */
-                if (!isValidPath(i->path)) addValidPath(*i);
+                if (isValidPath(i->path))
+                    updatePathInfo(*i);
+                else
+                    addValidPath(*i);
                 paths.insert(i->path);
             }
 

src/libstore/local-store.hh

@@ -44,8 +44,10 @@ struct OptimiseStats
 
 struct RunningSubstituter
 {
+    Path program;
     Pid pid;
     AutoCloseFD to, from, error;
+    FdSource fromBuf;
 };
 
 
@@ -289,6 +291,10 @@ private:
     void startSubstituter(const Path & substituter,
         RunningSubstituter & runningSubstituter);
 
+    string getLineFromSubstituter(RunningSubstituter & run);
+
+    template<class T> T getIntLineFromSubstituter(RunningSubstituter & run);
+
     Path createTempDirInStore();
 
     Path importPath(bool requireSignature, Source & source);
@@ -299,6 +305,10 @@ private:
 };
 
 
+typedef std::pair<dev_t, ino_t> Inode;
+typedef set<Inode> InodesSeen;
+
+
 /* "Fix", or canonicalise, the meta-data of the files in a store path
    after it has been built.  In particular:
    - the last modification date on each file is set to 1 (i.e.,
@@ -307,6 +317,7 @@ private:
      without execute permission; setuid bits etc. are cleared)
    - the owner and group are set to the Nix user and group, if we're
      in a setuid Nix installation. */
+void canonicalisePathMetaData(const Path & path, uid_t fromUid, InodesSeen & inodesSeen);
 void canonicalisePathMetaData(const Path & path, uid_t fromUid);
 
 void canonicaliseTimestampAndPermissions(const Path & path);

src/libstore/store-api.hh

@@ -328,7 +328,7 @@ Path addPermRoot(StoreAPI & store, const Path & storePath,
 
 
 /* Sort a set of paths topologically under the references relation.
-   If p refers to q, then p follows q in this list. */
+   If p refers to q, then p preceeds q in this list. */
 Paths topoSortPaths(StoreAPI & store, const PathSet & paths);
 
 

src/libutil/serialise.cc

@@ -90,6 +90,12 @@ size_t BufferedSource::read(unsigned char * data, size_t len)
 }
 
 
+bool BufferedSource::hasData()
+{
+    return bufPosOut < bufPosIn;
+}
+
+
 size_t FdSource::readUnbuffered(unsigned char * data, size_t len)
 {
     ssize_t n;

src/libutil/serialise.hh

@@ -63,6 +63,8 @@ struct BufferedSource : Source
     
     /* Underlying read call, to be overriden. */
     virtual size_t readUnbuffered(unsigned char * data, size_t len) = 0;
+
+    bool hasData();
 };
 
 

src/nix-daemon/nix-daemon.cc

@@ -273,7 +273,7 @@ struct SavingSourceAdapter : Source
 };
 
 
-static void performOp(unsigned int clientVersion,
+static void performOp(bool trusted, unsigned int clientVersion,
     Source & from, Sink & to, unsigned int op)
 {
     switch (op) {
@@ -554,7 +554,7 @@ static void performOp(unsigned int clientVersion,
                 if (name == "build-timeout")
                     string2Int(value, settings.buildTimeout);
                 else
-                    settings.set("untrusted-" + name, value);
+                    settings.set(trusted ? name : "untrusted-" + name, value);
             }
         }
         startWork();
@@ -643,7 +643,7 @@ static void performOp(unsigned int clientVersion,
 }
 
 
-static void processConnection()
+static void processConnection(bool trusted)
 {
     canSendStderr = false;
     myPid = getpid();
@@ -711,7 +711,7 @@ static void processConnection()
         opCount++;
 
         try {
-            performOp(clientVersion, from, to, op);
+            performOp(trusted, clientVersion, from, to, op);
         } catch (Error & e) {
             /* If we're not in a state were we can send replies, then
                something went wrong processing the input of the
@@ -839,6 +839,7 @@ static void daemonLoop()
             /* Get the identity of the caller, if possible. */
             uid_t clientUid = -1;
             pid_t clientPid = -1;
+            bool trusted = false;
 
 #if defined(SO_PEERCRED)
             ucred cred;
@@ -846,6 +847,7 @@ static void daemonLoop()
             if (getsockopt(remote, SOL_SOCKET, SO_PEERCRED, &cred, &credLen) != -1) {
                 clientPid = cred.pid;
                 clientUid = cred.uid;
+                if (clientUid == 0) trusted = true;
             }
 #endif
 
@@ -879,7 +881,7 @@ static void daemonLoop()
                     /* Handle the connection. */
                     from.fd = remote;
                     to.fd = remote;
-                    processConnection();
+                    processConnection(trusted);
 
                 } catch (std::exception & e) {
                     writeToStderr("child error: " + string(e.what()) + "\n");

src/nix-store/nix-store.cc

@@ -694,7 +694,9 @@ static void opExport(Strings opFlags, Strings opArgs)
         else throw UsageError(format("unknown flag `%1%'") % *i);
 
     FdSink sink(STDOUT_FILENO);
-    exportPaths(*store, opArgs, sign, sink);
+    Paths sorted = topoSortPaths(*store, PathSet(opArgs.begin(), opArgs.end()));
+    reverse(sorted.begin(), sorted.end());
+    exportPaths(*store, sorted, sign, sink);
 }
 
 

version

@@ -1 +1 @@
-1.5.2
+1.5.3