Alternative to cl/4661 discussed in #1044.
It can be assumed that `$tmpdir/build-top` can be created safely without
any risk, this way, we don't need to reuse the random directory creation
primitive.
Fixes#1044.
Change-Id: Iec52477f3047fc40959b183c607312d5a40fc8c9
Signed-off-by: Raito Bezarius <raito@lix.systems>
Nix daemon runs in a service context, usually with systemd.
If the Nix daemon unit adds `[Service] CacheDirectory=nix-daemon`, it should
successfully use `/var/cache/nix-daemon` automatically.
Instead, it uses `/root/.cache` right now, which is really bad.
By default, we add `[Service] CacheDirectory=nix` now which means that
caches are moved into `/var/cache/nix`.
Fixes#634.
Change-Id: I854b1045bfdce8f60110aea70bda1bf6657dfd7b
Signed-off-by: Raito Bezarius <raito@lix.systems>
Recently, in 54df89f601, support for mdbook 0.5
was introduced, including some logic to handle the `sections` -> `items`
rename. However, compatibility with 0.4's `sections` was only kept on the read
path, while writing 0.5's `items` unconditionally, which ends up in the bit
bucket on 0.4, effectively disabling substitution fully and leaving the include
directives in the final documentation. Restore writing into the `sections` when
they were there so that substitution works again.
Change-Id: Idd4d7653012660f3f7fc27f81f29b82d6a6a6964
This contains the depth of nix shells nesting.
Fixes#826
Co-authored-by: Qyriad <qyriad@qyriad.me>
Change-Id: If584c9d02730d6c857636dafdeab0c01f4ec8e0f
If the NIX_PATH shadows internal files, this will often break things,
hence we forbid it.
Fixes#998
Change-Id: I70e5d389532ada1c9f910c60281abe565e3ce6bb
Maintains compatibility with mdbook 0.4.x.
Includes comments for what to remove once 0.5.x is the only we care
about. Some other changes technically could be changed at that point,
but currently serve to enable universal support of 0.4.x and 0.5.x
Fixes#1051.
Change-Id: Ic5b405038d180bcd357bbd9e5716879e0c26e5f5
We have done it!
The functional/lang framework has fully been migrated to functional2 :D
closes: #856
Change-Id: I63ad8d7dbcd9b5267ca04af68df73b1ffa3d6461
Added builtins.warn` which takes two arguments: a message that is
displayed as a warning during evaluation which must be a string and a
value that is returned from the expression.
The next commits add new settings to control the behavior of the new
builtin: `debugger-on-warn` allows the user to start the debugger and
`abort-on-warn` aborts evaluation with an error.
Unlike upstream, I chose not to mark evaluation warnings from
`builtins.warn` as distinct from other warnings because that breaks the
commonly expected logging format `level: message`.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Upstream-PR: https://github.com/NixOS/nix/pull/10592
Fixes: https://git.lix.systems/lix-project/lix/issues/579
Change-Id: I8658c88e5c27952b65e8b9f5525a572e0680cc1f
This adds the enablement code to support HTTP/3 if the user requests it.
We leave it disabled because h3 is not onpar with h2 performance.
Change-Id: I1fd3d4c97b972dcf36bccacc6c9a8290e22b31e0
Signed-off-by: Raito Bezarius <raito@lix.systems>
Channels have moved from `nixos.org/channels` to `cnannels.nixos.org`.
This udates all relevant links (excluding release notes) to use the new
canonical URLs and replaces HTTP with HTTPS.
Fixes#1031.
Change-Id: I212821c44ac5e482c8e9eaa415c7d8ee17ff8341
Signed-off-by: adam <me@adamperkowski.dev>
We start this section with shortcomings of unsandboxed builds.
Fixes#1018.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ieb17e4340beab0c1197951813ae602de453a3fd9
Signed-off-by: Raito Bezarius <raito@lix.systems>
The minimum build directory length on Darwin with default settings
when we were still using `/tmp` was 22 bytes. Deriving build directory
names from my local store, the median and mode were 43, the mean was
around 44½, and the maximum was 127.
The switch to `/nix/var/nix/builds` over `/tmp` added a 15 byte
penalty, and the additional `/b` directory added another 2.
Now that we use opaque build directory names, the length is 48, so
we’re still at a ~3½–5 byte penalty over the previous status
quo. This change brings us down to 43, matching the previous median.
Note that these calculations do not take into account the fact that
`/tmp` is a symbolic link to `/private/tmp` on Darwin. Anything
that was canonicalizing paths would have had an additional 8 byte
penalty in the previous status quo that is not applicable here,
so we may already be ahead even without this change. If the more
opaque directory name here is undesirable, then that factor could
potentially help us squeeze by without. Alternatively, in combination
with dropping the `/b` on Darwin we could use `/nix/var/nix/bld` or
similar, but I feel that the paths in general are sufficiently opaque
that it should be okay to go with the shorter option here. Given that
some projects already had to reduce filename lengths to avoid this
limit even before the recent changes, I think it is best to try and
improve on the previous status quo.
Note that `/nix/var/nix/builds` will unfortunately not be cleaned up
on Darwin. However, we don’t clean up the directories inside it on
Darwin anyway, so hopefully that’s okay for now?
Closes: #913
Change-Id: I6a6a6964bffce7194bcddcaefb4c4a37569c7df5
the change to shareable thunks also removed a few cases of pointer
equality checks that allowed structures containing functions to be
considered equal to other sets containing the same functions, even
if the sets themselves were pointer-equal themselves. *so* busted.
Change-Id: If87fdab658f9037ce2a654f69a9e3da6ae2f53e5
This helps users to debug whether their current profile symlinks are
correctly set.
Expected outputs look like this:
```
❯ sudo ./outputs/out/bin/nix doctor
[snip]
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: local
[FAIL] Error: current generation cannot be discovered for profile:
'/nix/var/nix/profiles/default'
```
```
❯ ./outputs/out/bin/nix doctor
[snip]
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
[PASS] You have 28 generations for profile
'/nix/var/nix/profiles/per-user/raito/profile'
The current generation number is '290'
```
Change-Id: I50c69cbeac3291d668f4c2332803411579adc944
Signed-off-by: Raito Bezarius <raito@lix.systems>
Co-authored-by: Qyriad <qyriad@qyriad.me>
libarchive's xz offers single threaded xz compression which is very slow
and provides ~10-20Mbps compression speed in addition to maxing a core.
In exchange, it achieves optimal compression ratios among all our
compression methods.
Nonetheless, xz prevent the saturation of 1Gbps+ connections and slow
down significantly decompression for end users. As these connections and
faster hardware is becoming prevalent for cache servers and clients, we
offer to default to zstd.
Lix is a "compress once, decompress many times" application. To avoid
incurring a high penalty to end users very sensitive to compress ratio
(very slow Internet connections), we dampen the consequences of
switching to zstd by increasing the default zstd level to 12.
On one example, xz will compress a 4.4GB file to 632MB, zstd on 12 will
compress it to 775MB, that is a ~18 % increase over the optimal xz
compression. zstd took 18 seconds to produce this file.
Increasing to level 14 leads to a 773MB file while taking 37s.
Increasing to level 16 leads to 735MB file while taking 66s.
Finally, xz took 77s, so a 50 % reduction in time taken to compress in
exchange of an increase of 18 % of the compressed size.
This change will reduce issues encountered in #945 but is probably not
the root cause.
References:
- https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961
Change-Id: I7beda2bf2c1fed146dcb797b8f85dc290c486ab2
Signed-off-by: Raito Bezarius <raito@lix.systems>
As we saw actual users running into them throughout the debugging of
#920 and #1014.
It's best to document and point end users to this so they can take
measures for themselves.
Change-Id: I7d08407f4354055bf65fc6dd7d1624c5a9304402
Signed-off-by: Raito Bezarius <raito@lix.systems>
Eval errors will now print a simple, no frills chain of involved
derivations at the bottom. For example, trying to evaluate
`pkgs.xonsh.override { python3 = pkgs.python2; }` has the usual
Package ‘python-2.7.18.8’ in /nix/store/9v6qa656sq3xc58vkxslqy646p0ajj61-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:398 is marked as insecure, refusing to evaluate.
message, but now also includes the following:
note: trace involved the following derivations:
derivation 'xonsh-0.19.9'
derivation 'python2.7-xonsh-0.19.9'
derivation 'python2.7-setuptools-44.0.0'
To give the user information about why the erroring derivation was
involved in the first place.
We would like more structured information in the future, but this should
still be a significant improvement.
Change-Id: Icf6da52abd0a043cfb63943bf0b0c160c21ee59e
Includes link to the Buildkite SSO login, which is currently not visible
anywhere else in documentation.
Change-Id: Ic3ec6412127b2ac0df7e6efc4ec6020cc9d8b34f
These are a footgun and are not acceptable in functional2 due to thread
safety, effects on other tests, etc.
Co-authored-by: Commentator2.0 <lix@crystal-cavern.systems>
Change-Id: I8d7285061eaa9bab27edd52f3646024c8cf605e5
this has no performance impact in any benchmarks we've run. nul bytes
are still used as implicit truncation points in many places all over:
rejecting them in all locations that treat them as a string end point
requires large changes such as using a proper path library everywhere
Change-Id: I936158bd435f6abf009a689adfbc24496262c578
Fixes fj#940.
When running `nix-shell`, the `$NIX_BUILD_TOP` environment variable is
set to `$TMPDIR` or `/tmp`.
nixpkgs stdenv uses $NIX_BUILD_TOP to create `$NIX_BUILD_TOP/env-vars`
which contains all the environment variables set by stdenv. This is used
for debugging purposes in combination with `--keep-failed` to reload the
bash environment of a derivation.
`$TMPDIR` is often unset, therefore, `/tmp/env-vars` was constantly
being created. On a multi-user system or, when you run Lix as root, you
might create a `/tmp/env-vars` with different permission bits.
As a result, `nix-shell` can cease to function because that file will
fail creation for an unprivileged user for example.
fj#940 rightfully remark that the code is not consistent between
nix3-develop and nix-shell and it should be reworked.
Change-Id: Iddf15945385d8bd497b2800b37fee5e1f97689b7
Signed-off-by: Raito Bezarius <raito@lix.systems>
nix develop should ignore output checks in general.
This was done only for the old way of specifying output checks, the
structured attrs way requires rewriting the JSON and removing the output
checks pieces.
We take a brutal approach of removing as many as possible including
non-recommended ways of doing it.
Fixes#997.
Change-Id: Iaf83029016c71b5171e56e15d4eadc1a60a8be98
Signed-off-by: Raito Bezarius <raito@lix.systems>
Closes#987
The patch adds a flag `--no-instantiate` which only performs evaluation
without instantiating any derivations. Hence, GC root creation is also
skipped. To achieve that, Lix is also put in read-only mode and all
operations that require reading a derivation (e.g. constituents or
listing input derivations) are disabled fallback values are set.
This is a port of an upstream PR[1]. Given the divergence of the
codebases (different restructurings on both ends, no more CA derivations)
I decided to redo large portions from scratch instead of
cherry-picking the patches. Hence, the authorship.
Additionally the clean up of casts down to a local store are removed or
guarded behind an if, as done in the upstream PR.
[1] https://github.com/nix-community/nix-eval-jobs/pull/379
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Change-Id: Ib84f44e7799bc5577fd2ee98912458f16ebeab81
Fixes: https://git.lix.systems/lix-project/lix/issues/973
Information about which commands were executed is really valuable to
debug Lix and is much more user relevant than the vast majority of the
e.g. build loop junk printed at debug level. Currently we have a *whole
lot* of call sites where we call execv* which should probably be cleaned
up, but that's future work.
I chose to print argv0 rather than the executable path if these differ,
since the code is shorter and since the command could be a fully
resolved symlink or so where argv0 is the actual command name being run.
However, it's not exactly *hard* to write std::ranges::drop_view(args,
1).
Change-Id: I73c3abb20b229d5e2d64277aa29cbbeed7764bab
Previously two cryptography libraries were linked into Lix: OpenSSL used for
hashing and (in usual configurations) indirectly via curl for TLS, and Sodium
used only for handling the Ed25519 path info signatures. The latter is
functionally redundant since OpenSSL supports the same use case as well.
Reimplement the Ed25519 handling using OpenSSL and drop Sodium.
Fixes: https://git.lix.systems/lix-project/lix/issues/969
Change-Id: I6a6a696456b9d3ad7fdc2bf9b0759836a6247a38
Closes#551
This adds a special accessor that falls back to checking if a store-path
exists within a chroot if it's not a valid path. That way,
`genGraphString` can find out which files have which references before
the outputs are registered.
Change-Id: I03c9d508fa3c72e5c262194461a25d71f3f4de15
it's broken, can write arbitrary file paths when run as root, and only
supports strings and recursive sets of strings. this was only used for
manpage generation in a build system that has not woken up since 1976.
fixes#974fixes#227
Change-Id: I4f18599685a3077c15ddc02c759558f986c8c6e4
Closes#932
`connect-timeout` gets replaced by an exponential backoff for the
download timeout where the initial value is controlled by the setting
`initial-connect-timeout`.
Per iteration, the upper limit of the timeout is increased set to
timeout := min(max_connect_timeout, initial_connect_timeout * 2^i)
I decided to move the entire timeout / tracking of attempts into its own
class to not make the filetransfer implementation more complex. Also,
that allows us to write unit-tests for it.
Setting `--download-attempts` to `0` is forbidden now and an exception
will be thrown. For `--offline` we set it to `1`, the behavior is
equivalent to what it was before: whether the max tries were exceeded is
only checked after the first download exception got thrown, i.e. there's
still one attempt being made.
The end-result - with timeouts being caused by a wrongly set proxy -
looks like this:
$ env HTTPS_PROXY=1.1.1.1 nix store ping --store https://example.com
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 5006 milliseconds (curl error code=28); retrying in 422ms ms (attempt 1/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 10010 milliseconds (curl error code=28); retrying in 1003ms ms (attempt 2/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 20020 milliseconds (curl error code=28); retrying in 2018ms ms (attempt 3/5)
warning: error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 40007 milliseconds (curl error code=28); retrying in 4087ms ms (attempt 4/5)
error: unable to download 'https://example.com/nix-cache-info': Connection timed out after 80074 milliseconds (curl error code=28)
Change-Id: I9e8d08d78275bcf60080d663febc9e075243d36b
Concept: what if you could, in your fancy terminal, in the year of our
lord 2025, just click on the attrs you're looking at to go to where
they're defined. Currently we only expose this info as
builtins.unsafeGetAttrPos, which is inconvenient as it's not
discoverable to users.
By putting it in this more visible yet invisible spot, it's more likely
to be more useful to more people.
In the current state, this is not the most useful ever due to stuff like
https://github.com/neovim/neovim/discussions/35097. However, it can be
expanded by perhaps adding something like the url format setting ripgrep
has.
Change-Id: I3947f97d5c2056d59099af468d7b855486438227
See [my comment] on the Nix PR to restore the previous behaviour
for why I believe we should remove this for the next release. The PR
should still be backported to stable releases to avoid making breaking
changes to their semantics.
[my comment]: <https://github.com/NixOS/nix/pull/13741#issuecomment-3180851635>
Fixing this across supported Lix versions is required for Nixpkgs to
update toml11, which is a blocker for the CMake 4 update.
Change-Id: I6a6a69642e6b6cb13a9fccc0778e9158b53102d5
This version changes the handling of TOML timestamps, and throws an
error on out‐of‐range integer literals rather than the previous
saturating behaviour, as required by [the TOML v1.0.0 specification]:
> Arbitrary 64-bit signed integers (from −2^63 to 2^63−1) should be
> accepted and handled losslessly. If an integer cannot be represented
> losslessly, an error must be thrown.
[the TOML v1.0.0 specification]: <https://toml.io/en/v1.0.0#integer>
The only known use of this is a questionable Nixpkgs test that I have
proposed [a fix] for.
[a fix]: <https://github.com/NixOS/nixpkgs/pull/433710>
Bumping this ahead of Nixpkgs ensures we can test these cases on
HEAD in advance. I presume that the next Lix major version will be
released after 25.05 goes out of support, so it should be fine to
drop support for the old version of toml11.
The co‐authors of this commit are the contributors to the vendored
package definition from Nixpkgs.
Co-authored-by: Anderson Torres <torres.anderson.85@protonmail.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: Silvan Mosberger <silvan.mosberger@moduscreate.com>
Change-Id: I6a6a69644a188b6e09eee5c9cf91ddd3c81d24ee
JSON has only one numeric type, with vague semantics. [RFC 8259] says:
> This specification allows implementations to set limits on the range
> and precision of numbers accepted. Since software that implements
> IEEE 754 binary64 (double precision) numbers [IEEE754] is generally
> available and widely used, good interoperability can be achieved by
> implementations that expect no more precision or range than these
> provide, in the sense that implementations will approximate JSON
> numbers within the expected precision. A JSON number such as 1E400
> or 3.141592653589793238462643383279 may indicate potential
> interoperability problems, since it suggests that the software that
> created it expects receiving software to have greater capabilities
> for numeric magnitude and precision than is widely available.
>
> Note that when such software is used, numbers that are integers and
> are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
> sense that implementations will agree exactly on their numeric
> values.
[RFC 8259]: <https://www.rfc-editor.org/rfc/rfc8259.html#section-6>
Floating‐point numbers are annoying to deal with in Nix, so it
optimistically parses integer‐looking literals as Nix‐language
integers where possible. Nixpkgs relies on this behaviour, as it backs
its `lib.toInt` family of functions with `builtins.fromJSON` in lieu
of a real integer‐parsing built‐in, and treats floating‐point
outputs as an error. Therefore, dealing with integer‐looking
JSON number literals that are outside the interoperable range is
unavoidable.
However, this raises the question of how literals that look
like integers, but exceed the range of a Nix‐language integer,
should be handled. The JSON library we use attempts to represent
integer‐looking literals as a unsigned or signed C++ integer type
before falling back to floating‐point numbers.
This means that we were parsing literals below −2⁶³ as
floating‐point numbers, while rejecting ones above (2⁶³ + 1)
with an error. This was done to avoid the C++ undefined behaviour in
the previous code path, but is hard to justify. This change causes
them to both be parsed as floating‐point numbers.
The alternative would be to reject the signed case too. However,
I believe that is less consistent with how JSON handles its single
numeric type, less interoperable with JSON documents found in the wild,
and that it is preferable to avoid the function being needlessly
partial in this case. It does mean that round‐tripping is less
lossless than before, but extreme floating‐point values already
caused these kinds of issues, and from JSON’s point of view that’s
exactly what these literals represent. Only numeric values outside the
RFC’s suggested interoperable range will have any round‐tripping
issues, and we continue to guarantee the behaviour on all values that
can be represented as Nix‐language integers.
Change-Id: I6a6a696412383e8a2cc160397716cb7f7bc7a2d4
Raito Bezarius