Compare commits

..
Author SHA1 Message Date
Jade Lovelace f82a2a9aaa release: 2.90.0-rc1 "Vanilla Ice Cream"
Release produced with releng/create_release.xsh

Change-Id: Ib8df5ea5096d47b25f74988447146c1b5072b869
2024-06-14 19:40:13 -07:00
jadeandGerrit Code Review e1059bfa34 Merge changes from topic "releng" into main
* changes:
  releng: add sha256 for the manual tarball
  releng: fix upload of multiarch images to forgejo
  releng: fix git checking
  releng: fix logging inside interactive xonsh
  releng: support multiple systems
  version: update to 2.90.0-rc1
2024-06-15 02:38:09 +00:00
Jade Lovelace d5c670ad01 releng: add sha256 for the manual tarball
Whoops.

Change-Id: Ic6f8cdcb074d679e9b1fc3323c106cc853328dcc
2024-06-13 17:14:06 -07:00
Jade Lovelace 16ea19ced8 releng: fix upload of multiarch images to forgejo
Forgejo appears to immediately delete registry content that is
overwritten. This means that we are forced to delete our previous
workaround of making a temporary tag and use a new, more absurd
workaround of making an entire temporary image that we basically only
need to create to get its hash.

However, on the plus side, the new workaround doesn't create garbage
tags to begin with, which means that we don't have to deal with GitHub
not implementing the standardized tag delete endpoint and instead
only implementing a proprietary one.

Upstream-Bug: https://github.com/containers/skopeo/issues/2354
Change-Id: I220e7ce9a17fd230c38882f12c009a166dcc9336
2024-06-13 17:12:45 -07:00
Jade Lovelace 7be0d237e0 releng: fix git checking
Change-Id: I82ddd918311b48e596adb807b81221973113fe7a
2024-06-13 15:24:21 -07:00
Jade Lovelace e715e5fd31 releng: fix logging inside interactive xonsh
I don't know when this broke, it seems like it happened since the 24.05
upgrade, so xonsh 0.15.

What happened is that xonsh was trying to intercept log output, which
explodes if you have the logger survive past one command input. This is,
however, impossible to avoid if you are trying to use logging when you
import releng from inside xonsh for interactive use!

The error below is because the memory handler backing the stdout/stderr
of the one command that's just been run was closed after the command
completed.

Change-Id: I2be642aebf93da9818d08ff8b97c2e72ba5ac581

--- Logging error ---
Traceback (most recent call last):
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/base_shell.py", line 183, in write
    self.mem.write(s)
ValueError: I/O operation on closed file.
Call stack:
  File "/nix/store/xgdp1p1gv8ni1awnkzyqasnn6gz5wlvx-xonsh-0.15.1/bin/xonsh", line 8, in <module>
    sys.exit(main())
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/main.py", line 470, in main
    sys.exit(main_xonsh(args))
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/main.py", line 514, in main_xonsh
    shell.shell.cmdloop()
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/ptk_shell/shell.py", line 406, in cmd
loop
    line = self.singleline(auto_suggest=auto_suggest)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/xonsh/ptk_shell/shell.py", line 374, in sin
gleline
    line = self.prompter.prompt(**prompt_args)
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1
026, in prompt
    return self.app.run(
  File "/nix/store/34951j60xcsw6zj4v8lsaf491acv0by3-python3-3.11.9-env/lib/python3.11/site-packages/prompt_toolkit/application/application.py",
 line 1002, in run
    return asyncio.run(coro)
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 189, in run
    with Runner(debug=debug) as runner:
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 59, in __enter__
    self._lazy_init()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/runners.py", line 137, in _lazy_init
    self._loop = events.new_event_loop()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/events.py", line 810, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/events.py", line 699, in new_event_loop
    return self._loop_factory()
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/unix_events.py", line 64, in __init__
    super().__init__(selector)
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/asyncio/selector_events.py", line 54, in __init__
    logger.debug('Using selector: %s', selector.__class__.__name__)
Message: 'Using selector: %s'
Arguments: ('EpollSelector',)

Change-Id: I90959809129aaf96aad4577599031688599ed85e
2024-06-13 15:17:44 -07:00
Jade Lovelace 068576042b packaging: Move binaryTarball to a passthru attr in package.nix
This is motivated by flakes being bad and all the stuff that calls
things by "system" being utterly unable to cope with cross compilation.
So if we go shove it in package.nix it is suddenly usable from cross
contexts.

Usage:

```
nix build -L .#nix-riscv64-linux.binaryTarball
```

Change-Id: I702ebf2ac5bd9d1c57662f968b000073134df336
2024-06-13 15:14:22 -07:00
Jade Lovelace d194939ff5 flake.nix: add riscv64 cross target
It builds. I have not tested the binaries since I don't have hardware,
but I would be rather surprised if it were broken, given that nix *runs*
on this platform.

Change-Id: I0b474ffcd4a431bf117a303d0b65fa6532113f48
2024-06-13 15:04:06 -07:00
Jade Lovelace 74fb2e8c47 releng: support multiple systems
I guess this is kind of important to being able to "release it".

Change-Id: Id6f295d0b4944fa1203783a400a246727dbd94b6
2024-06-13 14:36:03 -07:00
Jade Lovelace ac28cff28f version: update to 2.90.0-rc1
Change-Id: I913852cfb88b3b300ffb1050a91784be659ae66a
2024-06-13 13:03:25 -07:00
alois31andGerrit Code Review 1d6fd94cf9 Merge "tests/libcmd: set HOME to a temporary directory" into main 2024-06-13 05:27:55 +00:00
Jade Lovelace d0b28f0e74 releng: fix docs upload
There were two bugs I found:
1. If the build isn't already done in the store, nix-store --realise
   does not know how to build it. You have to just give it the
   derivation and I guess it will realise all outputs, which is fine.
2. cp without -T will not overwrite an existing manual directory,
   creating a path manual/manual.

Change-Id: Ibebfd136a266da5330944a985e636ebb776f1909
2024-06-12 15:34:23 -07:00
Jade Lovelace 38e4e69633 .github: clean up stale things and wrong references
We do not need a stale bot. We do not need dependabot.

Fixes: https://git.lix.systems/lix-project/lix/issues/391
Change-Id: I983fae4dc4cd9022b12f70e330b5c984c5fc1b9d
2024-06-12 15:34:23 -07:00
Pierre BourdonandJade Lovelace 248ecb11af dev shell: Add bashInteractive
This was bothersome to me as a zsh+direnv user.

Change-Id: Ia5b54cc63647a5c6ced2b5412e972dac1abf8184
2024-06-12 15:34:23 -07:00
Jade Lovelace 73898cad0e tests/flake-registry: Fix occasional deadlocks
This seems to have been caused by having the wrong PID. I don't know why
it worked before in the sandbox, but the code was definitely wrong
before, so let's just fix it.

Change-Id: I556580bdf614c716566310e975a36daa6d6c9a91
2024-06-12 15:34:23 -07:00
Jade Lovelace 59b5965bbf doc: Write an index of environment variables used in testsuite
This was originally going to be just the testsuite but I kinda just
documented all of them.

I am tired of us not documenting these. This is a starting point to
producing an actually good index. I would like to enforce it in a
pre-commit hook eventually that we document all environment variables
used in Lix itself, even if it is terse dev facing docs.

This is full of a bunch of TODOs caused by auditing code. They should
probably be done at some point.

Change-Id: I7c0d3b257e19bae23d47d1efbd7361d203bccb0e
2024-06-12 15:34:23 -07:00
Jade Lovelace 211f79d4a2 doc/testing.md: Rewrite some outdated sections for meson and current source layout
Change-Id: Ia23f82c9a564b55bd799afbda59c28c9b0a65c13
2024-06-12 15:34:23 -07:00
Jade Lovelace 5f6eb6eb44 doc: rewrite the multi-user documentation to actually talk about security
It's in the security section, and it was totally outdated anyway.

I took the opportunity to write down the stuff we already believed.

Change-Id: I73e62ae85a82dad13ef846e31f377c3efce13cb0
2024-06-12 15:34:23 -07:00
Jade Lovelace d9345d8836 tests: verify that NIX_DAEMON_PACKAGE's version looks like a version
Followup to https://gerrit.lix.systems/c/lix/+/1417 to ensure that this
parser will never take something that doesn't look like a version.

It turns out this problem is less alarming than initially thought
because it only applies to the testsuite in a non-default mode.

Change-Id: I26aba24aaf0215f2b782966314b94784db766266
2024-06-12 15:34:23 -07:00
Jade Lovelace 6939ffc9f9 Check devShells in CI
We should not let these regress in CI by having broken dependencies or
similar. Still need to fix the evaluation error checking in
buildbot-nix, but this is a useful step regardless.

Fixes: https://git.lix.systems/lix-project/lix/issues/383

Change-Id: I3883184165440e66256c989117f2ab2e54c3aafd
2024-06-12 15:34:23 -07:00
Jade Lovelace 479055aee8 Misc workaround removals since 24.05 upgrade
Change-Id: I9491b103333cb0e25c245199e88365ded7800d2e
2024-06-12 15:34:23 -07:00
Pierre BourdonandJade Lovelace f7b6552699 [resubmit] flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
-- message from cl/1418 --

The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

-- jade resubmit changes --

This is a resubmission of https://gerrit.lix.systems/c/lix/+/1418, which
was reverted in https://gerrit.lix.systems/c/lix/+/1432 for breaking CI
evaluation without being detected.

I have run `nix flake check -Lv` on this one before submission and it
passes on my machine and crucially without eval errors, so the CI result
should be accurate.

It seems like someone renamed forbiddenDependenciesRegex to
forbiddenDependenciesRegexes in nixpkgs and also changed the type
incompatibly. That's pretty silly, but at least it's just an eval error.

Also, `xonsh` regressed the availability of `xonsh-unwrapped`, but it
was fixed by us in https://github.com/NixOS/nixpkgs/pull/317636, which
is now in our channel, so we update nixpkgs compared to the original
iteration of this to simply get that.

We originally had a regression related to some reorganization of the
nixpkgs lib test suite in which there was broken parameter passing.
This, too, we got quickfixed in nixpkgs, so we don't need any changes
for it: https://github.com/NixOS/nixpkgs/pull/317772

Related: https://gerrit.lix.systems/c/lix/+/1428
Fixes: https://git.lix.systems/lix-project/lix/issues/385

Change-Id: I26d41ea826fec900ebcad0f82a727feb6bcd28f3
2024-06-12 15:34:22 -07:00
Alois Wohlschlager 3c0434999e tests/libcmd: set HOME to a temporary directory
The libcmd unit test creates files (more specifically, the fetcher cache) in
its home directory. In the single-user sandbox, this leads to the creation of
/homeless-shelter, since this is the default HOME and the root is writable.
Unfortunately, this conflicts with the assumption of the functional tests that
this directory does not exist. Use a different home directory to prevent these
test failures, and thus restore the ability to build inside the single-user
sandbox.

Fixes: https://git.lix.systems/lix-project/lix/issues/365
Change-Id: I4df8c53d043234b95a7c0ac45fc5ee89e8d46aff
2024-06-12 22:13:55 +00:00
Qyriad f46194faa2 build: remove unused 'deps' variable
This never actually got used

Change-Id: I8f3f1d413124b27913d59a75cff42319cbaac385
2024-06-12 21:40:57 +00:00
jadeandGerrit Code Review 8a3d063a49 Merge changes from topic "releng" into main
* changes:
  releng: add prod environment, ready for release
  releng: automatically figure out if we should tag latest for docker
  releng: support multiarch docker images
  manual: rewrite the docker guide now that we have images
  Rewrite docker to be sensible and smaller
  Implement docker upload in the releng tools
2024-06-11 04:45:12 +00:00
jadeandGerrit Code Review f432e464dd Merge "tests: fix daemon version in isDaemonNewer function" into main 2024-06-10 23:22:05 +00:00
jadeandGerrit Code Review a986a8dfa1 Merge "Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"" into main 2024-06-10 04:48:12 +00:00
jadeandGerrit Code Review 8a09465c3a Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"
This reverts commit 28a079f841.

Reason for revert: This caused a pile of regressions in CI, and does not pass nix flake check. Some number of them are fixed in CL: https://gerrit.lix.systems/c/lix/+/1429 but there's more to be fixed.

We should defer this after 2.90.

Change-Id: Ib839d0fcb08eb52094af2b521145e3c1b4e0556f
2024-06-10 04:29:13 +00:00
Jade Lovelace 82dc712d93 releng: add prod environment, ready for release
I am *reasonably* confident that this releng infrastructure can actually
build a Lix 2.90 and release it successfully. Let's make it possible to
do, and add some cute colours to the confirmation message.

Change-Id: I85e498b6fb49ffc5e75c0a72c5e45fb1f69030d3
2024-06-09 20:33:24 -07:00
Jade Lovelace ce71d0e9ab releng: automatically figure out if we should tag latest for docker
For example, when releasing from release-2.90, if `main` has a 2.91 tag
ancestor, we know that 2.91 was released, so we should *not* tag latest.

Change-Id: Ia56b17a2ee03bbec74b7c271c742858c690d450d
2024-06-09 20:33:24 -07:00
Jade Lovelace 9aeb314e6a releng: support multiarch docker images
If we don't want to have separate registry tags by architecture (EWWWW),
we need to be able to build multiarch docker images. This is pretty
simple, and just requires making a manifest pointing to each of the
component images.

I was *going* to just do this API prodding with manifest-tool, but it
doesn't support putting metadata on the outer manifest, which is
actually kind of a problem because it then doesn't render the metadata
on github. So I guess we get a simple little containers API
implementation that is 90% auth code.

Change-Id: I8bdd118d4cbc13b23224f2fb174b232432686bea
2024-06-09 20:33:24 -07:00
Jade Lovelace 4392d89eea manual: rewrite the docker guide now that we have images
Change-Id: I5bdf47e67059ae4099552750a47ae070dbe094df
2024-06-09 20:33:24 -07:00
Jade Lovelace 9bb7fb8f69 Rewrite docker to be sensible and smaller
I have checked the image can build things and inspected `diff -ru`
compared to the old image. As far as I can tell it is more or less
the same besides the later git change.

Layers are now 65MB or less, and we aren't against the maxLayers limit
for the broken automatic layering to do anything but shove one store
path in a layer (which is good behaviour, actually).

This uses nix2container which streams images, so the build time is much
shorter.

I have also taken the opportunity to, in addition to fixing the 400MB
single layer (terrible, and what motivated this in the first place),
delete about 200MB of closure size inflicted by git vs gitMinimal
causing both perl and python to get into closure.

People mostly use this thing for CI, so I don't really think you need
advanced git operations, and large git can be added at the user side if
really motivated.

With love for whichever container developer somewhat ironically assumed
that one would not run skopeo in a minimal container that doesn't have a
/var/tmp.

Fixes: https://git.lix.systems/lix-project/lix/issues/378

Change-Id: Icc3aa20e64446276716fbbb87535fd5b50628010
2024-06-09 20:33:24 -07:00
jadeandGerrit Code Review 7dfa2a761e Merge changes from topic "releng" into main
* changes:
  releng: support pushing the manual to docs also
  Expose officialRelease from the flake
  Put into place initial release engineering
2024-06-09 08:28:52 +00:00
Jade Lovelace ff95b980d4 Implement docker upload in the releng tools
This uses skopeo to not think about docker daemons. I, however, noticed
that the docker image we had would have totally terrible cache hits, so
I rewrote it.

Fixes: https://git.lix.systems/lix-project/lix/issues/252

Change-Id: I3c5b6c1f3ba0b9dfcac212b2148f390e0cd542b7
2024-06-09 00:30:12 -07:00
Pierre Bourdon 28a079f841 flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

Change-Id: I8c590e9fe25c0f566d0cfeacb96d8cf50abf12e8
2024-06-09 01:25:53 +02:00
Pierre Bourdon 9281a12532 tests/nixos/nix-copy: fix NixOS >= 24.05 compatibility
4b128008c5d9fde881ce1b0a25e60ae0415a14d5 in nixpkgs introduced a default
hashedPasswordFile for root in NixOS tests, which takes precedence over
the password option set in the nix-copy test.

Change-Id: Iffaebec5992e50614b854033f0d14312c8d275b5
2024-06-08 17:59:08 +02:00
Mario Rodas a05de58ebd tests: fix daemon version in isDaemonNewer function
Since ad8a4b380e, the version printer returns "nix (Lix, like Nix) 2.x",
hence the `daemonVersion` was being set to the string "like".

Using `compareVersions` with a letter compares them lexicographically:

   builtins.compareVersions "like" "2.12pre20230103"  // => -1
   builtins.compareVersions "like" "2.16.0"           // => -1

This caused that `isDaemonNewer` always returned 1, falsy in Bash terms.
Therefore, the test suite skipped those tests where they use it.

Fixes https://git.lix.systems/lix-project/lix/issues/324

Change-Id: If6682515bf0bf8b8add641af9a4e98b50a9acb51
2024-06-08 04:20:00 +00:00
jadeandGerrit Code Review 4f94531209 Merge changes from topic "releng" into main
* changes:
  Add meson release note
  Move version to a JSON file so we can have release names
  Remove rl-next-dev
2024-06-07 03:53:31 +00:00
Jade Lovelace 98e8475147 releng: support pushing the manual to docs also
Change-Id: Ifd0b51425ee4955e0230fb2804a6f54ef0fe16e9
2024-06-06 20:53:08 -07:00
Jade Lovelace bdf1b264ad Expose officialRelease from the flake
Change-Id: If87beb3f31dfb5d59862294ac2e1c821ea864277
2024-06-06 20:53:08 -07:00
Jade Lovelace c32a01f9eb Put into place initial release engineering
This can release x86_64-linux binaries to staging, with ephemeral keys.
I think it's good enough to review at least at this point, so we don't
keep adding more stuff to it to make it harder to review.

Change-Id: Ie95e8f35d1252f5d014e819566f170b30eda152e
2024-06-06 20:53:08 -07:00
QyriadandGerrit Code Review ec768df004 Merge changes Ic4be41eb,I48db2385 into main
* changes:
  devshells: only enable pch for clang
  build: expose option to enable or disable precompiled std headers
2024-06-06 22:21:52 +00:00
Jade Lovelace 611b1de441 Add meson release note
Change-Id: I4d2d08dc77a3ab4dce9fbb129c1487aa8c9f1722
2024-06-06 15:08:12 -07:00
Jade Lovelace 9c77c62e73 Move version to a JSON file so we can have release names
Change-Id: I5ff3396a302565ee5ee6c2db97e048e403779076
2024-06-06 15:08:12 -07:00
Jade Lovelace 24057dcb6a Remove rl-next-dev
We realized that there's really no good place to put these dev facing
bulletins, and the user-facing release notes aren't really the worst
place to put them, I guess, and we do kind of hope that it converts
users to devs.

Change-Id: Id9387b2964fe291cb5a3f74ad6344157f19b540c
2024-06-06 15:08:12 -07:00
Jade Lovelace 1659404626 Add xonsh to the shell
Change-Id: If8f3825d2bdcc3f1d00583a11d890c1c8ab37b9f
2024-06-06 14:50:27 -07:00
Jade LovelaceandQyriad e0748377dc pname: nix -> lix
This had a regression last time: https://gerrit.lix.systems/c/lix/+/1196

But f3f68fcfa fixed upgrade-nix to not be broken, so this should be ok tbh.

Change-Id: I48ea1359790878bb8ead5d8a4b3f61caa4aabfb5
2024-06-06 20:42:29 +00:00
Qyriad 766e718f67 devshells: only enable pch for clang
clangd seems to break if GCC is using precompiled headers for C++'s
standard library, so this sets -Denable-pch-std=${stdenv.cc.isClang}

Fixes #374.

Change-Id: Ic4be41ebe7576ebcb9c208275596f953c2003109
2024-06-06 12:48:13 -06:00
Qyriad 06e65e537b build: expose option to enable or disable precompiled std headers
They are enabled by default, and Meson will also prints whether or not
they're enabled at the bottom at the end of configuration.

Change-Id: I48db238510bf9e74340b86f243f4bbe360794281
2024-06-06 12:46:26 -06:00
jadeandGerrit Code Review 8f9bcd20eb Merge "libstore/filetransfer: fix no-s3 build" into main 2024-06-06 03:08:14 +00:00
Linus HeckemannandJade Lovelace 609b721425 libstore/filetransfer: fix no-s3 build
Fixes a compiler error that looks like:

error: could not convert '[...]' from 'future<void>' to 'future<nix::FileTransferResult>'
Change-Id: I4aeadfeba0dadfdf133f25e6abce90ede7a86ca6
2024-06-05 15:50:57 -07:00
Pierre BourdonandGerrit Code Review 6e59b4b407 Merge "tests/nixos: make the tarball-flakes test better reflect real use cases" into main 2024-06-04 15:14:02 +00:00
Pierre Bourdon a3256a9375 tests/nixos: make the tarball-flakes test better reflect real use cases
In most real world cases, the Link header is set on the redirect, not on
the final file. This regressed in Lix earlier and while new unit tests
were added to cover it, this integration test should probably have also
caught it.

Change-Id: I2a9d8d952fff36f2c22cfd751451c2b523f7045c
2024-06-04 08:12:59 +02:00
Olmo Kramerandjade 72d85acba4 nix flake update: add test for multiple inputs from nix#10073
Upstream-PR: https://github.com/NixOS/nix/pull/10073
Change-Id: I53fcb43b387e55439e062e208877afeb88493bb4
2024-06-03 21:50:33 +00:00
Nikodem Rabulińskiandjade 5d3910330d Show message about --update-input being replaced by nix flake update
Fixes: https://git.lix.systems/lix-project/lix/issues/283

Change-Id: I6ee23874cb09f51d788521273076a25ba8764859
2024-06-03 21:50:33 +00:00
Nikodem Rabulińskiandjade cc3674ea93 Accept multiple arguments to nix flake update
Fixes: https://git.lix.systems/lix-project/lix/issues/194

Change-Id: Ia7bd4f7640384be9827dbb7e2c594f0aa5f1aff8
2024-06-03 21:50:33 +00:00
Qyriad c55e93ca23 Revert "nix3: always use the same verbosity default (info)"
This reverts commit d0390b5cf2.

Other parts of the codebase will need to be adjusted in response to a
default verbosity change. Let's just push this to after 2.90.

Fixes #362.
Fixes #367.

Change-Id: I04648473579146851bda41d764adc1ef954c355d
2024-06-01 18:29:19 -06:00
QyriadandGerrit Code Review d374a9908f Merge "build: fix static linking with a hack" into main 2024-06-01 19:17:13 +00:00
Raito Bezarius b8cb7abcf0 chore: rebrand Nix to Lix when it makes sense
Here's my guide so far:

$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2

All items from this query have been tackled. For the documentation side:
that's for https://git.lix.systems/lix-project/lix/issues/162.

Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.

Fixes: https://git.lix.systems/lix-project/lix/issues/148.
Fixes: https://git.lix.systems/lix-project/lix/issues/162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-06-01 20:31:24 +02:00
Linus HeckemannandGerrit Code Review 5312e60be6 Merge "libfetchers: allow fetching gitlab refs with >1 commit" into main 2024-06-01 09:54:11 +00:00
Qyriad e54d4c9381 build: fix static linking with a hack
This causes libstore, libexpr, libfetchers, and libutil to be linked
with -Wl,--whole-archive to executables, when building statically.

libstore for the store backends, libexpr for the primops, libfetchers
for the fetcher backends I assume(?), and libutil for the nix::logger
initializer (which notably shows in pre-main constructors when HOME is
not owned by the user. cursed.).

This workaround should be removed when #359 is fixed.

Fixes #306.

Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31 21:47:16 -06:00
jadeandGerrit Code Review c7ca87461d Merge "build-remote: truncate+hash store URI used in lockfile paths" into main 2024-05-31 19:22:32 +00:00
jadeandGerrit Code Review 7081889faa Merge "truncate WAL files on exit" into main 2024-05-31 19:21:30 +00:00
jadeandGerrit Code Review adedac70fa Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into main
* changes:
  gitignore: delete 90% of it
  build-time: remove 20% more by PCH'ing C++ stdlib
  shellHook: make it actually run
2024-05-31 19:19:29 +00:00
Linus Heckemann 82de36f77a libfetchers: allow fetching gitlab refs with >1 commit
Change-Id: I945c4c5512def9eff728bb67fe3c03ae17f99d6d
2024-05-31 21:12:04 +02:00
jadeandGerrit Code Review a75d7a5777 Merge "libutil: fix args assert being thrown on Darwin in nix-eval-jobs" into main 2024-05-31 18:57:13 +00:00
alois31andGerrit Code Review ff08d95420 Merge "libstore/build: copy ca-certificates too" into main 2024-05-31 16:44:18 +00:00
713cd7e9e7 truncate WAL files on exit
Fix for https://github.com/NixOS/nix/issues/10300

https://github.com/lix-project/lix/commit/18a26202737a74f216d285d92bd4a84761788026  enabled persistent WAL files that will never get truncated. to fix this, journal_size_limit is set to 2^40, which results in the WAL files being truncated to 0 on exit, as well as limiting the WAL files to 2^40 bytes following a checkpoint.

this aligns lix with the nix change: https://github.com/NixOS/nix/pull/10301

https://www.sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal
https://www.sqlite.org/pragma.html#pragma_journal_size_limit
https://github.com/sqlite/sqlite/blob/ed517a708284b6e00b6ae5f1e3f702bbfcbd32ed/src/wal.c#L2518

PR-Link: https://github.com/lix-project/lix/pull/9

Co-Authored-By: paparodeo <170618376+paparodeo@users.noreply.github.com>
Change-Id: I90ec1a467c92c582ff8c07dd363a4cf789782214
2024-05-31 12:22:15 +00:00
Lunaphiedandjade d4b7e6baca build-remote: truncate+hash store URI used in lockfile paths
Fixes: https://git.lix.systems/lix-project/lix/issues/157
Fixes: https://git.lix.systems/lix-project/lix/issues/221

Previously the entire escaped store URI was included. This would cause
build failures if a very long or deeply nested path was being used in
the store.

Now, we use the first 48 characters of the URL (escaped), then 16 bytes
of hash of the entire URL. This should never collide and limits the
length of the file name to a bit over 64, which is fine.

Change-Id: Ic1ba690a94e83749567c2c29460b8d1bcf2ac413
2024-05-31 12:18:24 +00:00
Jade Lovelace ac78c1dcd5 libutil: fix args assert being thrown on Darwin in nix-eval-jobs
This is because a dynamic_cast<nix::RootArgs *> of a (n-e-j) MyArgs
returns nullptr even though MyArgs has virtual nix::RootArgs as a
parent.

class MyArgs : virtual public nix::MixEvalArgs,
               virtual public nix::MixCommonArgs,
               virtual nix::RootArgs { ... };

So this should work right?? But it does not. We found out that it's
caused by -fvisibility=hidden in n-e-j, but honestly this code was bad
anyway.

The trivial solution is to simply stop relying on RTTI working properly
here, which is probably better OO architecture anyway. However, I am not
100% confident *this* is sound, since we have this horrible hierarchy:

      Args (defines getRoot)
     /        |           \
RootArgs  MixCommonArgs  MixEvalArgs
(overrides)

I am not confident that this is guaranteed to resolve from Args always
in the case of this override.

Assertion failed: (res), function getRoot, file src/libutil/args.cc, line 67.
6MyArgsProcess 60503 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x0000000100b1a41c liblixutil.dylib`nix::Args::processArgs(std::__1::list<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, bool) [inlined] nix::Args::getRoot(this=0x00000001000d0688) at args.cc:67:5 [opt]
   64       std::cout << typeid(*p).name();
   65
   66       auto * res = dynamic_cast<RootArgs *>(p);
-> 67       assert(res);
   68       return *res;
   69   }
   70
Target 0: (nix-eval-jobs) stopped.

(lldb) p this
(MyArgs *) 0x00000001000d0688
(lldb) p *this
(nix::Args) {
  longFlags = size=180  { ... }
  shortFlags = size=4  { ... }
  expectedArgs = size=1  { ... }
  processedArgs = size=0 {}
  hiddenCategories = size=1 {
    [0] = "Options to override configuration settings"
  }
  parent = nullptr
}

We also found that if we did this:
class [[gnu::visibility("default")]] RootArgs : virtual public Args

it would work properly (???!). This is of course, very strange, because
objdump -Ct output on liblixexpr.dylib is identical both with and
without it.

Possibly related: https://www.qt.io/blog/quality-assurance/one-way-dynamic_cast-across-library-boundaries-can-fail-and-how-to-fix-it

Fixes: https://git.lix.systems/lix-project/nix-eval-jobs/issues/2
Change-Id: I6b9ed968ed56420a9c4d2dffd18999d78c2761bd
2024-05-31 12:17:06 +00:00
terruandGerrit Code Review 0c6cb34de6 Merge "document context-dependent keywords" into main 2024-05-31 10:07:50 +00:00
Alois Wohlschlager cf756fdf3c libstore/build: copy ca-certificates too
In b469c6509b, the ca-certificates file was
missed. It should be copied too so that we don't end up bind-mounting a broken
symlink.

Change-Id: Ic9b292d602eb94b0e78f77f2a27a19d24665783c
2024-05-31 07:54:18 +00:00
Jade Lovelace a6b33cb3b2 gitignore: delete 90% of it
*laughs in meson putting it all in build/*

Change-Id: Ifcb0d3104cf9e64c4de91c3a92828899a209d00d
2024-05-30 22:24:55 +00:00
Jade Lovelace 0f99ed43f1 build-time: remove 20% more by PCH'ing C++ stdlib
It seems like someone implemented precompiled headers a long time ago
and then it never got ported to meson or maybe didn't work at all.

This is, however, blessedly easy to simply implement. I went looking for
`#define` that could affect the result of precompiling the headers, and
as far as I can tell we aren't doing any of that, so this should truly
just be free build time savings.

Previous state:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

New state:
**** Time summary:
Compilation (567 times):
  Parsing (frontend):         1123.0 s
  Codegen & opts (backend):   1078.1 s

I wonder if the "regression" in codegen time is just doing the PCH
operation a few times, because meson does it per-target.

Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30 21:54:21 +00:00
Jade Lovelace e6e5cacabe shellHook: make it actually run
When we changed this in I91cb6eb6668f3a8eace36ecbdb01eb367861d77b to
not run in nested shells, we didn't predict that `nix develop` would do
something ridiculous and append -env to things silently. `nix-shell` of
course does not do this, so we need to tolerate both.

Change-Id: Ibe7cf546823d7358ebc0414ecbe154e3e3194f45
2024-05-30 21:54:21 +00:00
Linus Heckemann 3df013597d libfetchers: handle nonexistent refs in GitLab repos more gracefully
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

$ outputs/out/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error:
       … while updating the lock file of flake 'git+file:///home/linus/projects/lix?ref=refs/heads/fix-gitlab-nonexistent&rev=915f16a619a36237a099b9aa9afed6d14ff613b4'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the ref really exist?

Change-Id: Id9bc79d98348500e152ed519bb3ac79a3d15c38d
2024-05-30 21:53:51 +00:00
jadeandGerrit Code Review 2f104bbe3b Merge "Revert "tests/filetransfer: reënable on Darwin"" into main 2024-05-30 21:53:38 +00:00
jadeandGerrit Code Review 533d469875 Revert "tests/filetransfer: reënable on Darwin"
This reverts commit 285bc67318.

Reason for revert: https://git.lix.systems/lix-project/lix/issues/364

For some reason this broke `main` even though the change we are reverting passed CI! Mysterious, haunted, etc. Needs more debugging, let's turn it off for now.

Change-Id: Ica4819d61cd35b83eb52985bfcb657e858f025a9
2024-05-30 21:38:32 +00:00
QyriadandGerrit Code Review 260db1ea64 Merge "build: fix static aws-cpp-sdk" into main 2024-05-30 20:53:49 +00:00
jadeandGerrit Code Review c161687b5f Merge "tests/filetransfer: reënable on Darwin" into main 2024-05-30 14:58:23 +00:00
jadeandGerrit Code Review 18aa3e1d57 Merge "Remove 100s of CPU time (10%) from build times (1465s -> 1302s)" into main 2024-05-30 14:57:37 +00:00
jadeandGerrit Code Review 53d40888ff Merge "unix-domain-socket.cc: add comment explaining why bindConnectProcHelper" into main 2024-05-30 14:56:57 +00:00
Maximilian BoschandGerrit Code Review 3d78b4847e Merge "release-notes: add missing credits/category to consistent-nix-build entry" into main 2024-05-30 14:38:47 +00:00
Maximilian BoschandRaito Bezarius 6abac7aacc release-notes: add missing credits/category to consistent-nix-build entry
Change-Id: I737422a2ff9d66be30cc432f8c1ddba9b1e71f4f
2024-05-30 14:05:12 +00:00
K900 da95bf8c82 libstore/filetransfer: remove debug print
foo.

Change-Id: I7d7db22f68046d2ecf3b594b4ee6fd9c9dac4be1
2024-05-30 16:42:45 +03:00
Qyriad 6475793678 build: fix static aws-cpp-sdk
Change-Id: I310830951106f194f6960a6b2d52b5081a7f6156
2024-05-30 00:40:25 -06:00
Jade Lovelace 7575db522e Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.

Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1465.3 s
  Codegen & opts (backend):   1110.9 s

<snip>

**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: serialise.hh strings.hh
  6x: <direct include>
  6x: archive.hh serialise.hh strings.hh
  ...

173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
  14x: store-api.hh nar-info.hh hash.hh
  11x: <direct include>
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh
  6x: fetchers.hh hash.hh
  ...

169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-derived-path.hh
  1x: installable-value.hh
  ...

159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
  19x: <direct include>
  14x: store-api.hh nar-info.hh hash.hh serialise.hh
  11x: serialise.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
  ...

156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
  37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
  19x: file-system.hh
  11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
  ...

132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

  done in 0.6s.
```

After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
  done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

<snip>

**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
  36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
  12x: globals.hh config.hh experimental-features.hh
  11x: file-system.hh file-descriptor.hh
  6x: <direct include>
  6x: serialise.hh strings.hh
  6x: fetchers.hh hash.hh serialise.hh strings.hh
  ...

154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
  40x: command.hh
  5x: command-installable-value.hh
  3x: installable-flake.hh
  2x: <direct include>
  2x: installable-attr-path.hh

153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
  55x: <direct include>
  39x: command.hh installable-value.hh installables.hh
  7x: libexpr.hh
  4x: local-store.hh
  4x: command-installable-value.hh installable-value.hh installables.hh
  3x: binary-cache-store.hh
  ...

149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh
  11x: file-system.hh
  10x: globals.hh
  6x: fetchers.hh
  6x: serialise.hh strings.hh error.hh
  5x: archive.hh
  ...

144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
  40x: command.hh installable-value.hh
  5x: command-installable-value.hh installable-value.hh
  3x: installable-flake.hh installable-value.hh
  2x: <direct include>
  1x: installable-value.hh
  1x: installable-derived-path.hh
  ...

136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
  42x: <direct include>
  7x: command-installable-value.hh
  2x: installable-attr-path.hh

133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
  36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
  16x: gtest.h
  11x: file-system.hh types.hh ref.hh
  10x: globals.hh types.hh ref.hh
  10x: json.hpp
  6x: serialise.hh
  ...

89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
  37x: command.hh installable-value.hh installables.hh
  25x: store-api.hh realisation.hh
  7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
  6x: libexpr.hh value.hh context.hh
  6x: shared.hh
  ...

87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
  35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
  12x: regex sstream istream
  10x: file-system.hh types.hh ref.hh memory unique_ptr.h
  10x: gtest.h memory unique_ptr.h
  10x: globals.hh types.hh ref.hh memory unique_ptr.h
  6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
  ...

85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
  37x: command.hh installable-value.hh installables.hh derived-path.hh
  20x: globals.hh
  20x: logging.hh
  16x: store-api.hh logging.hh
  6x: <direct include>
  6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
  ...

  done in 0.5s.
```

Change-Id: I27f0a2d566db17832cd9be935f12efe7f95b92d0
2024-05-29 22:16:15 -07:00
Jade Lovelace 285bc67318 tests/filetransfer: reënable on Darwin
Since we put __darwinAllowLocalNetworking in our derivation in
I752b81c85ebeaab4e582ac01c239d69d65580f37, this stuff will just work
fine. I checked our derivation works on the darwin community builder.

Change-Id: I40e3a801d6bb38efede79af4aded65c1e1f57cec
2024-05-29 20:41:22 -07:00
Jade Lovelace 26b3a1b9ce unix-domain-socket.cc: add comment explaining why bindConnectProcHelper
We reviewed this code a while ago, and we neglected to get a comment in
saying why it's Like This at the time. Let's fix that, since it is code
that looks very absurd at first glance.

Change-Id: Ib67b49605ef9ef1c84ecda1db16be74fc9105398
2024-05-29 19:50:04 -07:00
stuebinm d8e452a91b document context-dependent keywords
Documents some of the weirdness of __curPos and the or keyword.
This does not fit well into any existing section for either of
them, though the use of or as a quasi-operator is mentioned in
the section on operators.

Addresses https://git.lix.systems/lix-project/lix/issues/353

Change-Id: I7c906c8368843dca6944e8b22573b6d201cd9a76
2024-05-28 23:44:55 +02:00
161 changed files with 7152 additions and 6944 deletions
-18
View File
@@ -1,18 +0,0 @@
# Pull requests concerning the listed files will automatically invite the respective maintainers as reviewers.
# This file is not used for denoting any kind of ownership, but is merely a tool for handling notifications.
#
# Merge permissions are required for maintaining an entry in this file.
# For documentation on this mechanism, see https://help.github.com/articles/about-codeowners/
# Default reviewers if nothing else matches
* @edolstra
# This file
.github/CODEOWNERS @edolstra
# Public documentation
/doc @fricklerhandwerk
*.md @fricklerhandwerk
# Libstore layer
/src/libstore @thufschmitt
+1 -1
View File
@@ -9,7 +9,7 @@ assignees: ''
## Platform
<!-- select the platform on which you tried to install Nix -->
<!-- select the platform on which you tried to install Lix -->
- [ ] Linux: <!-- state your distribution, e.g. Arch Linux, Ubuntu, ... -->
- [ ] macOS
@@ -19,9 +19,10 @@ assignees: ''
<!-- make sure this issue is not redundant or obsolete -->
- [ ] checked [latest Nix manual] \([source])
- [ ] checked [open documentation issues and pull requests] for possible duplicates
- [ ] checked [latest Lix manual] \([source]\)
- [ ] checked [documentation issues] and [recent documentation changes] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/src
[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation
[latest Nix manual]: https://docs.lix.systems/manual/lix/nightly
[source]: https://git.lix.systems/lix-project/lix/src/main/doc/manual/src
[documentation issues]: https://git.lix.systems/lix-project/lix/issues?labels=151&state=all
[recent documentation changes]: https://gerrit.lix.systems/q/p:lix+path:%22%5Edoc/manual/.*%22
-35
View File
@@ -1,35 +0,0 @@
# Stale bot information
- Thanks for your contribution!
- To remove the stale label, just leave a new comment.
- _How to find the right people to ping?_ &rarr; [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
## Suggestions for PRs
1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else.
2. If it is unfinished but you plan to finish it, please mark it as a draft.
3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work.
4. To get things rolling again, rebase the PR against the target branch and address valid comments.
5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604).
6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions.
## Suggestions for issues
1. If it is resolved (either for you personally, or in general), please consider closing it.
2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough.
3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/).
4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude!
**Memorandum on closing issues**
Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.
## Useful GitHub search queries
- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+)
- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22)
- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+)
- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+)
- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
-6
View File
@@ -1,6 +0,0 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
-23
View File
@@ -1,23 +0,0 @@
"documentation":
- doc/manual/*
- src/nix/**/*.md
"store":
- src/libstore/store-api.*
- src/libstore/*-store.*
"fetching":
- src/libfetchers/**/*
"repl":
- src/libcmd/repl.*
- src/nix/repl.*
"new-cli":
- src/nix/**/*
"with-tests":
# Unit tests
- src/*/tests/**/*
# Functional and integration tests
- tests/functional/**/*
-9
View File
@@ -1,9 +0,0 @@
# Configuration for probot-stale - https://github.com/probot/stale
daysUntilStale: 180
daysUntilClose: false
exemptLabels:
- "critical"
- "never-stale"
staleLabel: "stale"
markComment: false
closeComment: false
+1 -129
View File
@@ -1,128 +1,5 @@
Makefile.config
perl/Makefile.config
# /
/aclocal.m4
/autom4te.cache
/precompiled-headers.h.gch
/config.*
/configure
/stamp-h1
/svn-revision
/libtool
/config
# /doc/manual/
/doc/manual/*.1
/doc/manual/*.5
/doc/manual/*.8
/doc/manual/generated/*
/doc/manual/nix.json
/doc/manual/conf-file.json
/doc/manual/language.json
/doc/manual/xp-features.json
/doc/manual/src/command-ref/experimental-features-shortlist.md
/doc/manual/src/contributing/experimental-feature-descriptions.md
/doc/manual/src/release-notes/rl-next-generated.md
# /scripts/
/scripts/nix-profile.sh
/scripts/nix-profile-daemon.sh
/scripts/nix-profile.fish
/scripts/nix-profile-daemon.fish
# /src/libexpr/
/src/libexpr/lexer-tab.cc
/src/libexpr/lexer-tab.hh
/src/libexpr/parser-tab.cc
/src/libexpr/parser-tab.hh
/src/libexpr/parser-tab.output
/src/libexpr/nix.tbl
/src/libexpr/tests
/tests/unit/libexpr/libnixexpr-tests
# /src/libstore/
*.gen.*
/src/libstore/tests
/tests/unit/libstore/libnixstore-tests
# /src/libutil/
/src/libutil/tests
/tests/unit/libutil/libnixutil-tests
/src/nix/nix
/src/nix/doc
# /src/nix-env/
/src/nix-env/nix-env
# /src/nix-instantiate/
/src/nix-instantiate/nix-instantiate
# /src/nix-store/
/src/nix-store/nix-store
/src/nix-prefetch-url/nix-prefetch-url
/src/nix-collect-garbage/nix-collect-garbage
# /src/nix-channel/
/src/nix-channel/nix-channel
# /src/nix-build/
/src/nix-build/nix-build
/src/nix-copy-closure/nix-copy-closure
/src/error-demo/error-demo
/src/build-remote/build-remote
# /tests/functional/
/tests/functional/test-tmp
/tests/functional/common/vars-and-functions.sh
/tests/functional/result*
/tests/functional/restricted-innocent
/tests/functional/shell
/tests/functional/shell.drv
/tests/functional/config.nix
/tests/functional/ca/config.nix
/tests/functional/dyn-drv/config.nix
/tests/functional/repl-result-out
/tests/functional/debugger-test-out
/tests/functional/test-libstoreconsumer/test-libstoreconsumer
# /tests/functional/lang/
/tests/functional/lang/*.out
/tests/functional/lang/*.out.xml
/tests/functional/lang/*.err
/tests/functional/lang/*.ast
/perl/lib/Nix/Config.pm
/perl/lib/Nix/Store.cc
/misc/systemd/nix-daemon.service
/misc/systemd/nix-daemon.socket
/misc/systemd/nix-daemon.conf
/misc/upstart/nix-daemon.conf
/src/resolve-system-dependencies/resolve-system-dependencies
outputs/
*.a
*.o
*.o.tmp
*.so
*.dylib
*.dll
*.exe
*.dep
*~
*.pc
*.plist
# GNU Global
GPATH
GRTAGS
@@ -132,17 +9,11 @@ GTAGS
# ccls
/.ccls-cache
# auto-generated compilation database
compile_commands.json
nix-rust/target
result
result-*
.vscode/
.direnv/
.envrc.local
# clangd and possibly more
.cache/
@@ -157,3 +28,4 @@ buildtime.bin
# We generate this with a Nix shell hook
/.pre-commit-config.yaml
/.nocontribmsg
/release
-1
View File
@@ -1 +0,0 @@
2.90.0
+2 -2
View File
@@ -6,14 +6,14 @@ Read more about us at https://lix.systems.
## Installation
On Linux and macOS the easiest way to install Nix is to run the following shell command
On Linux and macOS the easiest way to install Lix is to run the following shell command
(as a user other than root):
```console
$ curl -sSf -L https://install.lix.systems/lix | sh -s -- install
```
For systems that **already have Nix installed**, such as NixOS systems, read our [install page](https://lix.systems/install)
For systems that **already have a Nix implementation installed**, such as NixOS systems, read our [install page](https://lix.systems/install)
## Building And Developing
+17 -62
View File
@@ -1,56 +1,8 @@
diff --git a/darwin_stop_world.c b/darwin_stop_world.c
index 0468aaec..b348d869 100644
--- a/darwin_stop_world.c
+++ b/darwin_stop_world.c
@@ -356,6 +356,7 @@ GC_INNER void GC_push_all_stacks(void)
int nthreads = 0;
word total_size = 0;
mach_msg_type_number_t listcount = (mach_msg_type_number_t)THREAD_TABLE_SZ;
+ size_t stack_limit;
if (!EXPECT(GC_thr_initialized, TRUE))
GC_thr_init();
@@ -411,6 +412,19 @@ GC_INNER void GC_push_all_stacks(void)
GC_push_all_stack_sections(lo, hi, p->traced_stack_sect);
}
if (altstack_lo) {
+ // When a thread goes into a coroutine, we lose its original sp until
+ // control flow returns to the thread.
+ // While in the coroutine, the sp points outside the thread stack,
+ // so we can detect this and push the entire thread stack instead,
+ // as an approximation.
+ // We assume that the coroutine has similarly added its entire stack.
+ // This could be made accurate by cooperating with the application
+ // via new functions and/or callbacks.
+ stack_limit = pthread_get_stacksize_np(p->id);
+ if (altstack_lo >= altstack_hi || altstack_lo < altstack_hi - stack_limit) { // sp outside stack
+ altstack_lo = altstack_hi - stack_limit;
+ }
+
total_size += altstack_hi - altstack_lo;
GC_push_all_stack(altstack_lo, altstack_hi);
}
diff --git a/include/gc.h b/include/gc.h
index edab6c22..f2c61282 100644
--- a/include/gc.h
+++ b/include/gc.h
@@ -2172,6 +2172,11 @@ GC_API void GC_CALL GC_win32_free_heap(void);
(*GC_amiga_allocwrapper_do)(a,GC_malloc_atomic_ignore_off_page)
#endif /* _AMIGA && !GC_AMIGA_MAKINGLIB */
+#if !__APPLE__
+/* Patch doesn't work on apple */
+#define NIX_BOEHM_PATCH_VERSION 1
+#endif
+
#ifdef __cplusplus
} /* extern "C" */
#endif
diff --git a/pthread_stop_world.c b/pthread_stop_world.c
index b5d71e62..aed7b0bf 100644
index 2b45489..0e6d8ef 100644
--- a/pthread_stop_world.c
+++ b/pthread_stop_world.c
@@ -768,6 +768,8 @@ STATIC void GC_restart_handler(int sig)
@@ -776,6 +776,8 @@ STATIC void GC_restart_handler(int sig)
/* world is stopped. Should not fail if it isn't. */
GC_INNER void GC_push_all_stacks(void)
{
@@ -59,20 +11,23 @@ index b5d71e62..aed7b0bf 100644
GC_bool found_me = FALSE;
size_t nthreads = 0;
int i;
@@ -851,6 +853,37 @@ GC_INNER void GC_push_all_stacks(void)
hi = p->altstack + p->altstack_size;
@@ -868,6 +870,40 @@ GC_INNER void GC_push_all_stacks(void)
hi = p->altstack + p->altstack_size;
# endif
/* FIXME: Need to scan the normal stack too, but how ? */
/* FIXME: Assume stack grows down */
+ } else {
+#ifdef HAVE_PTHREAD_ATTR_GET_NP
+ if (!pthread_attr_init(&pattr)
+ || !pthread_attr_get_np(p->id, &pattr))
+#else /* HAVE_PTHREAD_GETATTR_NP */
+ if (pthread_getattr_np(p->id, &pattr))
+#endif
+ {
+ #ifdef HAVE_PTHREAD_ATTR_GET_NP
+ if (pthread_attr_init(&pattr) != 0) {
+ ABORT("GC_push_all_stacks: pthread_attr_init failed!");
+ }
+ if (pthread_attr_get_np(p->id, &pattr) != 0) {
+ ABORT("GC_push_all_stacks: pthread_attr_get_np failed!");
+ }
+ #else
+ if (pthread_getattr_np(p->id, &pattr)) {
+ ABORT("GC_push_all_stacks: pthread_getattr_np failed!");
+ }
+ #endif
+ if (pthread_attr_getstacksize(&pattr, &stack_limit)) {
+ ABORT("GC_push_all_stacks: pthread_attr_getstacksize failed!");
+ }
@@ -95,5 +50,5 @@ index b5d71e62..aed7b0bf 100644
+ #error "STACK_GROWS_UP not supported in boost_coroutine2 (as of june 2021), so we don't support it in Nix."
+ #endif
}
GC_push_all_stack_sections(lo, hi, traced_stack_sect);
# ifdef STACK_GROWS_UP
# ifdef STACKPTR_CORRECTOR_AVAILABLE
if (GC_sp_corrector != 0)
-12
View File
@@ -1,12 +0,0 @@
diff --git a/include/gc_allocator.h b/include/gc_allocator.h
index 597c7f13..587286be 100644
--- a/include/gc_allocator.h
+++ b/include/gc_allocator.h
@@ -312,6 +312,7 @@ public:
template<>
class traceable_allocator<void> {
+public:
typedef size_t size_type;
typedef ptrdiff_t difference_type;
typedef void* pointer;
@@ -1,11 +1,13 @@
---
synopsis: Clang build timing analysis
cls: 587
category: Development
---
We now have Clang build profiling available, which generates Chrome
tracing files for each compilation unit. To enable it, run `meson configure
build -Dprofile-build=enabled` then rerun the compilation.
build -Dprofile-build=enabled` in a Clang stdenv (`nix develop
.#native-clangStdenvPackages`) then rerun the compilation.
If you want to make the build go faster, do a clang build with meson, then run
`maintainers/buildtime_report.sh build`, then contemplate how to improve the
@@ -13,3 +15,8 @@ build time.
You can also look at individual object files' traces in
<https://ui.perfetto.dev>.
See [the wiki page][improving-build-times-wiki] for more details on how to do
this.
[improving-build-times-wiki]: https://wiki.lix.systems/link/8#bkmrk-page-title
@@ -1,5 +1,8 @@
---
synopsis: Show all FOD errors with `nix build --keep-going`
credits: [ma27]
category: Improvements
cls: [1108]
---
`nix build --keep-going` now behaves consistently with `nix-build --keep-going`. This means
+15
View File
@@ -0,0 +1,15 @@
---
synopsis: Lix is built with meson
# and many more
cls: [580, 627, 628, 707, 711, 712, 719]
credits: [Qyriad, horrors, jade, 9999years, winter]
category: Packaging
---
Lix is built exclusively with the meson build system thanks to a huge team-wide
effort, and the legacy `make`/`autoconf` based build system has been removed
altogether. This improves maintainability of Lix, enables things like saving
20% of compile times with precompiled headers, and generally makes the build
less able to produce obscure incremental compilation bugs.
Non-Nix-based downstream packaging needs rewriting accordingly.
+1 -1
View File
@@ -137,7 +137,7 @@ You can also build Lix for one of the [supported platforms](#platforms).
Lix can be built for various platforms, as specified in [`flake.nix`]:
[`flake.nix`]: https://github.com/nixos/nix/blob/master/flake.nix
[`flake.nix`]: https://git.lix.systems/lix-project/lix/src/branch/main/flake.nix
- `x86_64-linux`
- `x86_64-darwin`
+184 -42
View File
@@ -12,44 +12,51 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
> An example of some files, demonstrating much of what is described below
>
> ```
> src
> ├── libexpr
> │ ├── value/context.hh
> │ ├── value/context.cc
> │ │
> │ …
> └── tests
> │ ├── value/context.hh
> │ ├── value/context.cc
> │ │
> │ …
>
> ├── unit-test-data
> │ ├── libstore
> │ │ ├── worker-protocol/content-address.bin
> │ │ …
> │ …
>
> ├── src
> │   ├── libexpr
> │   │   ├── …
> │   │   ├── value
> │   │   │   ├── context.cc
> │   │   │   └── context.hh
> │ … …
> ├── tests
> │  
> │   └── unit
> │   ├── libcmd
> │   │   └── args.cc
> │   ├── libexpr
> │   │   ├── …
> │   │   └── value
> │   │   ├── context.cc
> │   │   └── print.cc
> │   ├── libexpr-support
> │   │   └── tests
> │   │   ├── libexpr.hh
> │   │   └── value
> │   │   ├── context.cc
> │   │   └── context.hh
> │   ├── libstore
> │   │   ├── common-protocol.cc
> │   │   ├── data
> │   │   │   ├── libstore
> │   │   │   │   ├── common-protocol
> │   │   │   │   │   ├── content-address.bin
> │   │   │   │   │   ├── drv-output.bin
> … … … … … …
> ```
<!-- FIXME(Lix): this might get renamed to liblixexpr, etc? -->
The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
The unit tests for each Lix library (`libnixexpr`, `libnixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
The data is in `unit-test-data`, with one subdir per library, with the same name as where the code goes.
For example, `libnixstore` code is in `src/libstore`, and its test data is in `unit-test-data/libstore`.
The path to the `unit-test-data` directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
> **Note**
> Due to the way googletest works, downstream unit test executables will actually include and re-run upstream library tests.
> Therefore it is important that the same value for `_NIX_TEST_UNIT_DATA` be used with the tests for each library.
> That is why we have the test data nested within a single `unit-test-data` directory.
The data is in `tests/unit/LIBNAME/data/LIBNAME`, with one subdir per library, with the same name as where the code goes.
For example, `liblixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
The path to the unit test data directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
### Running tests
You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test.
To get a list of tests, use `meson test -C build --list`.
To get a list of tests, use `meson test -C build --list` (or `just test --list` for short).
For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test).
@@ -74,20 +81,27 @@ See [below](#characterization-testing-1) for a broader discussion of characteriz
Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used.
For example:
```shell-session
$ _NIX_TEST_ACCEPT=1 make libstore-tests-exe_RUN
$ _NIX_TEST_ACCEPT=1 just test --suite check libstore-unit-tests
...
[ SKIPPED ] WorkerProtoTest.string_read
[ SKIPPED ] WorkerProtoTest.string_write
[ SKIPPED ] WorkerProtoTest.storePath_read
[ SKIPPED ] WorkerProtoTest.storePath_write
../tests/unit/libstore/common-protocol.cc:27: Skipped
Cannot read golden master because another test is also updating it
../tests/unit/libstore/common-protocol.cc:62: Skipped
Updating golden master
../tests/unit/libstore/common-protocol.cc:27: Skipped
Cannot read golden master because another test is also updating it
../tests/unit/libstore/common-protocol.cc:62: Skipped
Updating golden master
...
```
will regenerate the "golden master" expected result for the `libnixstore` characterization tests.
will regenerate the "golden master" expected result for the `liblixstore` characterization tests.
The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
## Functional tests
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`.
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`.
Each test is a bash script.
### Running the whole test suite
@@ -185,7 +199,7 @@ edit it like so:
```
<div class="warning">
FIXME(meson): the command here may be incorrect for meson.
FIXME(meson): the command here is incorrect for meson and this whole functionality may need rebuilding.
</div>
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
@@ -209,15 +223,11 @@ This technique is to include the exact output/behavior of a former version of Ni
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
<div class="warning">
FIXME(meson): this is incorrect for meson. `_NIX_TEST_ACCEPT=1` is still valid but the test invocation needs to change.
</div>
It is frequently useful to regenerate the expected output.
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
For example:
```bash
_NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test
_NIX_TEST_ACCEPT=1 just test --suite installcheck -v functional-lang
```
An interesting situation to document is the case when these tests are "overfitted".
@@ -320,3 +330,135 @@ solved this need?
~~>
-->
## Magic environment variables
FIXME: maybe this section should be moved elsewhere or turned partially into user docs, but I just need a complete index for now.
I actually want to ban people calling getenv without writing documentation, and produce a comprehensive list of env-vars used by Lix and enforce it.
This is a non-exhaustive list of almost all environment variables, magic or not, accepted or used by various parts of the test suite as well as Lix itself.
Please add more if you find them.
I looked for these in the testsuite with the following bad regexes:
```
rg '(?:[^A-Za-z]|^)(_[A-Z][^-\[ }/:");$(]+)' -r '$1' --no-filename --only-matching tests | sort -u > vars.txt
rg '\$\{?([A-Z][^-\[ }/:");]+)' -r '$1' --no-filename --only-matching tests | sort -u > vars.txt
```
I grepped `src/` for `get[eE]nv\("` to find the mentions in Lix code.
### Used by Lix testing support code
- `_NIX_TEST_ACCEPT` (optional) - Writes out the result of a characterization test as the new expected value.
**Expected value**: 1
- `_NIX_TEST_UNIT_DATA` - The path to the directory for the data for a given unit test suite.
**Expected value**: `tests/unit/libstore/data/libstore` or similar
### Used by Lix
- `_NIX_FORCE_HTTP` - Forces file URIs to be treated as remote ones.
Used by `src/libfetchers/git.cc`, `src/libstore/http-binary-cache-store.cc`,
`src/libstore/local-binary-cache-store.cc`. Seems to be for forcing Git
clones of `git+file://` URLs, making the HTTP binary
cache store accept `file://` URLs (presumably passing them to curl?), and
unknown reasons for the local binary cache.
FIXME(jade): is this obscuring a bug in https://git.lix.systems/lix-project/lix/issues/200?
**Expected value**: 1
- `NIX_ATTRS_SH_FILE`, `NIX_ATTRS_JSON_FILE` (output) - Set by Lix builders; see
`structuredAttrs` documentation.
- `NIX_BIN_DIR`, `NIX_STORE_DIR` (or its inconsistently-used old alias `NIX_STORE`), `NIX_DATA_DIR`,
`NIX_LOG_DIR`, `NIX_LOG_DIR`, `NIX_STATE_DIR`, `NIX_CONF_DIR` -
Overrides compile-time configuration of various locations used by Lix. See `src/libstore/globals.cc`.
**Expected value**: a directory
- `NIX_DAEMON_SOCKET_PATH` (optional) - Overrides the daemon socket path from `$NIX_STATE_DIR/daemon-socket/socket`.
**Expected value**: path to a socket
- `NIX_LOG_FD` (output) - An FD number for logs in `internal-json` format to be sent to.
Used for, mostly, "setPhase" in nixpkgs setup.sh, but can also be creatively used to print verbose log messages from derivations.
**Provided value**: number corresponding to an FD in the builder
- `NIX_PATH` - Search path for `<whatever>`. Documented elsewhere in the manual.
**Expected value**: `:` separated list of things that are not necessarily pointing to filesystem paths
- `NIX_REMOTE` - The default value of the Lix setting `store`.
**Expected value**: "daemon", usually. Could be "auto" or any other value acceptable in `store`.
- `NIX_BUILD_SHELL` - Documented elsewhere; the shell to invoke with `nix-shell` but not `nix develop`/`nix shell`.
The latter ignoring it altogether seems like a bug.
**Expected value**: the path to an executable shell
- `PRINT_PATH` - Undocumented. Used by `nix-prefetch-url` as an alternative form of `--print-path`. Why???
- `_NIX_IN_TEST` - If present with any value, makes `fetchClosure` accept file URLs in addition to HTTP ones. Why is this not `_NIX_FORCE_HTTP`??
Not used anywhere else.
- `NIX_ALLOW_EVAL` - Used by eval-cache tests to block evaluation if set to `0`.
**Expected value**: 1 or 0
- `EDITOR` - Used by `editorFor()`, which has some extremely sketchy editor-detection code for jumping to line numbers.
- `LISTEN_FDS` and `LISTEN_PID` - Used for systemd socket activation using the systemd socket activation protocol.
- `NIX_PAGER` (alternatively, `PAGER`) - Used to select a pager for Lix output. Why does this not use libutil `getEnv()`?
- `LESS` (output) - Sets the pager settings for `less` when invoked by Lix.
- `NIX_IGNORE_SYMLINK_STORE` - When set, Lix allows the store to be a symlink. Why do we support this?
Apparently [someone was using it enough to fix it](https://github.com/NixOS/nix/pull/4038).
- `NIX_SSL_CERT_FILE` (alternatively, `SSL_CERT_FILE`) - Used to set CA certificates for libcurl.
**Expected value**: "/etc/ssl/certs/ca-certificates.crt" or similar
- `NIX_REMOTE_SYSTEMS` - Used to set `builders`. Can we please deprecate this?
- `NIX_USER_CONF_FILES` - `:` separated list of config files to load before
`/nix/nix.conf` under each of `XDG_CONFIG_DIRS`.
- `NIX_CONFIG` - Newline separated configuration to load into Lix.
- `NIX_GET_COMPLETIONS` - Returns completions.
Unsure of the exact format, someone should document it; either way my shell never had any completions.
**Expected value**: number of completions to return.
- `IN_SYSTEMD` - Used to switch the logging format so that systemd gets the correct log levels. I think.
- `NIX_HELD_LOCKS` - Not used, what is this for?? We should surely remove it right after searching github?
- `GC_INITIAL_HEAP_SIZE` - Used to set the initial heap size, processed by boehmgc.
- `NIX_COUNT_CALLS` - Documented elsewhere; prints call counts for profiling purposes.
- `NIX_SHOW_STATS` - Documented elsewhere; prints various evaluation statistics like function calls, gc info, and similar.
- `NIX_SHOW_STATS_PATH` - Writes those statistics into a file at the given path instead of stdout. Undocumented.
- `NIX_SHOW_SYMBOLS` - Dumps the symbol table into the show-stats json output.
- `TERM` - If `dumb` or unset, disables ANSI colour output.
- `NO_COLOR`, `NOCOLOR` - Disables ANSI colour output.
- `_NIX_DEVELOPER_SHOW_UNKNOWN_LOCATIONS` - Highlights unknown locations in errors.
- `NIX_PROFILE` - Selects which profile `nix-env` will operate on. Documented elsewhere.
- `NIX_SSHOPTS` - Options passed to `ssh(1)` when using a ssh remote store.
Incorrectly documented on `nix-copy-closure` which is *surely* not the only place they are used??
- `_NIX_TEST_NO_LSOF` - Used on non-Linux, non-macOS platforms to disable using `lsof` when finding gc roots.
Since https://git.lix.systems/lix-project/lix/issues/156 was fixed, this should probably just be removed as it was a bad workaround for a macOS issue.
- `_NIX_TEST_GC_SYNC_1` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
- `_NIX_TEST_GC_SYNC_2` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
- `_NIX_TEST_FREE_SPACE_FILE` - Path to a file containing a decimal number with the free space that the GC is to believe it has.
- Various XDG vars
- `NIX_DEBUG_SQLITE_TRACES` - Dump all sqlite queries to the log at `notice` level.
- `_NIX_TEST_NO_SANDBOX` - Disables actually setting up the sandbox on macOS while leaving other logic the same. Unused on other platforms.
- `_NIX_TRACE_BUILT_OUTPUTS` - Dumps all the derivation paths alongside their outputs as lines into a file of the given name.
### Used by the functional test framework
- `NIX_DAEMON_PACKAGE` - Runs the test suite against an alternate Nix daemon with the current client.
**Expected value**: something like `/nix/store/...-nix-2.18.2`
- `NIX_CLIENT_PACKAGE` - Runs the test suite against an alternate Nix client with the current daemon.
**Expected value**: something like `/nix/store/...-nix-2.18.2`
- `NIX_TESTS_CA_BY_DEFAULT` - Pass `__contentAddressed`, `outputHashMode` and `outputHashAlgo` to builds of some input-addressed derivations in the test suite.
**Expected value**: 1
- `TEST_DATA` - Not an environment variable! This is used in repl characterization tests to refer to `tests/functional/repl_characterization/data`.
More specifically, that path is replaced with the string `$TEST_DATA` in output for reproducibility.
- `TEST_HOME` (output) - Set to the temporary directory that is set as `$HOME` inside the tests, underneath `$TEST_ROOT`.
- `TEST_ROOT` (output) - Set to the temporary directory that is created for each test to mess with.
- `_NIX_TEST_DAEMON_PID` (output) - Used to track the daemon pid to be able to kill it.
**Provided value**: Daemon pid as a base-10 integer, e.g. 2345
@@ -1,64 +1,62 @@
# Using Lix within Docker
Currently the Lix project doesn't ship docker images. However, we have the infrastructure to do it, it's just not yet been done. See https://git.lix.systems/lix-project/lix/issues/252
<!--
Lix is available on the following two container registries:
- [ghcr.io/lix-project/lix](https://ghcr.io/lix-project/lix)
- [git.lix.systems/lix-project/lix](https://git.lix.systems/lix-project/-/packages/container/lix)
To run the latest stable release of Lix with Docker run the following command:
```console
$ docker run -ti nixos/nix
Unable to find image 'nixos/nix:latest' locally
latest: Pulling from nixos/nix
5843afab3874: Pull complete
b52bf13f109c: Pull complete
1e2415612aa3: Pull complete
Digest: sha256:27f6e7f60227e959ee7ece361f75d4844a40e1cc6878b6868fe30140420031ff
Status: Downloaded newer image for nixos/nix:latest
35ca4ada6e96:/# nix --version
nix (Nix) 2.3.12
35ca4ada6e96:/# exit
~ » sudo podman run -it ghcr.io/lix-project/lix:latest
Trying to pull ghcr.io/lix-project/lix:latest...
bash-5.2# nix --version
nix (Lix, like Nix) 2.90.0
```
# What is included in Lix's Docker image?
The official Docker image is created using `pkgs.dockerTools.buildLayeredImage`
The official Docker image is created using [nix2container]
(and not with `Dockerfile` as it is usual with Docker images). You can still
base your custom Docker image on it as you would do with any other Docker
image.
The Docker image is also not based on any other image and includes minimal set
of runtime dependencies that are required to use Lix:
[nix2container]: https://github.com/nlewo/nix2container
- pkgs.nix
- pkgs.bashInteractive
- pkgs.coreutils-full
- pkgs.gnutar
- pkgs.gzip
- pkgs.gnugrep
- pkgs.which
- pkgs.curl
- pkgs.less
- pkgs.wget
- pkgs.man
- pkgs.cacert.out
- pkgs.findutils
The Docker image is also not based on any other image and includes the nixpkgs
that Lix was built with along with a minimal set of tools in the system profile:
- bashInteractive
- cacert.out
- coreutils-full
- curl
- findutils
- gitMinimal
- gnugrep
- gnutar
- gzip
- iana-etc
- less
- libxml2
- lix
- man
- openssh
- sqlite
- wget
- which
# Docker image with the latest development version of Lix
To get the latest image that was built by [Hydra](https://hydra.nixos.org) run
the following command:
FIXME: There are not currently images of development versions of Lix. Tracking issue: https://git.lix.systems/lix-project/lix/issues/381
You can build a Docker image from source yourself and copy it to either:
Podman: `nix run '.#dockerImage.copyTo' containers-storage:lix`
Docker: `nix run '.#dockerImage.copyToDockerDaemon'`
Then:
```console
$ curl -L https://hydra.nixos.org/job/nix/master/dockerImage.x86_64-linux/latest/download/1 | docker load
$ docker run -ti nix:2.5pre20211105
$ docker run -ti lix
```
You can also build a Docker image from source yourself:
```console
$ nix build ./\#hydraJobs.dockerImage.x86_64-linux
$ docker load -i ./result/image.tar.gz
$ docker run -ti nix:2.5pre20211105
```
-->
+114 -56
View File
@@ -1,32 +1,121 @@
# Multi-User Mode
To allow a Nix store to be shared safely among multiple users, it is
important that users are not able to run builders that modify the Nix
store or database in arbitrary ways, or that interfere with builds
started by other users. If they could do so, they could install a Trojan
horse in some package and compromise the accounts of other users.
To allow a Nix store to be shared safely among multiple users, it is important that users cannot meaningfully influence the execution of derivation builds such that they could inject malicious code into them without changing their (either input- or output- addressed) hash.
If they could do so, they could install a Trojan horse in some package and compromise the accounts of other users.
To prevent this, the Nix store and database are owned by some privileged
user (usually `root`) and builders are executed under special user
accounts (usually named `nixbld1`, `nixbld2`, etc.). When a unprivileged
user runs a Nix command, actions that operate on the Nix store (such as
builds) are forwarded to a *Nix daemon* running under the owner of the
Nix store/database that performs the operation.
To prevent this, the Nix store and database are owned by some privileged user (usually `root`) and builders are executed under unprivileged system user accounts (usually named `nixbld1`, `nixbld2`, etc.).
When an unprivileged user runs a Nix command, actions that operate on the Nix store (such as builds) are forwarded to a *Nix daemon* running under the owner of the Nix store/database that performs the operation.
> **Note**
>
> Multi-user mode has one important limitation: only root and a set of
> trusted users specified in `nix.conf` can specify arbitrary binary
> caches. So while unprivileged users may install packages from
> arbitrary Nix expressions, they may not get pre-built binaries.
The buried lede in the above sentence is that *currently*, even in multi-user mode using a daemon, if executing as the user that owns the store, Lix directly manipulates the store unless `--store daemon` is specified.
[We intend to change this in the future][multi-user-should-not-be-root].
## Setting up the build users
<div class="warning">
The Lix team considers the goal of the sandbox to be primarily for preventing reproducibility mistakes, and does not consider multi-user mode to be a strong security boundary between users.
Do not evaluate or build untrusted, potentially-malicious, Nix language code on machines that you care deeply about maintaining user isolation on.
Although we would consider any sandbox escapes to be serious security bugs and we intend to fix them, we are not confident enough in the daemon's security to call the daemon a security boundary.
</div>
[multi-user-should-not-be-root]: https://git.lix.systems/lix-project/lix/issues/18
## Trust model
There are two categories of users of the Lix daemon: trusted users and untrusted users.
The Lix daemon only allows connections from users that are either trusted users, or are specified in, or are members of groups specified in, [`allowed-users`](../command-ref/conf-file.md#conf-allowed-users) in `nix.conf`.
Trusted users are users and users of groups specified in [`trusted-users`](../command-ref/conf-file.md#conf-trusted-users) in `nix.conf`.
All users of the Lix daemon may do the following to bring things into the Nix store:
- Users may load derivations and output-addressed files into the store with `nix-store --add` or through Nix language code.
- Users may locally build derivations, either of the output-addressed or input-addressed variety, creating output paths.
Note that [fixed-output derivations only consider name and hash](https://github.com/NixOS/nix/issues/969), so it is possible to write a fixed-output derivation for something important with a bogus hash and have it resolve to something else already built in the store.
On systems with `sandbox` enabled (default on Linux; [not *yet* on macOS][sandbox-enable-macos]), derivations are either:
- Input-addressed, so they are run in the sandbox with no network access, with the following exceptions:
- The (poorly named, since it is not *just* about chroot) property `__noChroot` is set on the derivation and `sandbox` is set to `relaxed`.
- On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value. This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
- Output-addressed, so they are run with network access but their result must match an expected hash.
Trusted users may set any setting, including `sandbox = false`, so the sandbox state can be different at runtime from what is described in `nix.conf` for builds invoked with such settings.
- Users may copy appropriately-signed derivation outputs into the store.
By default, any paths *copied into a store* (such as by substitution) must have signatures from [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys) unless they are [output-addressed](../glossary.md#gloss-output-addressed-store-object).
Unsigned paths may be copied into a store if [`require-sigs`](../command-ref/conf-file.md#conf-require-sigs) is disabled in the daemon's configuration (not default), or if the client is a trusted user and passed `--no-check-sigs` to `nix copy`.
- Users may request that the daemon substitutes appropriately-signed derivation outputs from a binary cache in the daemon's [`substituters`](../command-ref/conf-file.md#conf-substituters) list.
Untrusted clients may also specify additional values for `substituters` (via e.g. `--extra-substituters` on a Nix command) that are listed in [`trusted-substituters`](../command-ref/conf-file.md#conf-trusted-substituters).
A client could in principle substitute such paths itself then copy them to the daemon (see clause above) if they are appropriately signed but are *not* from a trusted substituter, however this is not implemented in the current Lix client to our knowledge, at the time of writing.
This probably means that `trusted-substituters` is a redundant setting except insofar as such substitution would have to be done on the client rather than as root on the daemon; and it is highly defensible to not allow random usage of our HTTP client running as root.
[sandbox-enable-macos]: https://git.lix.systems/lix-project/lix/issues/386
### The Lix daemon as a security non-boundary
The Lix team and wider community does not consider the Lix daemon to be a *security boundary* against malicious Nix language code.
Although we do our best to make it secure, we do not recommend sharing a Lix daemon with potentially malicious users.
That means that public continuous integration (CI) builds of untrusted Nix code should not share builders with CI that writes into a cache used by trusted infrastructure.
For example, [hydra.nixos.org], which is the builder for [cache.nixos.org], does not execute untrusted Nix language code; a separate system, [ofborg] is used for CI of nixpkgs pull requests.
The build output of pull request CI is never pushed to [cache.nixos.org], and those systems are considered entirely untrusted.
This is because, among other things, the Lix sandbox is *more* susceptible to kernel exploits than Docker, which, unlike Lix, blocks nested user namespaces via `seccomp` in its default policy, and there have been many kernel bugs only exposed to unprivileged users via user namespaces allowing otherwise-root-only system calls.
In general, the Lix sandbox is set up to be relatively unrestricted while maintaining its goals of building useful, reproducible software; security is not its primary goal.
The Lix sandbox is a custom *non-rootless* Linux container implementation that has not been audited to nearly the same degree as Docker and similar systems.
Also, the Lix daemon is a complex and historied C++ executable running as root with very little privilege separation.
All of this means that a security hole in the Lix daemon gives immediate root access.
Systems like Docker (especially non-rootless Docker) should *themselves* probably not be used in a multi-tenant manner with mutually distrusting tenants, but the Lix daemon *especially* should not be used as such as of this writing.
The primary purpose of the sandbox is to strongly encourage packages to be reproducible, a goal which it is generally quite successful at.
[hydra.nixos.org]: https://hydra.nixos.org
[ofborg]: https://github.com/NixOS/ofborg
[cache.nixos.org]: https://cache.nixos.org
### Trusted users
Trusted users are permitted to set any setting and bypass security restrictions on the daemon.
They are currently in widespread use for a couple of reasons such as remote builds (which we [intend to fix](https://git.lix.systems/lix-project/lix/issues/171)).
Trusted users are effectively root on Nix daemons running as root (the default configuration) for *at least* the following reasons, and should be thus thought of as equivalent to passwordless sudo.
This is not a comprehensive list.
- They may copy an unsigned malicious built output into the store for `systemd` or anything else that will run as root, then when the system is upgraded, that path will be used from the local store rather than substituted.
- They may set the following settings that are commands the daemon will run as root:
- `build-hook`
- `diff-hook`
- `pre-build-hook`
- `post-build-hook`
- They may set `build-users-group`.
In particular, they may set it to empty string, which runs builds as root with respect to the rest of the system (!!).
We, too, [think that is absurd and intend to not accept such a configuration](https://git.lix.systems/lix-project/lix/issues/242).
It is then simply an exercise to the reader to find a daemon that does `SCM_CREDENTIALS` over a `unix(7)` socket and lets you run commands as root, and mount it into the sandbox with `extra-sandbox-paths`.
At the very least, the Lix daemon itself (since `root` is a trusted user by default) and probably `systemd` qualify for this.
- They may set the `builders` list, which will have ssh run as root.
We aren't sure if there is a way to abuse this for command execution but it's plausible.
Note that setting `accept-flake-config` allows arbitrary Nix flakes to set Nix settings in the `nixConfig` stanza.
Do not set this setting or pass `--accept-flake-config` while executing untrusted Nix language code as a trusted user for the reasons above!
## Build users
The *build users* are the special UIDs under which builds are performed.
They should all be members of the *build users group* `nixbld`. This
group should have no other members. The build users should not be
members of any other group. On Linux, you can create the group and users
as follows:
A build user is selected for a build by looking in the group specified by [`build-users-group`](../command-ref/conf-file.md#conf-build-users-group), by default, `nixbld`, then a member of that group not currently executing a build is selected for the build.
The build users should not be members of any other group.
There can never be more concurrent builds than the number of build users, unless using [`auto-allocate-uids`](../command-ref/conf-file.md#conf-auto-allocate-uids) ([tracking issue][auto-allocate-uids-issue]).
[auto-allocate-uids-issue]: https://git.lix.systems/lix-project/lix/issues/387
If, for some reason, you need to create such users manually, the following command will create 10 build users on Linux:
```console
$ groupadd -r nixbld
@@ -35,43 +124,12 @@ $ for n in $(seq 1 10); do useradd -c "Nix build user $n" \
nixbld$n; done
```
This creates 10 build users. There can never be more concurrent builds
than the number of build users, so you may want to increase this if you
expect to do many builds at the same time.
## Running the daemon
The [Nix daemon](../command-ref/nix-daemon.md) should be started as
follows (as `root`):
The [Nix daemon](../command-ref/nix-daemon.md) can be started manually as follows (as `root`):
```console
$ nix-daemon
# nix-daemon
```
Youll want to put that line somewhere in your systems boot scripts.
To let unprivileged users use the daemon, they should set the
[`NIX_REMOTE` environment variable](../command-ref/env-common.md) to
`daemon`. So you should put a line like
```console
export NIX_REMOTE=daemon
```
into the users login scripts.
## Restricting access
To limit which users can perform Nix operations, you can use the
permissions on the directory `/nix/var/nix/daemon-socket`. For instance,
if you want to restrict the use of Nix to the members of a group called
`nix-users`, do
```console
$ chgrp nix-users /nix/var/nix/daemon-socket
$ chmod ug=rwx,o= /nix/var/nix/daemon-socket
```
This way, users who are not in the `nix-users` group cannot connect to
the Unix domain socket `/nix/var/nix/daemon-socket/socket`, so they
cannot perform Nix operations.
In standard installations of Lix, the daemon is started by a `systemd` unit (Linux) or `launchd` service (macOS).
+5 -6
View File
@@ -7,9 +7,8 @@ management operations. All other users can then use the installed
packages, but they cannot perform package management operations
themselves.
Alternatively, you can configure Lix in “multi-user mode”. In this
model, all users can perform package management operations — for
instance, every user can install software without requiring root
privileges. Lix ensures that this is secure. For instance, its not
possible for one user to overwrite a package used by another user with a
Trojan horse.
Alternatively, you can configure Lix in “multi-user mode”. In this model, all users can perform package management operations — for instance, every user can install software for themselves without requiring root privileges.
Lix does its best to ensure that this is secure.
For instance, it would be considered a serious security bug for one untrusted user to be able to overwrite a package used by another user with a Trojan horse.
Nevertheless, the Lix team does not consider multi-user mode a strong security boundary, and does not recommend running untrusted user-supplied Nix language code on privileged machines, even if it is secure to the best of our knowledge at any moment in time.
@@ -1,13 +1,13 @@
# Obtaining the Source
The most recent sources of Nix can be obtained from its [Git
repository](https://github.com/NixOS/nix). For example, the following
The most recent sources of Lix can be obtained from its [Git
repository](https://git.lix.systems/lix-project/lix). For example, the following
command will check out the latest revision into a directory called
`nix`:
```console
$ git clone https://github.com/NixOS/nix
$ git clone https://git.lix.systems/lix-project/lix
```
Likewise, specific releases can be obtained from the
[tags](https://github.com/NixOS/nix/tags) of the repository.
[tags](https://git.lix.systems/lix-project/lix/tags) of the repository.
@@ -7,3 +7,14 @@ These constants are built into the Nix language evaluator:
{{#include @generated@/language/builtin-constants.md}}
</dl>
## Things which might be mistaken for constants
<dl>
<dt><code>__curPos</code></dt>
<dd>
This is not a constant but a [context-dependent keyword](@docroot@/language/constructs.md#keywords-__curPos)
</dd>
</dl>
+67
View File
@@ -380,3 +380,70 @@ let a = 1; in let a = 2; in let a = 3; in let a = 4; in ...
Comments can be single-line, started with a `#` character, or
inline/multi-line, enclosed within `/* ... */`.
## Context-dependent keywords
<dl>
<dt id="keywords-__curPos">
<a href="#keywords-__curPos"><code>__curPos</code></a>
</dt>
<dd>
A quasi-constant which will be replaced with an attribute set describing
the location where `__curPos` was used, with attributes `file`, `line`,
and `column`. For example, `import ./file.nix` will result in
```nix
{
column = 1;
file = "/path/to/some/file.nix";
line = 1;
}
```
assuming `file.nix` contains nothing but `__curPos`.
In context without a source file (such as `nix-repl`), it will always
be replaced with `null`:
```nix-repl
nix-repl> __curPos
null
```
While it may vaguely look like a builtin, this is a very different beast
that is handled directly by the parser. It thus cannot be shadowed,
bound to a different name, and is also not available under
[`builtins`](@docroot@/language/builtin-constants.md#builtins-builtins).
```nix-repl
nix-repl> let __curPos = "no"; in __curPos
null
```
Despite this `__curPos`, much like `or`, may still be used as an identifier,
it is only treated specially when it appears as an unqualified name:
```nix-repl
nix-repl> { __curPos = 1; }.__curPos
1
```
</dd>
<dt id="keywords-or">
<a href="#keywords-or"><code>or</code></a>
</dt>
<dd>
`or` is used in [Attribute selection](@docroot@/language/operators.html#attribute-selection),
where it is a keyword.
However, it is not a keyword in some other contexts, and can be used as
a binding name in attribute sets, let-bindings, non-initial function
application position, and as a label in attribute paths.
Its use as anything other than a keyword is discouraged.
</dd>
</dl>
+1 -1
View File
@@ -22,7 +22,7 @@ to subsequent chapters.
$ curl -sSf -L https://install.lix.systems/lix | sh -s -- install
```
For systems that **already have Nix installed**, such as NixOS systems, read our [install page](https://lix.systems/install)
For systems that **already have a Nix implementation installed**, such as NixOS systems, read our [install page](https://lix.systems/install)
The install script will use `sudo`, so make sure you have sufficient rights.
+1 -1
View File
@@ -229,7 +229,7 @@ This release has the following new features:
<https://cache.nixos.org/> or <ssh://machine>. The following store
types are supported:
- `LocalStore` (stori URI `local` or an absolute path) and the
- `LocalStore` (store URI `local` or an absolute path) and the
misnamed `RemoteStore` (`daemon`) provide access to a local Nix
store, the latter via the Nix daemon. You can use `auto` or the
empty string to auto-select a local or daemon store depending on
+151 -65
View File
@@ -1,7 +1,10 @@
{
pkgs ? import <nixpkgs> { },
# Git commit ID, if available
lixRevision ? null,
nix2container,
lib ? pkgs.lib,
name ? "nix",
name ? "lix",
tag ? "latest",
bundleNixpkgs ? true,
channelName ? "nixpkgs",
@@ -12,26 +15,51 @@
flake-registry ? null,
}:
let
layerContents = with pkgs; [
# pulls in glibc and openssl, about 60MB
{ contents = [ coreutils-full ]; }
# some stuff that is low in the closure graph and small ish, mostly to make
# incremental lix updates cheaper
{
contents = [
curl
libxml2
sqlite
];
}
# 50MB of git
{ contents = [ gitMinimal ]; }
# 144MB of nixpkgs
{
contents = [ channel ];
inProfile = false;
}
];
# These packages are left to be auto layered by nix2container, since it is
# less critical that they get layered sensibly and they tend to not be deps
# of anything in particular
autoLayered = with pkgs; [
bashInteractive
gnutar
gzip
gnugrep
which
less
wget
man
cacert.out
findutils
iana-etc
openssh
nix
];
defaultPkgs =
with pkgs;
[
nix
bashInteractive
coreutils-full
gnutar
gzip
gnugrep
which
curl
less
wget
man
cacert.out
findutils
iana-etc
git
openssh
]
lib.lists.flatten (
map (x: if !(x ? inProfile) || x.inProfile then x.contents else [ ]) layerContents
)
++ autoLayered
++ extraPkgs;
users =
@@ -139,16 +167,17 @@ let
))
+ "\n";
nixpkgs = pkgs.path;
channel = pkgs.runCommand "channel-nixpkgs" { } ''
mkdir $out
${lib.optionalString bundleNixpkgs ''
ln -s ${nixpkgs} $out/nixpkgs
echo "[]" > $out/manifest.nix
''}
'';
baseSystem =
let
nixpkgs = pkgs.path;
channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
mkdir $out
if [ "$bundleNixpkgs" ]; then
ln -s ${nixpkgs} $out/nixpkgs
echo "[]" > $out/manifest.nix
fi
'';
rootEnv = pkgs.buildPackages.buildEnv {
name = "root-profile-env";
paths = defaultPkgs;
@@ -187,7 +216,7 @@ let
profile = pkgs.buildPackages.runCommand "user-environment" { } ''
mkdir $out
cp -a ${rootEnv}/* $out/
ln -s ${manifest} $out/manifest.nix
ln -sf ${manifest} $out/manifest.nix
'';
flake-registry-path =
if (flake-registry == null) then
@@ -236,6 +265,7 @@ let
ln -s /nix/var/nix/profiles/share $out/usr/
mkdir -p $out/nix/var/nix/gcroots
ln -s /nix/var/nix/profiles $out/nix/var/nix/gcroots/profiles
mkdir $out/tmp
@@ -248,14 +278,14 @@ let
mkdir -p $out/nix/var/nix/profiles/per-user/root
ln -s ${profile} $out/nix/var/nix/profiles/default-1-link
ln -s $out/nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
ln -s /nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
ln -s /nix/var/nix/profiles/default $out/root/.nix-profile
ln -s ${channel} $out/nix/var/nix/profiles/per-user/root/channels-1-link
ln -s $out/nix/var/nix/profiles/per-user/root/channels-1-link $out/nix/var/nix/profiles/per-user/root/channels
ln -s /nix/var/nix/profiles/per-user/root/channels-1-link $out/nix/var/nix/profiles/per-user/root/channels
mkdir -p $out/root/.nix-defexpr
ln -s $out/nix/var/nix/profiles/per-user/root/channels $out/root/.nix-defexpr/channels
ln -s /nix/var/nix/profiles/per-user/root/channels $out/root/.nix-defexpr/channels
echo "${channelURL} ${channelName}" > $out/root/.nix-channels
mkdir -p $out/bin $out/usr/bin
@@ -273,43 +303,99 @@ let
ln -s $globalFlakeRegistryPath $out/nix/var/nix/gcroots/auto/$rootName
'')
);
in
pkgs.dockerTools.buildLayeredImageWithNixDb {
inherit name tag maxLayers;
layers = builtins.foldl' (
layersList: el:
let
layer = nix2container.buildLayer {
deps = el.contents;
layers = layersList;
};
in
layersList ++ [ layer ]
) [ ] layerContents;
contents = [ baseSystem ];
image = nix2container.buildImage {
extraCommands = ''
rm -rf nix-support
ln -s /nix/var/nix/profiles nix/var/nix/gcroots/profiles
'';
fakeRootCommands = ''
chmod 1777 tmp
chmod 1777 var/tmp
'';
inherit name tag maxLayers;
config = {
Cmd = [ "/root/.nix-profile/bin/bash" ];
Env = [
"USER=root"
"PATH=${
lib.concatStringsSep ":" [
"/root/.nix-profile/bin"
"/nix/var/nix/profiles/default/bin"
"/nix/var/nix/profiles/default/sbin"
]
}"
"MANPATH=${
lib.concatStringsSep ":" [
"/root/.nix-profile/share/man"
"/nix/var/nix/profiles/default/share/man"
]
}"
"SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"GIT_SSL_CAINFO=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"NIX_PATH=/nix/var/nix/profiles/per-user/root/channels:/root/.nix-defexpr/channels"
inherit layers;
copyToRoot = [ baseSystem ];
initializeNixDatabase = true;
perms = [
{
path = baseSystem;
regex = "(/var)?/tmp";
mode = "1777";
}
];
config = {
Cmd = [ "/root/.nix-profile/bin/bash" ];
Env = [
"USER=root"
"PATH=${
lib.concatStringsSep ":" [
"/root/.nix-profile/bin"
"/nix/var/nix/profiles/default/bin"
"/nix/var/nix/profiles/default/sbin"
]
}"
"MANPATH=${
lib.concatStringsSep ":" [
"/root/.nix-profile/share/man"
"/nix/var/nix/profiles/default/share/man"
]
}"
"SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"GIT_SSL_CAINFO=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
"NIX_PATH=/nix/var/nix/profiles/per-user/root/channels:/root/.nix-defexpr/channels"
];
Labels = {
"org.opencontainers.image.title" = "Lix";
"org.opencontainers.image.source" = "https://git.lix.systems/lix-project/lix";
"org.opencontainers.image.vendor" = "Lix project";
"org.opencontainers.image.version" = pkgs.nix.version;
"org.opencontainers.image.description" = "Minimal Lix container image, with some batteries included.";
} // lib.optionalAttrs (lixRevision != null) { "org.opencontainers.image.revision" = lixRevision; };
};
meta = {
description = "Docker image for Lix. This is built with nix2container; see that project's README for details";
longDescription = ''
Docker image for Lix, built with nix2container.
To copy it to your docker daemon, nix run .#dockerImage.copyToDockerDaemon
To copy it to podman, nix run .#dockerImage.copyTo containers-storage:lix
'';
};
};
in
image
// {
# We don't ship the tarball as the default output because it is a strange thing to want imo
tarball =
pkgs.buildPackages.runCommand "docker-image-tarball-${pkgs.nix.version}"
{
nativeBuildInputs = [ pkgs.buildPackages.bubblewrap ];
meta.description = "Docker image tarball with Lix for ${pkgs.system}";
}
''
mkdir -p $out/nix-support
image=$out/image.tar
# bwrap for foolish temp dir selection code that forces /var/tmp:
# https://github.com/containers/skopeo.git/blob/60ee543f7f7c242f46cc3a7541d9ac8ab1c89168/vendor/github.com/containers/image/v5/internal/tmpdir/tmpdir.go#L15-L18
mkdir -p $TMPDIR/fake-var/tmp
args=(--unshare-user --bind "$TMPDIR/fake-var" /var)
for dir in /*; do
args+=(--dev-bind "/$dir" "/$dir")
done
bwrap ''${args[@]} -- ${lib.getExe image.copyTo} docker-archive:$image
gzip $image
echo "file binary-dist $image" >> $out/nix-support/hydra-build-products
'';
}
Generated
+21 -4
View File
@@ -16,18 +16,34 @@
"type": "github"
}
},
"nix2container": {
"flake": false,
"locked": {
"lastModified": 1712990762,
"narHash": "sha256-hO9W3w7NcnYeX8u8cleHiSpK2YJo7ecarFTUlbybl7k=",
"owner": "nlewo",
"repo": "nix2container",
"rev": "20aad300c925639d5d6cbe30013c8357ce9f2a2e",
"type": "github"
},
"original": {
"owner": "nlewo",
"repo": "nix2container",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1715123187,
"narHash": "sha256-0czuu757t53lK6uWeo1a5/jJbCd9t4sOtLDFpts60DM=",
"lastModified": 1718111384,
"narHash": "sha256-7tSst0S5FOmcgvNtfy6cjZX5w8CabCVAfAeCkhY4OVg=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "0c592f9a288bdf764b6f24c757277c0e49757a46",
"rev": "a508a44af0c1b1b57785c34d8b54783536273eeb",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-23.11-small",
"ref": "nixos-24.05-small",
"repo": "nixpkgs",
"type": "github"
}
@@ -67,6 +83,7 @@
"root": {
"inputs": {
"flake-compat": "flake-compat",
"nix2container": "nix2container",
"nixpkgs": "nixpkgs",
"nixpkgs-regression": "nixpkgs-regression",
"pre-commit-hooks": "pre-commit-hooks"
+50 -38
View File
@@ -2,12 +2,16 @@
description = "The purely functional package manager";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11-small";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05-small";
nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
pre-commit-hooks = {
url = "github:cachix/git-hooks.nix";
flake = false;
};
nix2container = {
url = "github:nlewo/nix2container";
flake = false;
};
flake-compat = {
url = "github:edolstra/flake-compat";
flake = false;
@@ -20,6 +24,7 @@
nixpkgs,
nixpkgs-regression,
pre-commit-hooks,
nix2container,
flake-compat,
}:
@@ -54,12 +59,11 @@
(Run `touch .nocontribmsg` to hide this message.)
'';
officialRelease = false;
officialRelease = true;
# Set to true to build the release notes for the next release.
buildUnreleasedNotes = true;
version = lib.fileContents ./.version + versionSuffix;
versionSuffix =
if officialRelease then
""
@@ -83,10 +87,11 @@
crossSystems = [
"armv6l-linux"
"armv7l-linux"
# FIXME: doesn't evaluate, plausibly fixed in >=24.05, so recheck when
# we update to 24.05
# "x86_64-freebsd13"
"x86_64-netbsd"
"riscv64-linux"
# FIXME: still broken in 24.05: fails to build rustc(??) due to missing -lstdc++ dep
# "x86_64-freebsd"
# FIXME: broken dev shell due to python
# "x86_64-netbsd"
];
stdenvs = [
@@ -131,13 +136,11 @@
{
system = crossSystem;
}
// lib.optionalAttrs (crossSystem == "x86_64-freebsd13") { useLLVM = true; };
// lib.optionalAttrs (crossSystem == "x86_64-freebsd") { useLLVM = true; };
overlays = [
(overlayFor (p: p.${stdenv}))
(final: prev: { nixfmt = final.callPackage ./nix-support/nixfmt.nix { }; })
];
config.permittedInsecurePackages = [ "nix-2.13.6" ];
};
stdenvs = forAllStdenvs (make-pkgs null);
native = stdenvs.stdenvPackages;
@@ -149,9 +152,6 @@
}
);
binaryTarball =
nix: pkgs: pkgs.callPackage ./nix-support/binary-tarball.nix { inherit nix version; };
overlayFor =
getStdenv: final: prev:
let
@@ -164,7 +164,6 @@
nixUnstable = prev.nixUnstable;
check-headers = final.buildPackages.callPackage ./maintainers/check-headers.nix { };
clangbuildanalyzer = final.buildPackages.callPackage ./misc/clangbuildanalyzer.nix { };
default-busybox-sandbox-shell = final.busybox.override {
useMusl = true;
@@ -191,7 +190,7 @@
};
nix = final.callPackage ./package.nix {
inherit versionSuffix;
inherit versionSuffix officialRelease;
stdenv = currentStdenv;
busybox-sandbox-shell = final.busybox-sandbox-shell or final.default-busybox-sandbox-shell;
};
@@ -209,10 +208,14 @@
overlays.default = overlayFor (p: p.stdenv);
hydraJobs = {
# Binary package for various platforms.
build = forAllSystems (system: self.packages.${system}.nix);
devShell = forAllSystems (system: {
default = self.devShells.${system}.default;
clang = self.devShells.${system}.native-clangStdenvPackages;
});
rl-next = forAllSystems (
system:
let
@@ -227,20 +230,17 @@
in
{
user = rl-next-check "rl-next" ./doc/manual/rl-next;
dev = rl-next-check "rl-next-dev" ./doc/manual/rl-next-dev;
}
);
# Perl bindings for various platforms.
perlBindings = forAllSystems (system: nixpkgsFor.${system}.native.nix.perl-bindings);
perlBindings = forAllSystems (system: nixpkgsFor.${system}.native.nix.passthru.perl-bindings);
# Binary tarball for various platforms, containing a Nix store
# with the closure of 'nix' package.
binaryTarball = forAllSystems (
system: binaryTarball nixpkgsFor.${system}.native.nix nixpkgsFor.${system}.native
);
binaryTarball = forAllSystems (system: nixpkgsFor.${system}.native.nix.passthru.binaryTarball);
# docker image with Nix inside
# docker image with Lix inside
dockerImage = lib.genAttrs linux64BitSystems (system: self.packages.${system}.dockerImage);
# API docs for Nix's unstable internal C++ interfaces.
@@ -281,9 +281,20 @@
nixpkgsLibTests = forAllSystems (
system:
import (nixpkgs + "/lib/tests/release.nix") {
let
inherit (self.packages.${system}) nix;
pkgs = nixpkgsFor.${system}.native;
nixVersions = [ self.packages.${system}.nix ];
testWithNix = import (nixpkgs + "/lib/tests/test-with-nix.nix") { inherit pkgs lib nix; };
in
pkgs.symlinkJoin {
name = "nixpkgs-lib-tests";
paths =
[ testWithNix ]
# FIXME: This is disabled on darwin due to a nixpkgs bug https://github.com/NixOS/nixpkgs/issues/319147
# After that is fixed, it should be restored to use lib/tests/release.nix as before, rather than this reimplementation.
++ lib.optionals pkgs.stdenv.isLinux [
(import (nixpkgs + "/pkgs/test/release") { inherit pkgs lib nix; })
];
}
);
};
@@ -300,16 +311,23 @@
);
};
release-jobs = import ./releng/release-jobs.nix {
inherit (self) hydraJobs;
pkgs = nixpkgsFor.x86_64-linux.native;
};
# NOTE *do not* add fresh derivations to checks, always add them to
# hydraJobs first (so CI will pick them up) and only link them here
checks = forAvailableSystems (
system:
{
# devShells and packages already get checked by nix flake check, so
# this is just jobs that are special
binaryTarball = self.hydraJobs.binaryTarball.${system};
perlBindings = self.hydraJobs.perlBindings.${system};
nixpkgsLibTests = self.hydraJobs.tests.nixpkgsLibTests.${system};
rl-next = self.hydraJobs.rl-next.${system}.user;
rl-next-dev = self.hydraJobs.rl-next.${system}.dev;
# Will be empty attr set on i686-linux, and filtered out by forAvailableSystems.
pre-commit = self.hydraJobs.pre-commit.${system};
}
@@ -330,19 +348,13 @@
dockerImage =
let
pkgs = nixpkgsFor.${system}.native;
image = import ./docker.nix {
inherit pkgs;
tag = version;
};
nix2container' = import nix2container { inherit pkgs system; };
in
pkgs.runCommand "docker-image-tarball-${version}"
{ meta.description = "Docker image with Nix for ${system}"; }
''
mkdir -p $out/nix-support
image=$out/image.tar.gz
ln -s ${image} $image
echo "file binary-dist $image" >> $out/nix-support/hydra-build-products
'';
import ./docker.nix {
inherit pkgs;
nix2container = nix2container'.nix2container;
tag = pkgs.nix.version;
};
}
// builtins.listToAttrs (
map (crossSystem: {
@@ -365,7 +377,7 @@
pkgs: stdenv:
let
nix = pkgs.callPackage ./package.nix {
inherit stdenv versionSuffix;
inherit stdenv officialRelease versionSuffix;
busybox-sandbox-shell = pkgs.busybox-sandbox-shell or pkgs.default-busybox-sandbox;
internalApiDocs = true;
};
+2 -2
View File
@@ -1,6 +1,5 @@
from collections import defaultdict
import frontmatter
import sys
import pathlib
import textwrap
from typing import Any, Tuple
@@ -27,6 +26,7 @@ CATEGORIES = [
'Improvements',
'Fixes',
'Packaging',
'Development',
'Miscellany',
]
@@ -143,7 +143,7 @@ def run_on_dir(author_info: AuthorInfoDB, d):
for category in CATEGORIES:
if entries[category]:
print('\n#', category)
print('\n##', category)
do_category(author_info, entries[category])
def main():
+33 -25
View File
@@ -17,6 +17,19 @@
#
# Finally, src/nix/meson.build defines the Nix command itself, relying on all prior meson files.
#
# libstore, libexpr, and libfetchers have some special handling to make static builds work.
# Their use static constructors for dynamic registration of primops, store backends, etc
# gets borked during static link. We can't simply wholesale apply `link_whole :` either,
# because these libraries get linked multiple times since Lix's components are transitively
# dependent. So instead, each of those libraries have two dependency objects:
# liblix{store,expr,fetchers,util} and liblix{store,expr,fetchers,util}_mstatic ("maybe static").
# The _mstatic versions should be used in the `dependencies :` arguments to ALL EXECUTABLES
# but executables ONLY. When we are not building statically (default_library != 'static'),
# they are equivalent. When we are building statically, the _mstatic version will be
# `link_whole :` rather than `link_with :`.
# FIXME: This hack should be removed when https://git.lix.systems/lix-project/lix/issues/359
# is fixed.
#
# Unit tests are setup in tests/unit/meson.build, under the test suite "check".
#
# Functional tests are a bit more complicated. Generally they're defined in
@@ -26,7 +39,7 @@
# in the build directory.
project('lix', 'cpp',
version : run_command('bash', '-c', 'echo -n $(cat ./.version)$VERSION_SUFFIX', check : true).stdout().strip(),
version : run_command('bash', '-c', 'echo -n $(jq -r .version < ./version.json)$VERSION_SUFFIX', check : true).stdout().strip(),
default_options : [
'cpp_std=c++2a',
# TODO(Qyriad): increase the warning level
@@ -79,6 +92,8 @@ if not fs.is_absolute(sysconfdir)
sysconfdir = '/' / sysconfdir
endif
is_static = get_option('default_library') == 'static'
# All of this has to go before the rest of the dependency checking,
# so that internal-api-docs can be built with -Denable-build=false
@@ -114,6 +129,20 @@ endif
cxx = meson.get_compiler('cpp')
# clangd breaks when GCC is using precompiled headers lmao
# https://git.lix.systems/lix-project/lix/issues/374
should_pch = get_option('enable-pch-std')
summary('PCH C++ stdlib', should_pch, bool_yn : true)
if should_pch
# Unlike basically everything else that takes a file, Meson requires the arguments to
# cpp_pch : to be strings and doesn't accept files(). So absolute path it is.
cpp_pch = [meson.project_source_root() / 'src/pch/precompiled-headers.hh']
else
cpp_pch = []
endif
# Translate some historical and Mesony CPU names to Lixy CPU names.
# FIXME(Qyriad): the 32-bit x86 code is not tested right now, because cross compilation for Lix
# to those architectures is currently broken for other reasons, namely:
@@ -153,23 +182,18 @@ elif is_linux
# Clang sanitizers on Linux.
# FIXME(Qyriad): is that true?
endif
deps = [ ]
configdata = { }
#
# Dependencies
#
boehm = dependency('bdw-gc', required : get_option('gc'))
if boehm.found()
deps += boehm
endif
boehm = dependency('bdw-gc', required : get_option('gc'), version : '>=8.2.6')
configdata += {
'HAVE_BOEHMGC': boehm.found().to_int(),
}
boost = dependency('boost', required : true, modules : ['context', 'coroutine', 'container'])
deps += boost
# cpuid only makes sense on x86_64
cpuid_required = is_x64 ? get_option('cpuid') : false
@@ -177,7 +201,6 @@ cpuid = dependency('libcpuid', 'cpuid', required : cpuid_required)
configdata += {
'HAVE_LIBCPUID': cpuid.found().to_int(),
}
deps += cpuid
# seccomp only makes sense on Linux
seccomp_required = is_linux ? get_option('seccomp-sandboxing') : false
@@ -190,20 +213,17 @@ configdata += {
}
libarchive = dependency('libarchive', required : true)
deps += libarchive
brotli = [
dependency('libbrotlicommon', required : true),
dependency('libbrotlidec', required : true),
dependency('libbrotlienc', required : true),
]
deps += brotli
openssl = dependency('libcrypto', 'openssl', required : true)
deps += openssl
aws_sdk = dependency('aws-cpp-sdk-core', required : false)
aws_sdk_transfer = dependency('aws-cpp-sdk-transfer', required : aws_sdk.found())
aws_sdk_transfer = dependency('aws-cpp-sdk-transfer', required : aws_sdk.found(), fallback : ['aws_sdk', 'aws_cpp_sdk_transfer_dep'])
if aws_sdk.found()
# The AWS pkg-config adds -std=c++11.
# https://github.com/aws/aws-sdk-cpp/issues/2673
@@ -214,7 +234,6 @@ if aws_sdk.found()
links : true,
sources : true,
)
deps += aws_sdk
s = aws_sdk.version().split('.')
configdata += {
'AWS_VERSION_MAJOR': s[0].to_int(),
@@ -230,7 +249,7 @@ if aws_sdk.found()
)
endif
aws_s3 = dependency('aws-cpp-sdk-s3', required : false)
aws_s3 = dependency('aws-cpp-sdk-s3', required : aws_sdk.found(), fallback : ['aws_sdk', 'aws_cpp_sdk_s3_dep'])
if aws_s3.found()
# The AWS pkg-config adds -std=c++11.
# https://github.com/aws/aws-sdk-cpp/issues/2673
@@ -241,7 +260,6 @@ if aws_s3.found()
links : true,
sources : true,
)
deps += aws_s3
endif
configdata += {
@@ -249,26 +267,20 @@ configdata += {
}
sqlite = dependency('sqlite3', 'sqlite', version : '>=3.6.19', required : true)
deps += sqlite
sodium = dependency('libsodium', 'sodium', required : true)
deps += sodium
curl = dependency('libcurl', 'curl', required : true)
deps += curl
editline = dependency('libeditline', 'editline', version : '>=1.14', required : true)
deps += editline
lowdown = dependency('lowdown', version : '>=0.9.0', required : true)
deps += lowdown
# HACK(Qyriad): rapidcheck's pkg-config doesn't include the libs lol
# Note: technically we 'check' for rapidcheck twice, for the internal-api-docs handling above,
# but Meson will cache the result of the first one, and the required : arguments are different.
rapidcheck_meson = dependency('rapidcheck', required : enable_tests)
rapidcheck = declare_dependency(dependencies : rapidcheck_meson, link_args : ['-lrapidcheck'])
deps += rapidcheck
gtest = [
dependency('gtest', required : enable_tests),
@@ -276,13 +288,10 @@ gtest = [
dependency('gmock', required : enable_tests),
dependency('gmock_main', required : enable_tests),
]
deps += gtest
toml11 = dependency('toml11', version : '>=3.7.0', required : true, method : 'cmake')
deps += toml11
nlohmann_json = dependency('nlohmann_json', required : true)
deps += nlohmann_json
# lix-doc is a Rust project provided via buildInputs and unfortunately doesn't have any way to be detected.
# Just declare it manually to resolve this.
@@ -290,7 +299,6 @@ deps += nlohmann_json
# FIXME: build this with meson in the future after we drop Make (with which we
# *absolutely* are not going to make it work)
lix_doc = declare_dependency(link_args : [ '-llix_doc' ])
deps += lix_doc
#
# Build-time tools
+4
View File
@@ -64,3 +64,7 @@ option('internal-api-docs', type : 'feature', value : 'auto',
option('profile-dir', type : 'string', value : 'etc/profile.d',
description : 'the path to install shell profile files',
)
option('enable-pch-std', type : 'boolean', value : true,
description : 'whether to use precompiled headers for C++\'s standard library (breaks clangd if you\'re using GCC)',
)
-33
View File
@@ -1,33 +0,0 @@
# Upstreaming here, can be deleted once it's upstreamed:
# https://github.com/NixOS/nixpkgs/pull/297102
{
stdenv,
lib,
cmake,
fetchFromGitHub,
}:
stdenv.mkDerivation (finalAttrs: {
pname = "clangbuildanalyzer";
version = "1.5.0";
src = fetchFromGitHub {
owner = "aras-p";
repo = "ClangBuildAnalyzer";
rev = "v${finalAttrs.version}";
sha256 = "sha256-kmgdk634zM0W0OoRoP/RzepArSipa5bNqdVgdZO9gxo=";
};
nativeBuildInputs = [ cmake ];
meta = {
description = "Tool for analyzing Clang's -ftrace-time files";
homepage = "https://github.com/aras-p/ClangBuildAnalyzer";
maintainers = with lib.maintainers; [ lf- ];
license = lib.licenses.unlicense;
platforms = lib.platforms.unix;
# `long long int` != `size_t`
# There's no convenient lib.platforms.32bit or anything, but it's easy enough to do ourselves.
badPlatforms = lib.filter (plat: (lib.systems.elaborate plat).is32bit) lib.platforms.all;
mainProgram = "ClangBuildAnalyzer";
};
})
+6 -5
View File
@@ -63,7 +63,7 @@ pre-commit-run {
files = ''^doc/manual/(change-authors\.yml|rl-next(-dev)?)'';
pass_filenames = false;
entry = ''
${lib.getExe pkgs.build-release-notes} --change-authors doc/manual/change-authors.yml doc/manual/rl-next doc/manual/rl-next-dev
${lib.getExe pkgs.build-release-notes} --change-authors doc/manual/change-authors.yml doc/manual/rl-next
'';
};
change-authors-sorted = {
@@ -87,11 +87,12 @@ pre-commit-run {
"file"
"header"
];
# generated files; these will never actually be seen by this
# check, and are left here as documentation
excludes = [
"(parser|lexer)-tab\\.hh$"
"\\.gen\\.hh$"
''^src/pch/.*$''
# generated files; these will never actually be seen by this
# check, and are left here as documentation
''(parser|lexer)-tab\.hh$''
''\.gen\.hh$''
];
entry = lib.getExe pkgs.check-headers;
};
+3 -4
View File
@@ -3,7 +3,6 @@
cacert,
nix,
system,
version,
}:
let
installerClosureInfo = buildPackages.closureInfo {
@@ -13,12 +12,12 @@ let
];
};
meta.description = "Distribution-independent Nix bootstrap binaries for ${system}";
meta.description = "Distribution-independent Lix bootstrap binaries for ${system}";
in
buildPackages.runCommand "nix-binary-tarball-${version}" { inherit meta; } ''
buildPackages.runCommand "lix-binary-tarball-${nix.version}" { inherit meta; } ''
cp ${installerClosureInfo}/registration $TMPDIR/reginfo
dir=nix-${version}-${system}
dir=lix-${nix.version}-${system}
fn=$out/$dir.tar.xz
mkdir -p $out/nix-support
echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
+56 -23
View File
@@ -50,7 +50,7 @@
# internal fork of nix-doc providing :doc in the repl
lix-doc ? __forDefaults.lix-doc,
pname ? "nix",
pname ? "lix",
versionSuffix ? "",
officialRelease ? false,
# Set to true to build the release notes for the next release.
@@ -69,8 +69,6 @@
# `boehmgc-nix` then this will almost certainly have duplicate patches, which means
# the patches won't apply and we'll get a build failure.
./boehmgc-coroutine-sp-fallback.diff
# https://github.com/ivmai/bdwgc/pull/586
./boehmgc-traceable_allocator-public.diff
];
};
@@ -87,7 +85,8 @@ let
inherit (lib) fileset;
inherit (stdenv) hostPlatform buildPlatform;
version = lib.fileContents ./.version + versionSuffix;
versionJson = builtins.fromJSON (builtins.readFile ./version.json);
version = versionJson.version + versionSuffix;
aws-sdk-cpp-nix = aws-sdk-cpp.override {
apis = [
@@ -137,13 +136,14 @@ let
# that would interfere with repo semantics.
baseFiles = fileset.fileFilter (f: f.name != ".gitignore") ./.;
configureFiles = fileset.unions [ ./.version ];
configureFiles = fileset.unions [ ./version.json ];
topLevelBuildFiles = fileset.unions ([
./meson.build
./meson.options
./meson
./scripts/meson.build
./subprojects
]);
functionalTestFiles = fileset.unions [
@@ -168,7 +168,6 @@ stdenv.mkDerivation (finalAttrs: {
./boehmgc-coroutine-sp-fallback.diff
./doc
./misc
./precompiled-headers.h
./src
./COPYING
]
@@ -259,7 +258,7 @@ stdenv.mkDerivation (finalAttrs: {
++ lib.optional internalApiDocs rapidcheck
++ lib.optional hostPlatform.isx86_64 libcpuid
# There have been issues building these dependencies
++ lib.optional (hostPlatform == buildPlatform) aws-sdk-cpp-nix
++ lib.optional (hostPlatform.canExecute buildPlatform) aws-sdk-cpp-nix
++ lib.optionals (finalAttrs.dontBuild) maybePropagatedInputs;
checkInputs = [
@@ -377,25 +376,31 @@ stdenv.mkDerivation (finalAttrs: {
platforms = lib.platforms.unix;
};
passthru.perl-bindings = pkgs.callPackage ./perl { inherit fileset stdenv; };
# Export the patched version of boehmgc.
# flake.nix exports that into its overlay.
passthru = {
inherit (__forDefaults) boehmgc-nix editline-lix build-release-notes;
inherit officialRelease;
# The collection of dependency logic for this derivation is complicated enough that
# it's easier to parameterize the devShell off an already called package.nix.
mkDevShell =
{
mkShell,
just,
nixfmt,
glibcLocales,
pre-commit-checks,
bashInteractive,
clang-tools,
llvmPackages,
clangbuildanalyzer,
glibcLocales,
just,
llvmPackages,
nixfmt,
skopeo,
xonsh,
# Lix specific packages
pre-commit-checks,
contribNotice,
}:
let
@@ -403,10 +408,17 @@ stdenv.mkDerivation (finalAttrs: {
# Required to make non-NixOS Linux not complain about missing locale files during configure in a dev shell
LOCALE_ARCHIVE = "${lib.getLib pkgs.glibcLocales}/lib/locale/locale-archive";
};
# for some reason that seems accidental and was changed in
# NixOS 24.05-pre, clang-tools is pinned to LLVM 14 when
# default LLVM is newer.
clang-tools_llvm = clang-tools.override { inherit llvmPackages; };
pythonPackages = (
p: [
p.yapf
p.python-frontmatter
p.requests
p.xdg-base-dirs
(p.toPythonModule xonsh.passthru.unwrapped)
]
);
pythonEnv = python3.withPackages pythonPackages;
# pkgs.mkShell uses pkgs.stdenv by default, regardless of inputsFrom.
actualMkShell = mkShell.override { inherit stdenv; };
@@ -424,13 +436,25 @@ stdenv.mkDerivation (finalAttrs: {
# For Meson to find Boost.
env = finalAttrs.env;
# I guess this is necessary because mesonFlags to mkDerivation doesn't propagate in inputsFrom,
# which only propagates stuff set in hooks? idk.
inherit (finalAttrs) mesonFlags;
mesonFlags =
# I guess this is necessary because mesonFlags to mkDerivation doesn't propagate in inputsFrom,
# which only propagates stuff set in hooks? idk.
finalAttrs.mesonFlags
# Clangd breaks when GCC is using precompiled headers, so for the devshell specifically
# we make precompiled C++ stdlib conditional on using Clang.
# https://git.lix.systems/lix-project/lix/issues/374
++ [ (lib.mesonBool "enable-pch-std" stdenv.cc.isClang) ];
packages =
lib.optional (stdenv.cc.isClang && hostPlatform == buildPlatform) clang-tools_llvm
lib.optional (stdenv.cc.isClang && hostPlatform == buildPlatform) clang-tools
++ [
# Why are we providing a bashInteractive? Well, when you run
# `bash` from inside `nix develop`, say, because you are using it
# via direnv, you will by default get bash (unusable edition).
bashInteractive
pythonEnv
# docker image tool
skopeo
just
nixfmt
# Load-bearing order. Must come before clang-unwrapped below, but after clang_tools above.
@@ -448,7 +472,10 @@ stdenv.mkDerivation (finalAttrs: {
shellHook = ''
# don't re-run the hook in (other) nested nix-shells
function lixShellHook() {
if [[ $name != lix-shell-env ]]; then
# n.b. how the heck does this become -env-env? well, `nix develop` does it:
# https://git.lix.systems/lix-project/lix/src/commit/7575db522e9008685c4009423398f6900a16bcce/src/nix/develop.cc#L240-L241
# this is, of course, absurd.
if [[ $name != lix-shell-env && $name != lix-shell-env-env ]]; then
return;
fi
@@ -480,5 +507,11 @@ stdenv.mkDerivation (finalAttrs: {
'';
}
);
perl-bindings = pkgs.callPackage ./perl { inherit fileset stdenv; };
binaryTarball = pkgs.callPackage ./nix-support/binary-tarball.nix {
nix = finalAttrs.finalPackage;
};
};
})
+1 -1
View File
@@ -29,7 +29,7 @@ NEED_PROG(xz, xz)
AC_MSG_CHECKING([whether Perl is recent enough])
if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or die;'; then
AC_MSG_RESULT(no)
AC_MSG_ERROR([Your Perl version is too old. Nix requires Perl 5.8.0 or newer.])
AC_MSG_ERROR([Your Perl version is too old. Lix requires Perl 5.8.0 or newer.])
fi
AC_MSG_RESULT(yes)
+1 -1
View File
@@ -23,7 +23,7 @@ perl.pkgs.toPerlModule (
src = fileset.toSource {
root = ../.;
fileset = fileset.unions ([
../.version
../version.json
./lib
./meson.build
]);
+1 -1
View File
@@ -1,5 +1,5 @@
project('lix-perl', 'cpp',
version : run_command('bash', '-c', 'echo -n $(cat ../.version)$VERSION_SUFFIX', check : true).stdout().strip(),
version : run_command('bash', '-c', 'echo -n $(jq -r .version < ../version.json)$VERSION_SUFFIX', check : true).stdout().strip(),
default_options : [
'cpp_std=c++2a',
# TODO(Qyriad): increase the warning level
+132
View File
@@ -0,0 +1,132 @@
# Release engineering
This directory contains the release engineering scripts for Lix.
## Release process
### Prerequisites
* FIXME: validation via misc tests in nixpkgs, among other things? What other
validation do we need before we can actually release?
* Have a release post ready to go as a PR on the website repo.
* No [release-blocker bugs][release-blockers].
[release-blockers]: https://git.lix.systems/lix-project/lix/issues?q=&type=all&sort=&labels=145&state=open&milestone=0&project=0&assignee=0&poster=0
### Process
The following process can be done either against the staging environment or the
live environment.
For staging, the buckets are `staging-releases`, `staging-cache`, etc.
FIXME: obtainment of signing key for signing cache paths?
First, we prepare the release. `python -m releng prepare` is used for this.
* Gather everything in `doc/manual/rl-next` and put it in
`doc/manual/src/release-notes/rl-MAJOR.md`.
Then we tag the release with `python -m releng tag`:
* Git HEAD is detached.
* `officialRelease = true` is set in `flake.nix`, this is committed, and a
release is tagged.
* The tag is merged back into the last branch (either `main` for new releases
or `release-MAJOR` for maintenance releases) with `git merge -s ours VERSION`
creating a history link but ignoring the tree of the release tag.
* Git HEAD is once again detached onto the release tag.
Then, we build the release artifacts with `python -m releng build`:
* Source tarball is generated with `git archive`, then checksummed.
* Manifest for `nix upgrade-nix` is produced and put in `s3://releases` at
`/manifest.nix` and `/lix/lix-VERSION`.
* Release is built: `hydraJobs.binaryTarball` jobs are built, and joined into a
derivation that depends on all of them and adds checksum files. This and the
sources go into `s3://releases/lix/lix-VERSION`.
At this point we have a `release/artifacts` and `release/manual` directory
which are ready to publish, and tags ready for publication. No keys are
required to do this part.
Next, we do the publication with `python -m releng upload`:
* Artifacts for this release are uploaded:
* s3://releases/manifest.nix, changing the default version of Lix for
`nix upgrade-nix`.
* s3://releases/lix/lix-VERSION/ gets the following contents
* Binary tarballs
* Docs: `manual/` (FIXME: should we actually do this? what about putting it
on docs.lix.systems? I think doing both is correct, since the Web site
should not be an archive of random old manuals)
* Docs as tarball in addition to web.
* Source tarball
* Docker image (FIXME: upload to forgejo registry and github registry [in the future][upload-docker])
* s3://docs/manual/lix/MAJOR
* s3://docs/manual/lix/stable
* The tag is uploaded to the remote repo.
* **Manually** build the installer using the scripts in the installer repo and upload.
FIXME: This currently requires a local Apple Macintosh® aarch64 computer, but
we could possibly automate doing it from the aarch64-darwin builder.
* **Manually** Push the main/release branch directly to gerrit.
* If this is a new major release, branch-off to `release-MAJOR` and push *that* branch
directly to gerrit as well (FIXME: special creds for doing this as a service
account so we don't need to have the gerrit perms to shoot ourselves in the
foot by default because pushing to main is bad?).
FIXME: automate branch-off to `release-*` branch.
* **Manually** (FIXME?) switch back to the release branch, which now has the
correct revision.
* Post!!
* Merge release blog post to [lix-website].
* Toot about it! https://chaos.social/@lix_project
* Tweet about it! https://twitter.com/lixproject
[lix-website]: https://git.lix.systems/lix-project/lix-website
[upload-docker]: https://git.lix.systems/lix-project/lix/issues/252
### Installer
The installer is cross-built to several systems from a Mac using
`build-all.xsh` and `upload-to-lix.xsh` in the installer repo (FIXME: currently
at least; maybe this should be moved here?) .
It installs a binary tarball (FIXME: [it should be taught to substitute from
cache instead][installer-substitute])
from some URL; this is the `hydraJobs.binaryTarball`. The default URLs differ
by architecture and are [configured here][tarball-urls].
[installer-substitute]: https://git.lix.systems/lix-project/lix-installer/issues/13
[tarball-urls]: https://git.lix.systems/lix-project/lix-installer/src/commit/693592ed10d421a885bec0a9dd45e87ab87eb90a/src/settings.rs#L14-L28
## Infrastructure summary
* releases.lix.systems (`s3://releases`):
* Each release gets a directory: https://releases.lix.systems/lix/lix-2.90-beta.1
* Binary tarballs: `nix-2.90.0-beta.1-x86_64-linux.tar.xz`, from `hydraJobs.binaryTarball`
* Manifest: `manifest.nix`, an attrset of the store paths by architecture.
* Manifest for `nix upgrade-nix` to the latest release at `/manifest.nix`.
* cache.lix.systems (`s3://cache`):
* Receives all artifacts for released versions of Lix; is a plain HTTP binary cache.
* install.lix.systems (`s3://install`):
```
~ » aws s3 ls s3://install/lix/
PRE lix-2.90-beta.0/
PRE lix-2.90-beta.1/
PRE lix-2.90.0pre20240411/
PRE lix-2.90.0pre20240412/
2024-05-05 18:59:11 6707344 lix-installer-aarch64-darwin
2024-05-05 18:59:16 7479768 lix-installer-aarch64-linux
2024-05-05 18:59:14 7982208 lix-installer-x86_64-darwin
2024-05-05 18:59:17 8978352 lix-installer-x86_64-linux
~ » aws s3 ls s3://install/lix/lix-2.90-beta.1/
2024-05-05 18:59:01 6707344 lix-installer-aarch64-darwin
2024-05-05 18:59:06 7479768 lix-installer-aarch64-linux
2024-05-05 18:59:03 7982208 lix-installer-x86_64-darwin
2024-05-05 18:59:07 8978352 lix-installer-x86_64-linux
```
+79
View File
@@ -0,0 +1,79 @@
from xonsh.main import setup
setup()
del setup
import logging
import sys
import xonsh.base_shell
from . import environment
from . import create_release
from . import keys
from . import version
from . import cli
from . import docker
from . import docker_assemble
from . import gitutils
def setup_logging():
"""
Sets up logging to work properly. The following are intended to work:
- ipython/xonsh configuration files adding log handlers out of band
- Reloading the module in xonsh/ipython not causing Bonus Loggers (which is
why we check if there is already a handler. This also helps the previous
case)
- Importing the releng module from xonsh and poking at it interactively
"""
LEVELS = {
# Root logger must be DEBUG so that anything else can be DEBUG
None: logging.DEBUG,
# Everything in releng
__name__: logging.DEBUG,
# Log spam caused by prompt_toolkit
'asyncio': logging.INFO,
}
for name, level in LEVELS.items():
logger = logging.getLogger(name)
logger.setLevel(level)
root_logger = logging.getLogger()
fmt = logging.Formatter('{asctime} {levelname} {name}: {message}',
datefmt='%b %d %H:%M:%S',
style='{')
if not any(
isinstance(h, logging.StreamHandler) for h in root_logger.handlers):
stderr = sys.stderr
# XXX: Horrible hack required by the virtual stderr xonsh uses for each entered
# command getting closed after the command is run: we need to pull out
# the real stderr because this survives across multiple command runs.
#
# This only applies when running xonsh in interactive mode and importing releng.
if isinstance(sys.stderr, xonsh.base_shell._TeeStd):
stderr = stderr.std # type: ignore
hand = logging.StreamHandler(stream=stderr)
hand.set_name('releng root handler')
hand.setFormatter(fmt)
root_logger.addHandler(hand)
setup_logging()
def reload():
import importlib
importlib.reload(environment)
importlib.reload(create_release)
importlib.reload(keys)
importlib.reload(version)
importlib.reload(cli)
importlib.reload(docker)
importlib.reload(docker_assemble)
importlib.reload(gitutils)
+3
View File
@@ -0,0 +1,3 @@
from . import cli
cli.main()
+116
View File
@@ -0,0 +1,116 @@
from . import create_release
from . import docker
from .environment import RelengEnvironment
from . import environment
import argparse
import sys
def do_build(args):
if args.target == 'all':
create_release.build_artifacts(args.profile, no_check_git=args.no_check_git)
elif args.target == 'manual':
# n.b. args.profile does nothing here, you will just get the x86_64-linux manual no matter what.
eval_result = create_release.eval_jobs(args.profile)
create_release.build_manual(eval_result)
else:
raise ValueError('invalid target, unreachable')
def do_tag(args):
create_release.do_tag_merge(force_tag=args.force_tag,
no_check_git=args.no_check_git)
def do_upload(env: RelengEnvironment, args):
create_release.setup_creds(env)
if args.target == 'all':
docker.check_all_logins(env)
create_release.upload_artifacts(env,
force_push_tag=args.force_push_tag,
noconfirm=args.noconfirm,
no_check_git=args.no_check_git)
elif args.target == 'manual':
create_release.upload_manual(env)
else:
raise ValueError('invalid target, unreachable')
def do_prepare(args):
create_release.prepare_release_notes()
def main():
ap = argparse.ArgumentParser(description='*Lix ur release engineering*')
def fail(args):
ap.print_usage()
sys.exit(1)
ap.set_defaults(cmd=fail)
sps = ap.add_subparsers()
prepare = sps.add_parser(
'prepare',
help='Prepares for a release by moving the release notes over.')
prepare.set_defaults(cmd=do_prepare)
tag = sps.add_parser(
'tag',
help=
'Create the tag for the current release in .version and merge it back to the current branch, then switch to it'
)
tag.add_argument('--no-check-git',
action='store_true',
help="Don't check git state before tagging. For testing.")
tag.add_argument('--force-tag',
action='store_true',
help='Overwrite the existing tag. For testing.')
tag.set_defaults(cmd=do_tag)
build = sps.add_parser(
'build',
help=
'Build an artifacts/ directory with the things that would be released')
build.add_argument(
'--no-check-git',
action='store_true',
help="Don't check git state before building. For testing.")
build.add_argument('--target',
choices=['manual', 'all'],
help='Whether to build everything or just the manual')
build.add_argument('--profile',
default='all',
choices=('all', 'x86_64-linux-only'),
help='Which systems to build targets for.')
build.set_defaults(cmd=do_build)
upload = sps.add_parser(
'upload', help='Upload artifacts to cache and releases bucket')
upload.add_argument(
'--no-check-git',
action='store_true',
help="Don't check git state before uploading. For testing.")
upload.add_argument('--force-push-tag',
action='store_true',
help='Force push the tag. For testing.')
upload.add_argument(
'--target',
choices=['manual', 'all'],
default='all',
help='Whether to upload a release or just the nightly/otherwise manual'
)
upload.add_argument(
'--noconfirm',
action='store_true',
help="Don't ask for confirmation. For testing/automation.")
upload.add_argument('--environment',
choices=list(environment.ENVIRONMENTS.keys()),
default='staging',
help='Environment to release to')
upload.set_defaults(cmd=lambda args: do_upload(
environment.ENVIRONMENTS[args.environment], args))
args = ap.parse_args()
args.cmd(args)
+312
View File
@@ -0,0 +1,312 @@
import json
import subprocess
import itertools
import textwrap
from pathlib import Path
import tempfile
import hashlib
import datetime
from . import environment
from .environment import RelengEnvironment
from . import keys
from . import docker
from .version import VERSION, RELEASE_NAME, MAJOR
from .gitutils import verify_are_on_tag, git_preconditions
$RAISE_SUBPROC_ERROR = True
$XONSH_SHOW_TRACEBACK = True
GCROOTS_DIR = Path('./release/gcroots')
BUILT_GCROOTS_DIR = Path('./release/gcroots-build')
DRVS_TXT = Path('./release/drvs.txt')
ARTIFACTS = Path('./release/artifacts')
MANUAL = Path('./release/manual')
RELENG_MSG = "Release created with releng/create_release.xsh"
BUILD_CORES = 16
MAX_JOBS = 2
def setup_creds(env: RelengEnvironment):
key = keys.get_ephemeral_key(env)
$AWS_SECRET_ACCESS_KEY = key.secret_key
$AWS_ACCESS_KEY_ID = key.id
$AWS_DEFAULT_REGION = 'garage'
$AWS_ENDPOINT_URL = environment.S3_ENDPOINT
def official_release_commit_tag(force_tag=False):
print('[+] Setting officialRelease in flake.nix and tagging')
prev_branch = $(git symbolic-ref --short HEAD).strip()
git switch --detach
sed -i 's/officialRelease = false/officialRelease = true/' flake.nix
git add flake.nix
message = f'release: {VERSION} "{RELEASE_NAME}"\n\nRelease produced with releng/create_release.xsh'
git commit -m @(message)
git tag @(['-f'] if force_tag else []) -a -m @(message) @(VERSION)
return prev_branch
def merge_to_release(prev_branch):
git switch @(prev_branch)
# Create a merge back into the release branch so that git tools understand
# that the release branch contains the tag, without the release commit
# actually influencing the tree.
merge_msg = textwrap.dedent("""\
release: merge release {VERSION} back to mainline
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
{RELENG_MSG}
""").format(VERSION=VERSION, RELENG_MSG=RELENG_MSG)
git merge -m @(merge_msg) -s ours @(VERSION)
def realise(paths: list[str]):
args = [
'--realise',
'--max-jobs',
MAX_JOBS,
'--cores',
BUILD_CORES,
'--log-format',
'bar-with-logs',
'--add-root',
BUILT_GCROOTS_DIR
]
nix-store @(args) @(paths)
def eval_jobs(build_profile):
nej_output = $(nix-eval-jobs --workers 4 --gc-roots-dir @(GCROOTS_DIR) --force-recurse --flake f'.#release-jobs.{build_profile}')
return [json.loads(s) for s in nej_output.strip().split('\n')]
def upload_drv_paths_and_outputs(env: RelengEnvironment, paths: list[str]):
proc = subprocess.Popen([
'nix',
'copy',
'-v',
'--to',
env.cache_store_uri(),
'--stdin',
],
stdin=subprocess.PIPE,
env=__xonsh__.env.detype(),
)
proc.stdin.write('\n'.join(itertools.chain(paths, x + '^*' for x in paths)).encode())
proc.stdin.close()
rv = proc.wait()
if rv != 0:
raise subprocess.CalledProcessError(rv, proc.args)
def make_manifest(eval_result):
manifest = {vs['system']: vs['outputs']['out'] for vs in eval_result}
def manifest_line(system, out):
return f' {system} = "{out}";'
manifest_text = textwrap.dedent("""\
# This file was generated by releng/create_release.xsh in Lix
{{
{lines}
}}
""").format(lines='\n'.join(manifest_line(s, p) for (s, p) in manifest.items()))
return manifest_text
def make_git_tarball(to: Path):
git archive --verbose --prefix=lix-@(VERSION)/ --format=tar.gz -o @(to) @(VERSION)
def confirm(prompt, expected):
resp = input(prompt)
if resp != expected:
raise ValueError('Unconfirmed')
def sha256_file(f: Path):
hasher = hashlib.sha256()
with open(f, 'rb') as h:
while data := h.read(1024 * 1024):
hasher.update(data)
return hasher.hexdigest()
def make_artifacts_dir(eval_result, d: Path):
d.mkdir(exist_ok=True, parents=True)
version_dir = d / 'lix' / f'lix-{VERSION}'
version_dir.mkdir(exist_ok=True, parents=True)
tarballs_drv = next(p for p in eval_result if p['attr'] == 'tarballs')
cp --no-preserve=mode -r @(tarballs_drv['outputs']['out'])/* @(version_dir)
# FIXME: upgrade-nix searches for manifest.nix at root, which is rather annoying
with open(d / 'manifest.nix', 'w') as h:
h.write(make_manifest(eval_result))
with open(version_dir / 'manifest.nix', 'w') as h:
h.write(make_manifest(eval_result))
print('[+] Make sources tarball')
filename = f'lix-{VERSION}.tar.gz'
git_tarball = version_dir / filename
make_git_tarball(git_tarball)
file_hash = sha256_file(git_tarball)
print(f'Hash: {file_hash}')
with open(version_dir / f'{filename}.sha256', 'w') as h:
h.write(file_hash)
def prepare_release_notes():
print('[+] Preparing release notes')
RELEASE_NOTES_PATH = Path('doc/manual/rl-next')
if RELEASE_NOTES_PATH.isdir():
notes_body = subprocess.check_output(['build-release-notes', '--change-authors', 'doc/manual/change-authors.yml', 'doc/manual/rl-next']).decode()
else:
# I guess nobody put release notes on their changes?
print('[-] Warning: seemingly missing any release notes, not worrying about it')
notes_body = ''
rl_path = Path(f'doc/manual/src/release-notes/rl-{MAJOR}.md')
existing_rl = ''
try:
with open(rl_path, 'r') as fh:
existing_rl = fh.read()
except FileNotFoundError:
pass
date = datetime.datetime.now().strftime('%Y-%m-%d')
minor_header = f'# Lix {VERSION} ({date})'
header = f'# Lix {MAJOR} "{RELEASE_NAME}"'
if existing_rl.startswith(header):
# strip the header off for minor releases
lines = existing_rl.splitlines()
header = lines[0]
existing_rl = '\n'.join(lines[1:])
else:
header += f' ({date})\n\n'
header += '\n' + minor_header + '\n'
notes = header
notes += notes_body
notes += "\n\n"
notes += existing_rl
# make pre-commit happy about one newline
notes = notes.rstrip()
notes += "\n"
with open(rl_path, 'w') as fh:
fh.write(notes)
commit_msg = textwrap.dedent("""\
release: release notes for {VERSION}
{RELENG_MSG}
""").format(VERSION=VERSION, RELENG_MSG=RELENG_MSG)
git add @(rl_path)
git rm doc/manual/rl-next/*.md
git commit -m @(commit_msg)
def upload_artifacts(env: RelengEnvironment, noconfirm=False, no_check_git=False, force_push_tag=False):
if not no_check_git:
verify_are_on_tag()
git_preconditions()
assert 'AWS_SECRET_ACCESS_KEY' in __xonsh__.env
tree @(ARTIFACTS)
env_part = f'environment {env.name}'
not noconfirm and confirm(
f'Would you like to release {ARTIFACTS} as {VERSION} in {env.colour(env_part)}? Type "I want to release this to {env.name}" to confirm\n',
f'I want to release this to {env.name}'
)
docker_images = list((ARTIFACTS / f'lix/lix-{VERSION}').glob(f'lix-{VERSION}-docker-image-*.tar.gz'))
assert docker_images
print('[+] Upload to cache')
with open(DRVS_TXT) as fh:
upload_drv_paths_and_outputs(env, [x.strip() for x in fh.readlines() if x])
print('[+] Upload docker images')
for target in env.docker_targets:
docker.upload_docker_images(target, docker_images)
print('[+] Upload to release bucket')
aws s3 cp --recursive @(ARTIFACTS)/ @(env.releases_bucket)/
print('[+] Upload manual')
upload_manual(env)
print('[+] git push tag')
git push @(['-f'] if force_push_tag else []) @(env.git_repo) f'{VERSION}:refs/tags/{VERSION}'
def do_tag_merge(force_tag=False, no_check_git=False):
if not no_check_git:
git_preconditions()
prev_branch = official_release_commit_tag(force_tag=force_tag)
merge_to_release(prev_branch)
git switch --detach @(VERSION)
def build_manual(eval_result):
(drv, manual) = next((x['drvPath'], x['outputs']['doc']) for x in eval_result if x['attr'] == 'build.x86_64-linux')
print('[+] Building manual')
realise([drv])
cp --no-preserve=mode -T -vr @(manual)/share/doc/nix/manual @(MANUAL)
def upload_manual(env: RelengEnvironment):
stable = json.loads($(nix eval --json '.#nix.officialRelease'))
if stable:
version = MAJOR
else:
version = 'nightly'
print('[+] aws s3 sync manual')
aws s3 sync @(MANUAL)/ @(env.docs_bucket)/manual/lix/@(version)/
if stable:
aws s3 sync @(MANUAL)/ @(env.docs_bucket)/manual/lix/stable/
def build_artifacts(build_profile, no_check_git=False):
rm -rf release/
if not no_check_git:
verify_are_on_tag()
git_preconditions()
print('[+] Evaluating')
eval_result = eval_jobs(build_profile)
drv_paths = [x['drvPath'] for x in eval_result]
print('[+] Building')
realise(drv_paths)
build_manual(eval_result)
with open(DRVS_TXT, 'w') as fh:
# don't bother putting the release tarballs themselves because they are duplicate and huge
fh.write('\n'.join(x['drvPath'] for x in eval_result if x['attr'] != 'lix-release-tarballs'))
make_artifacts_dir(eval_result, ARTIFACTS)
print(f'[+] Done! See {ARTIFACTS}')
+83
View File
@@ -0,0 +1,83 @@
import json
import logging
from pathlib import Path
import tempfile
import requests
from .environment import DockerTarget, RelengEnvironment
from .version import VERSION, MAJOR
from . import gitutils
from .docker_assemble import Registry, OCIIndex, OCIIndexItem
from . import docker_assemble
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
def check_all_logins(env: RelengEnvironment):
for target in env.docker_targets:
check_login(target)
def check_login(target: DockerTarget):
log.info('Checking login for %s', target.registry_name)
skopeo login @(target.registry_name())
def upload_docker_images(target: DockerTarget, paths: list[Path]):
if not paths: return
sess = requests.Session()
sess.headers['User-Agent'] = 'lix-releng'
tag_names = [DockerTarget.resolve(tag, version=VERSION, major=MAJOR) for tag in target.tags]
# latest only gets tagged for the current release branch of Lix
if not gitutils.is_maintenance_branch('HEAD'):
tag_names.append('latest')
meta = {}
reg = docker_assemble.Registry(sess)
manifests = []
with tempfile.TemporaryDirectory() as tmp:
tmp = Path(tmp)
for path in paths:
digest_file = tmp / (path.name + '.digest')
tmp_image = tmp / 'tmp-image.tar.gz'
# insecure-policy: we don't have any signature policy, we are just uploading an image
#
# Absurd: we copy it into an OCI image first so we can get the hash
# we need to upload it untagged, because skopeo has no "don't tag
# this" option.
# The reason for this is that forgejo's container registry throws
# away old versions of tags immediately, so we cannot use a temp
# tag, and it *does* reduce confusion to not upload tags that
# should not be used.
#
# Workaround for: https://github.com/containers/skopeo/issues/2354
log.info('skopeo copy to temp oci-archive %s', tmp_image)
skopeo --insecure-policy copy --format oci --all --digestfile @(digest_file) docker-archive:@(path) oci-archive:@(tmp_image)
inspection = json.loads($(skopeo inspect oci-archive:@(tmp_image)))
docker_arch = inspection['Architecture']
docker_os = inspection['Os']
meta = inspection['Labels']
log.info('Pushing image %s for %s to %s', path, docker_arch, target.registry_path)
digest = digest_file.read_text().strip()
skopeo --insecure-policy copy --preserve-digests --all oci-archive:@(tmp_image) f'docker://{target.registry_path}@{digest}'
# skopeo doesn't give us the manifest size directly, so we just ask the registry
metadata = reg.image_info(target.registry_path, digest)
manifests.append(OCIIndexItem(metadata=metadata, architecture=docker_arch, os=docker_os))
log.info('Pushed images to %r, building a bigger and more menacing manifest from %r with metadata %r', target, manifests, meta)
# send the multiarch manifest to each tag
index = OCIIndex(manifests=manifests, annotations=meta)
for tag in tag_names:
reg.upload_index(target.registry_path, tag, index)
+391
View File
@@ -0,0 +1,391 @@
from typing import Any, Literal, Optional
import re
from pathlib import Path
import json
import dataclasses
import time
from urllib.parse import unquote
import urllib.request
import logging
import requests.auth
import requests
import xdg_base_dirs
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
DEBUG_REQUESTS = False
if DEBUG_REQUESTS:
urllib3_logger = logging.getLogger('requests.packages.urllib3')
urllib3_logger.setLevel(logging.DEBUG)
urllib3_logger.propagate = True
# So, there is a bunch of confusing stuff happening in this file. The gist of why it's Like This is:
#
# nix2container does not concern itself with tags (reasonably enough):
# https://github.com/nlewo/nix2container/issues/59
#
# This is fine. But then we noticed: docker images don't play nice if you have
# multiple architectures you want to abstract over if you don't do special
# things. Those special things are images with manifests containing multiple
# images.
#
# Docker has a data model vaguely analogous to git: you have higher level
# objects referring to a bunch of content-addressed blobs.
#
# A multiarch image is more or less just a manifest that refers to more
# manifests; in OCI it is an Index.
#
# See the API spec here: https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#definitions
# And the Index spec here: https://github.com/opencontainers/image-spec/blob/v1.0.1/image-index.md
#
# skopeo doesn't *know* how to make multiarch *manifests*:
# https://github.com/containers/skopeo/issues/1136
#
# There is a tool called manifest-tool that is supposed to do this
# (https://github.com/estesp/manifest-tool) but it doesn't support putting in
# annotations on the outer image, and I *really* didn't want to write golang to
# fix that. Thus, a little bit of homebrew containers code.
#
# Essentially what we are doing in here is splatting a bunch of images into the
# registry without tagging them (with a silly workaround to skopeo issues),
# then simply sending a new composite manifest ourselves.
DockerArchitecture = Literal['amd64'] | Literal['arm64']
MANIFEST_MIME = 'application/vnd.oci.image.manifest.v1+json'
INDEX_MIME = 'application/vnd.oci.image.index.v1+json'
@dataclasses.dataclass(frozen=True, order=True)
class ImageMetadata:
size: int
digest: str
"""sha256:SOMEHEX"""
@dataclasses.dataclass(frozen=True, order=True)
class OCIIndexItem:
"""Information about an untagged uploaded image."""
metadata: ImageMetadata
architecture: DockerArchitecture
os: str = 'linux'
def serialize(self):
return {
'mediaType': MANIFEST_MIME,
'size': self.metadata.size,
'digest': self.metadata.digest,
'platform': {
'architecture': self.architecture,
'os': self.os,
}
}
@dataclasses.dataclass(frozen=True)
class OCIIndex:
manifests: list[OCIIndexItem]
annotations: dict[str, str]
def serialize(self):
return {
'schemaVersion': 2,
'manifests': [item.serialize() for item in sorted(self.manifests)],
'annotations': self.annotations
}
@dataclasses.dataclass
class TaggingOperation:
manifest: OCIIndex
tags: list[str]
"""Tags this image is uploaded under"""
runtime_dir = xdg_base_dirs.xdg_runtime_dir()
config_dir = xdg_base_dirs.xdg_config_home()
AUTH_FILES = ([runtime_dir / 'containers/auth.json'] if runtime_dir else []) + \
[config_dir / 'containers/auth.json', Path.home() / '.docker/config.json']
# Copied from Werkzeug https://github.com/pallets/werkzeug/blob/62e3ea45846d06576199a2f8470be7fe44c867c1/src/werkzeug/http.py#L300-L325
def parse_list_header(value: str) -> list[str]:
"""Parse a header value that consists of a list of comma separated items according
to `RFC 9110 <https://httpwg.org/specs/rfc9110.html#abnf.extension>`__.
This extends :func:`urllib.request.parse_http_list` to remove surrounding quotes
from values.
.. code-block:: python
parse_list_header('token, "quoted value"')
['token', 'quoted value']
This is the reverse of :func:`dump_header`.
:param value: The header value to parse.
"""
result = []
for item in urllib.request.parse_http_list(value):
if len(item) >= 2 and item[0] == item[-1] == '"':
item = item[1:-1]
result.append(item)
return result
# https://www.rfc-editor.org/rfc/rfc2231#section-4
_charset_value_re = re.compile(
r"""
([\w!#$%&*+\-.^`|~]*)' # charset part, could be empty
[\w!#$%&*+\-.^`|~]*' # don't care about language part, usually empty
([\w!#$%&'*+\-.^`|~]+) # one or more token chars with percent encoding
""",
re.ASCII | re.VERBOSE,
)
# Copied from: https://github.com/pallets/werkzeug/blob/62e3ea45846d06576199a2f8470be7fe44c867c1/src/werkzeug/http.py#L327-L394
def parse_dict_header(value: str) -> dict[str, str | None]:
"""Parse a list header using :func:`parse_list_header`, then parse each item as a
``key=value`` pair.
.. code-block:: python
parse_dict_header('a=b, c="d, e", f')
{"a": "b", "c": "d, e", "f": None}
This is the reverse of :func:`dump_header`.
If a key does not have a value, it is ``None``.
This handles charsets for values as described in
`RFC 2231 <https://www.rfc-editor.org/rfc/rfc2231#section-3>`__. Only ASCII, UTF-8,
and ISO-8859-1 charsets are accepted, otherwise the value remains quoted.
:param value: The header value to parse.
.. versionchanged:: 3.0
Passing bytes is not supported.
.. versionchanged:: 3.0
The ``cls`` argument is removed.
.. versionchanged:: 2.3
Added support for ``key*=charset''value`` encoded items.
.. versionchanged:: 0.9
The ``cls`` argument was added.
"""
result: dict[str, str | None] = {}
for item in parse_list_header(value):
key, has_value, value = item.partition("=")
key = key.strip()
if not has_value:
result[key] = None
continue
value = value.strip()
encoding: str | None = None
if key[-1] == "*":
# key*=charset''value becomes key=value, where value is percent encoded
# adapted from parse_options_header, without the continuation handling
key = key[:-1]
match = _charset_value_re.match(value)
if match:
# If there is a charset marker in the value, split it off.
encoding, value = match.groups()
assert encoding
encoding = encoding.lower()
# A safe list of encodings. Modern clients should only send ASCII or UTF-8.
# This list will not be extended further. An invalid encoding will leave the
# value quoted.
if encoding in {"ascii", "us-ascii", "utf-8", "iso-8859-1"}:
# invalid bytes are replaced during unquoting
value = unquote(value, encoding=encoding)
if len(value) >= 2 and value[0] == value[-1] == '"':
value = value[1:-1]
result[key] = value
return result
def parse_www_authenticate(www_authenticate):
scheme, _, rest = www_authenticate.partition(' ')
scheme = scheme.lower()
rest = rest.strip()
parsed = parse_dict_header(rest.rstrip('='))
return parsed
class AuthState:
def __init__(self, auth_files: list[Path] = AUTH_FILES):
self.auth_map: dict[str, str] = {}
for f in auth_files:
self.auth_map.update(AuthState.load_auth_file(f))
self.token_cache: dict[str, str] = {}
@staticmethod
def load_auth_file(path: Path) -> dict[str, str]:
if path.exists():
with path.open() as fh:
try:
json_obj = json.load(fh)
return {k: v['auth'] for k, v in json_obj['auths'].items()}
except (json.JSONDecodeError, KeyError) as e:
log.exception('JSON decode error in %s', path, exc_info=e)
return {}
def get_token(self, hostname: str) -> Optional[str]:
return self.token_cache.get(hostname)
def obtain_token(self, session: requests.Session, token_endpoint: str,
scope: str, service: str, image_path: str) -> str:
authority, _, _ = image_path.partition('/')
if tok := self.get_token(authority):
return tok
creds = self.find_credential_for(image_path)
if not creds:
raise ValueError('No credentials available for ' + image_path)
resp = session.get(token_endpoint,
params={
'client_id': 'lix-releng',
'scope': scope,
'service': service,
},
headers={
'Authorization': 'Basic ' + creds
}).json()
token = resp['token']
self.token_cache[authority] = token
return token
def find_credential_for(self, image_path: str):
trails = image_path.split('/')
for i in range(len(trails)):
prefix = '/'.join(trails[:len(trails) - i])
if prefix in self.auth_map:
return self.auth_map[prefix]
return None
class RegistryAuthenticator(requests.auth.AuthBase):
"""Authenticates to an OCI compliant registry"""
def __init__(self, auth_state: AuthState, session: requests.Session,
image: str):
self.auth_map: dict[str, str] = {}
self.image = image
self.session = session
self.auth_state = auth_state
def response_hook(self, r: requests.Response,
**kwargs: Any) -> requests.Response:
if r.status_code == 401:
www_authenticate = r.headers.get('www-authenticate', '').lower()
parsed = parse_www_authenticate(www_authenticate)
assert parsed
tok = self.auth_state.obtain_token(
self.session,
parsed['realm'], # type: ignore
parsed['scope'], # type: ignore
parsed['service'], # type: ignore
self.image)
new_req = r.request.copy()
new_req.headers['Authorization'] = 'Bearer ' + tok
return self.session.send(new_req)
else:
return r
def __call__(self,
r: requests.PreparedRequest) -> requests.PreparedRequest:
authority, _, _ = self.image.partition('/')
auth_may = self.auth_state.get_token(authority)
if auth_may:
r.headers['Authorization'] = 'Bearer ' + auth_may
r.register_hook('response', self.response_hook)
return r
class Registry:
def __init__(self, session: requests.Session):
self.auth_state = AuthState()
self.session = session
def image_info(self, image_path: str, manifest_id: str) -> ImageMetadata:
authority, _, path = image_path.partition('/')
resp = self.session.head(
f'https://{authority}/v2/{path}/manifests/{manifest_id}',
headers={'Accept': MANIFEST_MIME},
auth=RegistryAuthenticator(self.auth_state, self.session,
image_path))
resp.raise_for_status()
return ImageMetadata(int(resp.headers['content-length']),
resp.headers['docker-content-digest'])
def delete_tag(self, image_path: str, tag: str):
authority, _, path = image_path.partition('/')
resp = self.session.delete(
f'https://{authority}/v2/{path}/manifests/{tag}',
headers={'Content-Type': INDEX_MIME},
auth=RegistryAuthenticator(self.auth_state, self.session,
image_path))
resp.raise_for_status()
def _upload_index(self, image_path: str, tag: str, index: OCIIndex):
authority, _, path = image_path.partition('/')
body = json.dumps(index.serialize(),
separators=(',', ':'),
sort_keys=True)
resp = self.session.put(
f'https://{authority}/v2/{path}/manifests/{tag}',
data=body,
headers={'Content-Type': INDEX_MIME},
auth=RegistryAuthenticator(self.auth_state, self.session,
image_path))
resp.raise_for_status()
return resp.headers['Location']
def upload_index(self,
image_path: str,
tag: str,
index: OCIIndex,
retries=20,
retry_delay=1):
# eventual consistency lmao
for _ in range(retries):
try:
return self._upload_index(image_path, tag, index)
except requests.HTTPError as e:
if e.response.status_code != 404:
raise
time.sleep(retry_delay)
+134
View File
@@ -0,0 +1,134 @@
from typing import Callable
import urllib.parse
import re
import functools
import subprocess
import dataclasses
S3_HOST = 's3.lix.systems'
S3_ENDPOINT = 'https://s3.lix.systems'
DEFAULT_STORE_URI_BITS = {
'region': 'garage',
'endpoint': 's3.lix.systems',
'want-mass-query': 'true',
'write-nar-listing': 'true',
'ls-compression': 'zstd',
'narinfo-compression': 'zstd',
'compression': 'zstd',
'parallel-compression': 'true',
}
@dataclasses.dataclass
class DockerTarget:
registry_path: str
"""Registry path without the tag, e.g. ghcr.io/lix-project/lix"""
tags: list[str]
"""List of tags this image should take. There must be at least one."""
@staticmethod
def resolve(item: str, version: str, major: str) -> str:
"""
Applies templates:
- version: the Lix version e.g. 2.90.0
- major: the major Lix version e.g. 2.90
"""
return item.format(version=version, major=major)
def registry_name(self) -> str:
[a, _, _] = self.registry_path.partition('/')
return a
@dataclasses.dataclass
class RelengEnvironment:
name: str
colour: Callable[[str], str]
cache_store_overlay: dict[str, str]
cache_bucket: str
releases_bucket: str
docs_bucket: str
git_repo: str
docker_targets: list[DockerTarget]
def cache_store_uri(self):
qs = DEFAULT_STORE_URI_BITS.copy()
qs.update(self.cache_store_overlay)
return self.cache_bucket + "?" + urllib.parse.urlencode(qs)
SGR = '\x1b['
RED = '31;1m'
GREEN = '32;1m'
RESET = '0m'
def sgr(colour: str, text: str) -> str:
return f'{SGR}{colour}{text}{SGR}{RESET}'
STAGING = RelengEnvironment(
name='staging',
colour=functools.partial(sgr, GREEN),
docs_bucket='s3://staging-docs',
cache_bucket='s3://staging-cache',
cache_store_overlay={'secret-key': 'staging.key'},
releases_bucket='s3://staging-releases',
git_repo='ssh://git@git.lix.systems/lix-project/lix-releng-staging',
docker_targets=[
# latest will be auto tagged if appropriate
DockerTarget('git.lix.systems/lix-project/lix-releng-staging',
tags=['{version}', '{major}']),
DockerTarget('ghcr.io/lix-project/lix-releng-staging',
tags=['{version}', '{major}']),
],
)
GERRIT_REMOTE_RE = re.compile(r'^ssh://(\w+@)?gerrit.lix.systems:2022/lix$')
def guess_gerrit_remote():
"""
Deals with people having unknown gerrit username.
"""
out = [
x.split()[1] for x in subprocess.check_output(
['git', 'remote', '-v']).decode().splitlines()
]
return next(x for x in out if GERRIT_REMOTE_RE.match(x))
PROD = RelengEnvironment(
name='production',
colour=functools.partial(sgr, RED),
docs_bucket='s3://docs',
cache_bucket='s3://cache',
# FIXME: we should decrypt this with age into a tempdir in the future, but
# the issue is how to deal with the recipients file. For now, we should
# just delete it after doing a release.
cache_store_overlay={'secret-key': 'prod.key'},
releases_bucket='s3://releases',
git_repo=guess_gerrit_remote(),
docker_targets=[
# latest will be auto tagged if appropriate
DockerTarget('git.lix.systems/lix-project/lix',
tags=['{version}', '{major}']),
DockerTarget('ghcr.io/lix-project/lix', tags=['{version}', '{major}']),
],
)
ENVIRONMENTS = {
'staging': STAGING,
'production': PROD,
}
@dataclasses.dataclass
class S3Credentials:
name: str
id: str
secret_key: str
+39
View File
@@ -0,0 +1,39 @@
import subprocess
import json
from .version import VERSION
def version_compare(v1: str, v2: str):
return json.loads($(nix-instantiate --eval --json --argstr v1 @(v1) --argstr v2 @(v2) --expr '{v1, v2}: builtins.compareVersions v1 v2'))
def latest_tag_on_branch(branch: str) -> str:
return $(git describe --abbrev=0 @(branch) e>/dev/null).strip()
def is_maintenance_branch(branch: str) -> bool:
try:
main_tag = latest_tag_on_branch('main')
current_tag = latest_tag_on_branch(branch)
return version_compare(current_tag, main_tag) < 0
except subprocess.CalledProcessError:
# This is the case before Lix releases 2.90, since main *has* no
# release tag on it.
# FIXME: delete this case after 2.91
return False
def verify_are_on_tag():
current_tag = $(git describe --tag).strip()
assert current_tag == VERSION
def git_preconditions():
# verify there is nothing in index ready to stage
proc = !(git diff-index --quiet --cached HEAD --)
assert proc.rtn == 0
# verify there is nothing *stageable* and tracked
proc = !(git diff-files --quiet)
assert proc.rtn == 0
+19
View File
@@ -0,0 +1,19 @@
import subprocess
import json
from . import environment
def get_ephemeral_key(
env: environment.RelengEnvironment) -> environment.S3Credentials:
output = subprocess.check_output([
'ssh', '-l', 'root', environment.S3_HOST, 'garage-ephemeral-key',
'new', '--name', f'releng-{env.name}', '--read', '--write',
'--age-secs', '3600',
env.releases_bucket.removeprefix('s3://'),
env.cache_bucket.removeprefix('s3://'),
env.docs_bucket.removeprefix('s3://'),
])
d = json.loads(output.decode())
return environment.S3Credentials(name=d['name'],
id=d['id'],
secret_key=d['secret_key'])
+92
View File
@@ -0,0 +1,92 @@
{ hydraJobs, pkgs }:
let
inherit (pkgs) lib;
lix = hydraJobs.build.x86_64-linux;
# This is all so clumsy because we can't use arguments to functions in
# flakes, and certainly not with n-e-j.
profiles = {
# Used for testing
x86_64-linux-only = {
systems = [ "x86_64-linux" ];
dockerSystems = [ "x86_64-linux" ];
};
all = {
systems = [
"x86_64-linux"
"aarch64-linux"
"aarch64-darwin"
"x86_64-darwin"
];
dockerSystems = [
"x86_64-linux"
"aarch64-linux"
];
};
};
doTarball =
{
target,
targetName,
rename ? null,
}:
''
echo "doing: ${target}"
# expand wildcard
filename=$(echo ${target}/${targetName})
basename="$(basename $filename)"
echo $filename $basename
cp -v "$filename" "$out"
${lib.optionalString (rename != null) ''
mv "$out/$basename" "$out/${rename}"
basename="${rename}"
''}
sha256sum --binary $filename | cut -f1 -d' ' > $out/$basename.sha256
'';
targetsFor =
{ systems, dockerSystems }:
builtins.map (system: {
target = hydraJobs.binaryTarball.${system};
targetName = "*.tar.xz";
}) systems
++ builtins.map (system: {
target = hydraJobs.dockerImage.${system}.tarball;
targetName = "image.tar.gz";
rename = "lix-${lix.version}-docker-image-${system}.tar.gz";
}) dockerSystems;
manualTar = pkgs.runCommand "lix-manual-tarball" { } ''
mkdir -p $out
cp -r ${lix.doc}/share/doc/nix/manual lix-${lix.version}-manual
tar -cvzf "$out/lix-${lix.version}-manual.tar.gz" lix-${lix.version}-manual
'';
tarballsFor =
{ systems, dockerSystems }:
pkgs.runCommand "lix-release-tarballs" { } ''
mkdir -p $out
${lib.concatMapStringsSep "\n" doTarball (targetsFor {
inherit systems dockerSystems;
})}
${doTarball {
target = manualTar;
targetName = "lix-*.tar.gz";
}}
cp -r ${lix.doc}/share/doc/nix/manual $out
'';
in
(builtins.mapAttrs (
_:
{ systems, dockerSystems }:
{
build = lib.filterAttrs (x: _: builtins.elem x systems) hydraJobs.build;
tarballs = tarballsFor { inherit systems dockerSystems; };
}
) profiles)
// {
inherit (hydraJobs) build;
inherit tarballsFor;
}
+6
View File
@@ -0,0 +1,6 @@
import json
version_json = json.load(open('version.json'))
VERSION = version_json['version']
MAJOR = '.'.join(VERSION.split('.')[:2])
RELEASE_NAME = version_json['release_name']
+16 -3
View File
@@ -3,6 +3,7 @@
#include <algorithm>
#include <set>
#include <memory>
#include <string_view>
#include <tuple>
#include <iomanip>
#if __APPLE__
@@ -20,9 +21,9 @@
#include "local-store.hh"
#include "legacy.hh"
#include "experimental-features.hh"
#include "hash.hh"
using namespace nix;
using std::cin;
static void handleAlarm(int sig) {
}
@@ -35,9 +36,19 @@ std::string escapeUri(std::string uri)
static std::string currentLoad;
static std::string makeLockFilename(const std::string & storeUri) {
// We include 48 bytes of escaped URI to give an idea of what the lock
// is on, then 16 bytes of hash to disambiguate.
// This avoids issues with the escaped URI being very long and causing
// path too long errors, while also avoiding any possibility of collision
// caused by simple truncation.
auto hash = hashString(HashType::htSHA256, storeUri).to_string(Base::Base32, false);
return escapeUri(storeUri).substr(0, 48) + "-" + hash.substr(0, 16);
}
static AutoCloseFD openSlotLock(const Machine & m, uint64_t slot)
{
return openLockFile(fmt("%s/%s-%d", currentLoad, escapeUri(m.storeUri), slot), true);
return openLockFile(fmt("%s/%s-%d", currentLoad, makeLockFilename(m.storeUri), slot), true);
}
static bool allSupportedLocally(Store & store, const std::set<std::string>& requiredFeatures) {
@@ -263,7 +274,9 @@ connected:
auto inputs = readStrings<PathSet>(source);
auto wantedOutputs = readStrings<StringSet>(source);
AutoCloseFD uploadLock = openLockFile(currentLoad + "/" + escapeUri(storeUri) + ".upload-lock", true);
auto lockFileName = currentLoad + "/" + makeLockFilename(storeUri) + ".upload-lock";
AutoCloseFD uploadLock = openLockFile(lockFileName, true);
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("waiting for the upload lock to '%s'", storeUri));
+3 -3
View File
@@ -45,7 +45,7 @@ MixEvalArgs::MixEvalArgs()
-I /etc/nixos
```
will cause Nix to look for paths relative to `/home/eelco/Dev` and
will cause Lix to look for paths relative to `/home/eelco/Dev` and
`/etc/nixos`, in that order. This is equivalent to setting the
`NIX_PATH` environment variable to
@@ -61,7 +61,7 @@ MixEvalArgs::MixEvalArgs()
-I /etc/nixos
```
will cause Nix to search for `<nixpkgs/path>` in
will cause Lix to search for `<nixpkgs/path>` in
`/home/eelco/Dev/nixpkgs-branch/path` and `/etc/nixos/nixpkgs/path`.
If a path in the Nix search path starts with `http://` or `https://`,
@@ -73,7 +73,7 @@ MixEvalArgs::MixEvalArgs()
-I nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz
```
tells Nix to download and use the current contents of the `master`
tells Lix to download and use the current contents of the `master`
branch in the `nixpkgs` repository.
The URLs of the tarballs from the official `nixos.org` channels
+1
View File
@@ -54,6 +54,7 @@ libcmd = library(
nlohmann_json,
lix_doc
],
cpp_pch : cpp_pch,
install : true,
# FIXME(Qyriad): is this right?
install_rpath : libdir,
+3 -3
View File
@@ -64,15 +64,15 @@ struct EvalSettings : Config
Pure evaluation mode ensures that the result of Nix expressions is fully determined by explicitly declared inputs, and not influenced by external state:
- Restrict file system and network access to files specified by cryptographic hash
- Disable [`bultins.currentSystem`](@docroot@/language/builtin-constants.md#builtins-currentSystem) and [`builtins.currentTime`](@docroot@/language/builtin-constants.md#builtins-currentTime)
- Disable [`builtins.currentSystem`](@docroot@/language/builtin-constants.md#builtins-currentSystem) and [`builtins.currentTime`](@docroot@/language/builtin-constants.md#builtins-currentTime)
)"
};
Setting<bool> enableImportFromDerivation{
this, true, "allow-import-from-derivation",
R"(
By default, Nix allows you to `import` from a derivation, allowing
building at evaluation time. With this option set to false, Nix will
By default, Lix allows you to `import` from a derivation, allowing
building at evaluation time. With this option set to false, Lix will
throw an error when evaluating an expression that uses this feature,
allowing users to ensure their evaluation will not require any
builds to take place.
+1 -1
View File
@@ -22,6 +22,7 @@
#include <algorithm>
#include <chrono>
#include <iostream>
#include <sstream>
#include <cstring>
#include <optional>
#include <unistd.h>
@@ -29,7 +30,6 @@
#include <sys/resource.h>
#include <fstream>
#include <functional>
#include <iostream>
#include <sys/resource.h>
#include <nlohmann/json.hpp>
+5 -5
View File
@@ -26,10 +26,9 @@
#pragma clang diagnostic ignored "-Wimplicit-fallthrough"
#endif
#include <boost/lexical_cast.hpp>
#include "nixexpr.hh"
#include "parser-tab.hh"
#include "strings.hh"
using namespace nix;
@@ -132,9 +131,10 @@ or { return OR_KW; }
{ID} { yylval->id = {yytext, (size_t) yyleng}; return ID; }
{INT} { errno = 0;
try {
yylval->n = boost::lexical_cast<int64_t>(yytext);
} catch (const boost::bad_lexical_cast &) {
std::optional<int64_t> numMay = string2Int<int64_t>(yytext);
if (numMay.has_value()) {
yylval->n = *numMay;
} else {
throw ParseError(ErrorInfo{
.msg = HintFmt("invalid integer '%1%'", yytext),
.pos = state->positions[CUR_POS],
+11 -14
View File
@@ -86,25 +86,11 @@ libexpr_sources = files(
'flake/flake.cc',
'flake/flakeref.cc',
'flake/lockfile.cc',
'primops/arithmetic.cc',
'primops/attrset.cc',
'primops/context.cc',
'primops/control.cc',
'primops/debug.cc',
'primops/derivation.cc',
'primops/hash.cc',
'primops/fetchClosure.cc',
'primops/fetchMercurial.cc',
'primops/fetchTree.cc',
'primops/fromTOML.cc',
'primops/import.cc',
'primops/json.cc',
'primops/list.cc',
'primops/path.cc',
'primops/string.cc',
'primops/system.cc',
'primops/toXML.cc',
'primops/types.cc',
'value/context.cc',
)
@@ -159,6 +145,7 @@ libexpr = library(
include_directories : [
'../libmain',
],
cpp_pch : cpp_pch,
install : true,
# FIXME(Qyriad): is this right?
install_rpath : libdir,
@@ -175,6 +162,16 @@ liblixexpr = declare_dependency(
link_with : libexpr,
)
# FIXME: remove when https://git.lix.systems/lix-project/lix/issues/359 is fixed.
if is_static
liblixexpr_mstatic = declare_dependency(
include_directories : include_directories('.'),
link_whole : libexpr,
)
else
liblixexpr_mstatic = liblixexpr
endif
# FIXME: not using the pkg-config module because it creates way too many deps
# while meson migration is in progress, and we want to not include boost here
configure_file(
+1
View File
@@ -6,6 +6,7 @@
#include "escape-string.hh"
#include <cstdlib>
#include <sstream>
namespace nix {
+4261 -339
View File
File diff suppressed because it is too large Load Diff
+3 -141
View File
@@ -1,10 +1,9 @@
#pragma once
///@file
#include "derivations.hh"
#include "eval.hh"
#include <regex>
#include <tuple>
#include <vector>
namespace nix {
@@ -45,150 +44,13 @@ struct RegisterPrimOp
/**
* Load a ValueInitializer from a DSO and return whatever it initializes
*/
void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v);
void prim_importNative(EvalState & state, const PosIdx pos, Value * * args, Value & v);
/**
* Execute a program and parse its output
*/
void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v);
void prim_lessThan(EvalState & state, const PosIdx pos, Value ** args, Value & v);
void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v);
void prim_trace(EvalState & state, const PosIdx pos, Value ** args, Value & v);
void prim_second(EvalState & state, const PosIdx pos, Value ** args, Value & v);
void prim_exec(EvalState & state, const PosIdx pos, Value * * args, Value & v);
void makePositionThunks(EvalState & state, const PosIdx pos, Value & line, Value & column);
void mkOutputString(
EvalState & state,
BindingsBuilder & attrs,
const StorePath & drvPath,
const std::pair<std::string, DerivationOutput> & o
);
#if HAVE_BOEHMGC
typedef std::list<Value *, gc_allocator<Value *>> ValueList;
#else
typedef std::list<Value *> ValueList;
#endif
/**
* getAttr wrapper
*/
Bindings::iterator
getAttr(EvalState & state, Symbol attrSym, Bindings * attrSet, std::string_view errorCtx);
/**
* Struct definitions
*/
struct RegexCache
{
// TODO use C++20 transparent comparison when available
std::unordered_map<std::string_view, std::regex> cache;
std::list<std::string> keys;
std::regex get(std::string_view re)
{
auto it = cache.find(re);
if (it != cache.end()) {
return it->second;
}
keys.emplace_back(re);
return cache.emplace(keys.back(), std::regex(keys.back(), std::regex::extended))
.first->second;
}
};
struct CompareValues
{
EvalState & state;
const PosIdx pos;
const std::string_view errorCtx;
CompareValues(EvalState & state, const PosIdx pos, const std::string_view && errorCtx)
: state(state)
, pos(pos)
, errorCtx(errorCtx){};
bool operator()(Value * v1, Value * v2) const
{
return (*this)(v1, v2, errorCtx);
}
bool operator()(Value * v1, Value * v2, std::string_view errorCtx) const
{
try {
if (v1->type() == nFloat && v2->type() == nInt) {
return v1->fpoint < v2->integer;
}
if (v1->type() == nInt && v2->type() == nFloat) {
return v1->integer < v2->fpoint;
}
if (v1->type() != v2->type()) {
state.error<EvalError>("cannot compare %s with %s", showType(*v1), showType(*v2))
.debugThrow();
}
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (v1->type()) {
case nInt:
return v1->integer < v2->integer;
case nFloat:
return v1->fpoint < v2->fpoint;
case nString:
return strcmp(v1->string.s, v2->string.s) < 0;
case nPath:
return strcmp(v1->_path, v2->_path) < 0;
case nList:
// Lexicographic comparison
for (size_t i = 0;; i++) {
if (i == v2->listSize()) {
return false;
} else if (i == v1->listSize()) {
return true;
} else if (!state.eqValues(
*v1->listElems()[i], *v2->listElems()[i], pos, errorCtx
))
{
return (*this)(
v1->listElems()[i],
v2->listElems()[i],
"while comparing two list elements"
);
}
}
default:
state
.error<EvalError>(
"cannot compare %s with %s; values of that type are incomparable",
showType(*v1),
showType(*v2)
)
.debugThrow();
#pragma GCC diagnostic pop
}
} catch (Error & e) {
if (!errorCtx.empty()) {
e.addTrace(nullptr, errorCtx);
}
throw;
}
}
};
struct RealisePathFlags
{
// Whether to check that the path is allowed in pure eval mode
bool checkForPureEval = true;
};
SourcePath
realisePath(EvalState & state, const PosIdx pos, Value & v, const RealisePathFlags flags = {});
}
-312
View File
@@ -1,312 +0,0 @@
#include <math.h> // for ceil, floor
#include <limits> // for numeric_limits
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp
#include "pos-idx.hh" // for PosIdx, noPos
#include "primops.hh" // for CompareValues, prim_lessThan
#include "value.hh" // for Value, nFloat, NixInt, NixFloat
namespace nix {
/**
* builtins.add
*/
static void prim_add(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
state.forceValue(*args[1], pos);
if (args[0]->type() == nFloat || args[1]->type() == nFloat) {
v.mkFloat(
state.forceFloat(*args[0], pos, "while evaluating the first argument of the addition")
+ state.forceFloat(
*args[1], pos, "while evaluating the second argument of the addition"
)
);
} else {
v.mkInt(
state.forceInt(*args[0], pos, "while evaluating the first argument of the addition")
+ state.forceInt(*args[1], pos, "while evaluating the second argument of the addition")
);
}
}
PrimOp primop_add({
.name = "__add",
.args = {"e1", "e2"},
.doc = R"(
Return the sum of the numbers *e1* and *e2*.
Return a float if either *e1* or *e2* is a float, otherwise
return an integer.
)",
.fun = prim_add,
});
/**
* builtins.bitAnd
*/
static void prim_bitAnd(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
v.mkInt(
state.forceInt(
*args[0], pos, "while evaluating the first argument passed to builtins.bitAnd"
)
& state.forceInt(
*args[1], pos, "while evaluating the second argument passed to builtins.bitAnd"
)
);
}
PrimOp primop_bitAnd({
.name = "__bitAnd",
.args = {"e1", "e2"},
.doc = R"(
Return the bitwise AND of the integers *e1* and *e2*.
)",
.fun = prim_bitAnd,
});
/**
* builtins.bitOr
*/
static void prim_bitOr(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
v.mkInt(
state.forceInt(
*args[0], pos, "while evaluating the first argument passed to builtins.bitOr"
)
| state.forceInt(
*args[1], pos, "while evaluating the second argument passed to builtins.bitOr"
)
);
}
PrimOp primop_bitOr({
.name = "__bitOr",
.args = {"e1", "e2"},
.doc = R"(
Return the bitwise OR of the integers *e1* and *e2*.
)",
.fun = prim_bitOr,
});
/**
* builtins.bitXor
*/
static void prim_bitXor(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
v.mkInt(
state.forceInt(
*args[0], pos, "while evaluating the first argument passed to builtins.bitXor"
)
^ state.forceInt(
*args[1], pos, "while evaluating the second argument passed to builtins.bitXor"
)
);
}
PrimOp primop_bitXor({
.name = "__bitXor",
.args = {"e1", "e2"},
.doc = R"(
Return the bitwise XOR of the integers *e1* and *e2*.
)",
.fun = prim_bitXor,
});
/**
* builtins.ceil
*/
static void prim_ceil(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto value = state.forceFloat(
*args[0],
args[0]->determinePos(pos),
"while evaluating the first argument passed to builtins.ceil"
);
v.mkInt(ceil(value));
}
PrimOp primop_ceil({
.name = "__ceil",
.args = {"double"},
.doc = R"(
Converts an IEEE-754 double-precision floating-point number (*double*) to
the next higher integer.
If the datatype is neither an integer nor a "float", an evaluation error will be
thrown.
)",
.fun = prim_ceil,
});
/**
* builtins.div
*/
static void prim_div(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
state.forceValue(*args[1], pos);
NixFloat f2 =
state.forceFloat(*args[1], pos, "while evaluating the second operand of the division");
if (f2 == 0) {
state.error<EvalError>("division by zero").atPos(pos).debugThrow();
}
if (args[0]->type() == nFloat || args[1]->type() == nFloat) {
v.mkFloat(
state.forceFloat(*args[0], pos, "while evaluating the first operand of the division")
/ f2
);
} else {
NixInt i1 =
state.forceInt(*args[0], pos, "while evaluating the first operand of the division");
NixInt i2 =
state.forceInt(*args[1], pos, "while evaluating the second operand of the division");
/* Avoid division overflow as it might raise SIGFPE. */
if (i1 == std::numeric_limits<NixInt>::min() && i2 == -1) {
state.error<EvalError>("overflow in integer division").atPos(pos).debugThrow();
}
v.mkInt(i1 / i2);
}
}
PrimOp primop_div({
.name = "__div",
.args = {"e1", "e2"},
.doc = R"(
Return the quotient of the numbers *e1* and *e2*.
)",
.fun = prim_div,
});
/**
* builtins.floor
*/
static void prim_floor(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto value = state.forceFloat(
*args[0],
args[0]->determinePos(pos),
"while evaluating the first argument passed to builtins.floor"
);
v.mkInt(floor(value));
}
PrimOp primop_floor({
.name = "__floor",
.args = {"double"},
.doc = R"(
Converts an IEEE-754 double-precision floating-point number (*double*) to
the next lower integer.
If the datatype is neither an integer nor a "float", an evaluation error will be
thrown.
)",
.fun = prim_floor,
});
/**
* builtins.lessThan
*/
void prim_lessThan(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
state.forceValue(*args[1], pos);
// pos is exact here, no need for a message.
CompareValues comp(state, noPos, "");
v.mkBool(comp(args[0], args[1]));
}
PrimOp primop_lessThan({
.name = "__lessThan",
.args = {"e1", "e2"},
.doc = R"(
Return `true` if the number *e1* is less than the number *e2*, and
`false` otherwise. Evaluation aborts if either *e1* or *e2* does not
evaluate to a number.
)",
.fun = prim_lessThan,
});
/**
* builtins.mul
*/
static void prim_mul(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
state.forceValue(*args[1], pos);
if (args[0]->type() == nFloat || args[1]->type() == nFloat) {
v.mkFloat(
state.forceFloat(*args[0], pos, "while evaluating the first of the multiplication")
* state.forceFloat(
*args[1], pos, "while evaluating the second argument of the multiplication"
)
);
} else {
v.mkInt(
state.forceInt(
*args[0], pos, "while evaluating the first argument of the multiplication"
)
* state.forceInt(
*args[1], pos, "while evaluating the second argument of the multiplication"
)
);
}
}
PrimOp primop_mul({
.name = "__mul",
.args = {"e1", "e2"},
.doc = R"(
Return the product of the numbers *e1* and *e2*.
)",
.fun = prim_mul,
});
/**
* builtins.sub
*/
static void prim_sub(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
state.forceValue(*args[1], pos);
if (args[0]->type() == nFloat || args[1]->type() == nFloat) {
v.mkFloat(
state.forceFloat(
*args[0], pos, "while evaluating the first argument of the subtraction"
)
- state.forceFloat(
*args[1], pos, "while evaluating the second argument of the subtraction"
)
);
} else {
v.mkInt(
state.forceInt(*args[0], pos, "while evaluating the first argument of the subtraction")
- state.forceInt(
*args[1], pos, "while evaluating the second argument of the subtraction"
)
);
}
}
PrimOp primop_sub({
.name = "__sub",
.args = {"e1", "e2"},
.doc = R"(
Return the difference between the numbers *e1* and *e2*.
)",
.fun = prim_sub,
});
}
-698
View File
@@ -1,698 +0,0 @@
#include <string.h> // for size_t, memcpy, strcmp
#include <algorithm> // for sort, min, set_difference
#include <boost/container/small_vector.hpp> // for small_vector
#include <boost/container/vector.hpp> // for operator+, operator-
#include <iterator> // for back_insert_iterator
#include <map> // for map, _Rb_tree_iterator
#include <set> // for set
#include <string_view> // for string_view, operator<=>
#include <utility> // for pair
#include <vector> // for vector
#include "attr-set.hh" // for Attr, Bindings, Bindings...
#include "eval-error.hh" // for TypeError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp, prim_...
#include "gc-small-vector.hh" // for SmallValueVector
#include "pos-idx.hh" // for PosIdx, noPos
#include "primops.hh" // for CompareValues, ValueList
#include "symbol-table.hh" // for Symbol, SymbolTable, Sym...
#include "value.hh" // for Value, ValueVector, Valu...
namespace nix {
Bindings::iterator
getAttr(EvalState & state, Symbol attrSym, Bindings * attrSet, std::string_view errorCtx)
{
Bindings::iterator value = attrSet->find(attrSym);
if (value == attrSet->end()) {
state.error<TypeError>("attribute '%s' missing", state.symbols[attrSym])
.withTrace(noPos, errorCtx)
.debugThrow();
}
return value;
}
/**
* builtins.attrNames
*/
static void prim_attrNames(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.attrNames");
state.mkList(v, args[0]->attrs->size());
size_t n = 0;
for (auto & i : *args[0]->attrs) {
(v.listElems()[n++] = state.allocValue())->mkString(state.symbols[i.name]);
}
std::sort(v.listElems(), v.listElems() + n, [](Value * v1, Value * v2) {
return strcmp(v1->string.s, v2->string.s) < 0;
});
}
PrimOp primop_attrNames({
.name = "__attrNames",
.args = {"set"},
.doc = R"(
Return the names of the attributes in the set *set* in an
alphabetically sorted list. For instance, `builtins.attrNames { y
= 1; x = "foo"; }` evaluates to `[ "x" "y" ]`.
)",
.fun = prim_attrNames,
});
/**
* builtins.attrNames
*/
static void prim_attrValues(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.attrValues");
state.mkList(v, args[0]->attrs->size());
unsigned int n = 0;
for (auto & i : *args[0]->attrs) {
v.listElems()[n++] = (Value *) &i;
}
std::sort(v.listElems(), v.listElems() + n, [&](Value * v1, Value * v2) {
std::string_view s1 = state.symbols[((Attr *) v1)->name],
s2 = state.symbols[((Attr *) v2)->name];
return s1 < s2;
});
for (unsigned int i = 0; i < n; ++i) {
v.listElems()[i] = ((Attr *) v.listElems()[i])->value;
}
}
PrimOp primop_attrValues({
.name = "__attrValues",
.args = {"set"},
.doc = R"(
Return the values of the attributes in the set *set* in the order
corresponding to the sorted attribute names.
)",
.fun = prim_attrValues,
});
/**
* builtins.catAttrs
*/
static void prim_catAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto attrName = state.symbols.create(state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.catAttrs"
));
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.catAttrs"
);
SmallValueVector<nonRecursiveStackReservation> res(args[1]->listSize());
size_t found = 0;
for (auto v2 : args[1]->listItems()) {
state.forceAttrs(
*v2,
pos,
"while evaluating an element in the list passed as second argument to builtins.catAttrs"
);
Bindings::iterator i = v2->attrs->find(attrName);
if (i != v2->attrs->end()) {
res[found++] = i->value;
}
}
state.mkList(v, found);
for (unsigned int n = 0; n < found; ++n) {
v.listElems()[n] = res[n];
}
}
PrimOp primop_catAttrs({
.name = "__catAttrs",
.args = {"attr", "list"},
.doc = R"(
Collect each attribute named *attr* from a list of attribute
sets. Attrsets that don't contain the named attribute are
ignored. For example,
```nix
builtins.catAttrs "a" [{a = 1;} {b = 0;} {a = 2;}]
```
evaluates to `[1 2]`.
)",
.fun = prim_catAttrs,
});
/**
* builtins.genericClosure
*/
static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(
*args[0], noPos, "while evaluating the first argument passed to builtins.genericClosure"
);
/* Get the start set. */
Bindings::iterator startSet = getAttr(
state,
state.sStartSet,
args[0]->attrs,
"in the attrset passed as argument to builtins.genericClosure"
);
state.forceList(
*startSet->value,
noPos,
"while evaluating the 'startSet' attribute passed as argument to builtins.genericClosure"
);
ValueList workSet;
for (auto elem : startSet->value->listItems()) {
workSet.push_back(elem);
}
if (startSet->value->listSize() == 0) {
v = *startSet->value;
return;
}
/* Get the operator. */
Bindings::iterator op = getAttr(
state,
state.sOperator,
args[0]->attrs,
"in the attrset passed as argument to builtins.genericClosure"
);
state.forceFunction(
*op->value,
noPos,
"while evaluating the 'operator' attribute passed as argument to builtins.genericClosure"
);
/* Construct the closure by applying the operator to elements of
`workSet', adding the result to `workSet', continuing until
no new elements are found. */
ValueList res;
// `doneKeys' doesn't need to be a GC root, because its values are
// reachable from res.
auto cmp = CompareValues(
state, noPos, "while comparing the `key` attributes of two genericClosure elements"
);
std::set<Value *, decltype(cmp)> doneKeys(cmp);
while (!workSet.empty()) {
Value * e = *(workSet.begin());
workSet.pop_front();
state.forceAttrs(
*e,
noPos,
"while evaluating one of the elements generated by (or initially passed to) "
"builtins.genericClosure"
);
Bindings::iterator key = getAttr(
state,
state.sKey,
e->attrs,
"in one of the attrsets generated by (or initially passed to) builtins.genericClosure"
);
state.forceValue(*key->value, noPos);
if (!doneKeys.insert(key->value).second) {
continue;
}
res.push_back(e);
/* Call the `operator' function with `e' as argument. */
Value newElements;
state.callFunction(*op->value, 1, &e, newElements, noPos);
state.forceList(
newElements,
noPos,
"while evaluating the return value of the `operator` passed to builtins.genericClosure"
);
/* Add the values returned by the operator to the work set. */
for (auto elem : newElements.listItems()) {
state.forceValue(*elem, noPos); // "while evaluating one one of the elements returned by
// the `operator` passed to builtins.genericClosure");
workSet.push_back(elem);
}
}
/* Create the result list. */
state.mkList(v, res.size());
unsigned int n = 0;
for (auto & i : res) {
v.listElems()[n++] = i;
}
}
PrimOp primop_genericClosure(PrimOp{
.name = "__genericClosure",
.args = {"attrset"},
.arity = 1,
.doc = R"(
Take an *attrset* with values named `startSet` and `operator` in order to
return a *list of attrsets* by starting with the `startSet` and recursively
applying the `operator` function to each `item`. The *attrsets* in the
`startSet` and the *attrsets* produced by `operator` must contain a value
named `key` which is comparable. The result is produced by calling `operator`
for each `item` with a value for `key` that has not been called yet including
newly produced `item`s. The function terminates when no new `item`s are
produced. The resulting *list of attrsets* contains only *attrsets* with a
unique key. For example,
```
builtins.genericClosure {
startSet = [ {key = 5;} ];
operator = item: [{
key = if (item.key / 2 ) * 2 == item.key
then item.key / 2
else 3 * item.key + 1;
}];
}
```
evaluates to
```
[ { key = 5; } { key = 16; } { key = 8; } { key = 4; } { key = 2; } { key = 1; } ]
```
)",
.fun = prim_genericClosure,
});
/**
* builtins.getAttr
*/
void prim_getAttr(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto attr = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.getAttr"
);
state.forceAttrs(
*args[1], pos, "while evaluating the second argument passed to builtins.getAttr"
);
Bindings::iterator i = getAttr(
state,
state.symbols.create(attr),
args[1]->attrs,
"in the attribute set under consideration"
);
// !!! add to stack trace?
if (state.countCalls && i->pos) {
state.attrSelects[i->pos]++;
}
state.forceValue(*i->value, pos);
v = *i->value;
}
PrimOp primop_getAttr({
.name = "__getAttr",
.args = {"s", "set"},
.doc = R"(
`getAttr` returns the attribute named *s* from *set*. Evaluation
aborts if the attribute doesnt exist. This is a dynamic version of
the `.` operator, since *s* is an expression rather than an
identifier.
)",
.fun = prim_getAttr,
});
/**
* builtins.groupBy
*/
static void prim_groupBy(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.groupBy"
);
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.groupBy"
);
ValueVectorMap attrs;
for (auto vElem : args[1]->listItems()) {
Value res;
state.callFunction(*args[0], *vElem, res, pos);
auto name = state.forceStringNoCtx(
res,
pos,
"while evaluating the return value of the grouping function passed to builtins.groupBy"
);
auto sym = state.symbols.create(name);
auto vector = attrs.try_emplace(sym, ValueVector()).first;
vector->second.push_back(vElem);
}
auto attrs2 = state.buildBindings(attrs.size());
for (auto & i : attrs) {
auto & list = attrs2.alloc(i.first);
auto size = i.second.size();
state.mkList(list, size);
memcpy(list.listElems(), i.second.data(), sizeof(Value *) * size);
}
v.mkAttrs(attrs2.alreadySorted());
}
PrimOp primop_groupBy({
.name = "__groupBy",
.args = {"f", "list"},
.doc = R"(
Groups elements of *list* together by the string returned from the
function *f* called on each element. It returns an attribute set
where each attribute value contains the elements of *list* that are
mapped to the same corresponding attribute name returned by *f*.
For example,
```nix
builtins.groupBy (builtins.substring 0 1) ["foo" "bar" "baz"]
```
evaluates to
```nix
{ b = [ "bar" "baz" ]; f = [ "foo" ]; }
```
)",
.fun = prim_groupBy,
});
/**
* builtins.hasAttr
*/
static void prim_hasAttr(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto attr = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.hasAttr"
);
state.forceAttrs(
*args[1], pos, "while evaluating the second argument passed to builtins.hasAttr"
);
v.mkBool(args[1]->attrs->find(state.symbols.create(attr)) != args[1]->attrs->end());
}
PrimOp primop_hasAttr({
.name = "__hasAttr",
.args = {"s", "set"},
.doc = R"(
`hasAttr` returns `true` if *set* has an attribute named *s*, and
`false` otherwise. This is a dynamic version of the `?` operator,
since *s* is an expression rather than an identifier.
)",
.fun = prim_hasAttr,
});
/**
* builtins.intersectAttrs
*/
static void prim_intersectAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(
*args[0], pos, "while evaluating the first argument passed to builtins.intersectAttrs"
);
state.forceAttrs(
*args[1], pos, "while evaluating the second argument passed to builtins.intersectAttrs"
);
Bindings & left = *args[0]->attrs;
Bindings & right = *args[1]->attrs;
auto attrs = state.buildBindings(std::min(left.size(), right.size()));
// The current implementation has good asymptotic complexity and is reasonably
// simple. Further optimization may be possible, but does not seem productive,
// considering the state of eval performance in 2022.
//
// I have looked for reusable and/or standard solutions and these are my
// findings:
//
// STL
// ===
// std::set_intersection is not suitable, as it only performs a simultaneous
// linear scan; not taking advantage of random access. This is O(n + m), so
// linear in the largest set, which is not acceptable for callPackage in Nixpkgs.
//
// Simultaneous scan, with alternating simple binary search
// ===
// One alternative algorithm scans the attrsets simultaneously, jumping
// forward using `lower_bound` in case of inequality. This should perform
// well on very similar sets, having a local and predictable access pattern.
// On dissimilar sets, it seems to need more comparisons than the current
// algorithm, as few consecutive attrs match. `lower_bound` could take
// advantage of the decreasing remaining search space, but this causes
// the medians to move, which can mean that they don't stay in the cache
// like they would with the current naive `find`.
//
// Double binary search
// ===
// The optimal algorithm may be "Double binary search", which doesn't
// scan at all, but rather divides both sets simultaneously.
// See "Fast Intersection Algorithms for Sorted Sequences" by Baeza-Yates et al.
// https://cs.uwaterloo.ca/~ajsaling/papers/intersection_alg_app10.pdf
// The only downsides I can think of are not having a linear access pattern
// for similar sets, and having to maintain a more intricate algorithm.
//
// Adaptive
// ===
// Finally one could run try a simultaneous scan, count misses and fall back
// to double binary search when the counter hit some threshold and/or ratio.
if (left.size() < right.size()) {
for (auto & l : left) {
Bindings::iterator r = right.find(l.name);
if (r != right.end()) {
attrs.insert(*r);
}
}
} else {
for (auto & r : right) {
Bindings::iterator l = left.find(r.name);
if (l != left.end()) {
attrs.insert(r);
}
}
}
v.mkAttrs(attrs.alreadySorted());
}
PrimOp primop_intersectAttrs({
.name = "__intersectAttrs",
.args = {"e1", "e2"},
.doc = R"(
Return a set consisting of the attributes in the set *e2* which have the
same name as some attribute in *e1*.
Performs in O(*n* log *m*) where *n* is the size of the smaller set and *m* the larger set's size.
)",
.fun = prim_intersectAttrs,
});
/**
* builtins.mapAttrs
*/
static void prim_mapAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(
*args[1], pos, "while evaluating the second argument passed to builtins.mapAttrs"
);
auto attrs = state.buildBindings(args[1]->attrs->size());
for (auto & i : *args[1]->attrs) {
Value * vName = state.allocValue();
Value * vFun2 = state.allocValue();
vName->mkString(state.symbols[i.name]);
vFun2->mkApp(args[0], vName);
attrs.alloc(i.name).mkApp(vFun2, i.value);
}
v.mkAttrs(attrs.alreadySorted());
}
PrimOp primop_mapAttrs({
.name = "__mapAttrs",
.args = {"f", "attrset"},
.doc = R"(
Apply function *f* to every element of *attrset*. For example,
```nix
builtins.mapAttrs (name: value: value * 10) { a = 1; b = 2; }
```
evaluates to `{ a = 10; b = 20; }`.
)",
.fun = prim_mapAttrs,
});
/**
* builtins.mapAttrs
*/
static void prim_removeAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(
*args[0], pos, "while evaluating the first argument passed to builtins.removeAttrs"
);
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.removeAttrs"
);
/* Get the attribute names to be removed.
We keep them as Attrs instead of Symbols so std::set_difference
can be used to remove them from attrs[0]. */
// 64: large enough to fit the attributes of a derivation
boost::container::small_vector<Attr, 64> names;
names.reserve(args[1]->listSize());
for (auto elem : args[1]->listItems()) {
state.forceStringNoCtx(
*elem,
pos,
"while evaluating the values of the second argument passed to builtins.removeAttrs"
);
names.emplace_back(state.symbols.create(elem->string.s), nullptr);
}
std::sort(names.begin(), names.end());
/* Copy all attributes not in that set. Note that we don't need
to sort v.attrs because it's a subset of an already sorted
vector. */
auto attrs = state.buildBindings(args[0]->attrs->size());
std::set_difference(
args[0]->attrs->begin(),
args[0]->attrs->end(),
names.begin(),
names.end(),
std::back_inserter(attrs)
);
v.mkAttrs(attrs.alreadySorted());
}
PrimOp primop_removeAttrs({
.name = "removeAttrs",
.args = {"set", "list"},
.doc = R"(
Remove the attributes listed in *list* from *set*. The attributes
dont have to exist in *set*. For instance,
```nix
removeAttrs { x = 1; y = 2; z = 3; } [ "a" "x" "z" ]
```
evaluates to `{ y = 2; }`.
)",
.fun = prim_removeAttrs,
});
/**
* builtins.zipAttrsWith
*/
static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
// we will first count how many values are present for each given key.
// we then allocate a single attrset and pre-populate it with lists of
// appropriate sizes, stash the pointers to the list elements of each,
// and populate the lists. after that we replace the list in the every
// attribute with the merge function application. this way we need not
// use (slightly slower) temporary storage the GC does not know about.
std::map<Symbol, std::pair<size_t, Value **>> attrsSeen;
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.zipAttrsWith"
);
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.zipAttrsWith"
);
const auto listSize = args[1]->listSize();
const auto listElems = args[1]->listElems();
for (unsigned int n = 0; n < listSize; ++n) {
Value * vElem = listElems[n];
state.forceAttrs(
*vElem,
noPos,
"while evaluating a value of the list passed as second argument to "
"builtins.zipAttrsWith"
);
for (auto & attr : *vElem->attrs) {
attrsSeen[attr.name].first++;
}
}
auto attrs = state.buildBindings(attrsSeen.size());
for (auto & [sym, elem] : attrsSeen) {
auto & list = attrs.alloc(sym);
state.mkList(list, elem.first);
elem.second = list.listElems();
}
v.mkAttrs(attrs.alreadySorted());
for (unsigned int n = 0; n < listSize; ++n) {
Value * vElem = listElems[n];
for (auto & attr : *vElem->attrs) {
*attrsSeen[attr.name].second++ = attr.value;
}
}
for (auto & attr : *v.attrs) {
auto name = state.allocValue();
name->mkString(state.symbols[attr.name]);
auto call1 = state.allocValue();
call1->mkApp(args[0], name);
auto call2 = state.allocValue();
call2->mkApp(call1, attr.value);
attr.value = call2;
}
}
PrimOp primop_zipAttrsWith({
.name = "__zipAttrsWith",
.args = {"f", "list"},
.doc = R"(
Transpose a list of attribute sets into an attribute set of lists,
then apply `mapAttrs`.
`f` receives two arguments: the attribute name and a non-empty
list of all values encountered for that attribute name.
The result is an attribute set where the attribute names are the
union of the attribute names in each element of `list`. The attribute
values are the return values of `f`.
```nix
builtins.zipAttrsWith
(name: values: { inherit name values; })
[ { a = "x"; } { a = "y"; b = "z"; } ]
```
evaluates to
```
{
a = { name = "a"; values = [ "x" "y" ]; };
b = { name = "b"; values = [ "z" ]; };
}
```
)",
.fun = prim_zipAttrsWith,
});
}
+211 -442
View File
@@ -1,190 +1,142 @@
#include <map> // for map, _Rb_tree_const_iterator
#include <memory> // for shared_ptr
#include <string> // for basic_string, operator<=>, char...
#include <utility> // for move, pair
#include <variant> // for visit, get_if
#include <vector> // for vector
#include "attr-set.hh" // for Attr, Bindings, BindingsBuilder
#include "config.hh" // for Setting, ExperimentalFeatureSet...
#include "derivations.hh" // for isDerivation
#include "derived-path.hh" // for DerivedPath, makeConstantStoreP...
#include "downstream-placeholder.hh" // for DownstreamPlaceholder
#include "error.hh" // for Error
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval-settings.hh" // for EvalSettings, evalSettings
#include "eval.hh" // for EvalState, PrimOp
#include "experimental-features.hh" // for ExperimentalFeature, Xp
#include "fmt.hh" // for HintFmt
#include "globals.hh" // for Settings, settings
#include "outputs-spec.hh" // for OutputsSpec
#include "path.hh" // for StorePath, StorePathSet
#include "pos-idx.hh" // for PosIdx, noPos
#include "ref.hh" // for ref
#include "store-api.hh" // for Store, resolveDerivedPath, copy...
#include "symbol-table.hh" // for SymbolStr, Symbol, SymbolTable
#include "types.hh" // for StringMap, BackedStringView
#include "value.hh" // for Value
#include "value/context.hh" // for NixStringContextElem, NixString...
#include "primops.hh"
#include "eval-inline.hh"
#include "derivations.hh"
#include "store-api.hh"
namespace nix {
StringMap EvalState::realiseContext(const NixStringContext & context)
{
std::vector<DerivedPath::Built> drvs;
StringMap res;
for (auto & c : context) {
auto ensureValid = [&](const StorePath & p) {
if (!store->isValidPath(p)) {
error<InvalidPathError>(store->printStorePath(p)).debugThrow();
}
};
std::visit(
overloaded{
[&](const NixStringContextElem::Built & b) {
drvs.push_back(DerivedPath::Built{
.drvPath = b.drvPath,
.outputs = OutputsSpec::Names{b.output},
});
ensureValid(b.drvPath->getBaseStorePath());
},
[&](const NixStringContextElem::Opaque & o) {
auto ctxS = store->printStorePath(o.path);
res.insert_or_assign(ctxS, ctxS);
ensureValid(o.path);
},
[&](const NixStringContextElem::DrvDeep & d) {
/* Treat same as Opaque */
auto ctxS = store->printStorePath(d.drvPath);
res.insert_or_assign(ctxS, ctxS);
ensureValid(d.drvPath);
},
},
c.raw
);
}
if (drvs.empty()) {
return {};
}
if (!evalSettings.enableImportFromDerivation) {
error<EvalError>(
"cannot build '%1%' during evaluation because the option "
"'allow-import-from-derivation' is disabled",
drvs.begin()->to_string(*store)
)
.debugThrow();
}
/* Build/substitute the context. */
std::vector<DerivedPath> buildReqs;
for (auto & d : drvs) {
buildReqs.emplace_back(DerivedPath{d});
}
buildStore->buildPaths(buildReqs, bmNormal, store);
StorePathSet outputsToCopyAndAllow;
for (auto & drv : drvs) {
auto outputs = resolveDerivedPath(*buildStore, drv, &*store);
for (auto & [outputName, outputPath] : outputs) {
outputsToCopyAndAllow.insert(outputPath);
/* Get all the output paths corresponding to the placeholders we had */
if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) {
res.insert_or_assign(
DownstreamPlaceholder::fromSingleDerivedPathBuilt(SingleDerivedPath::Built{
.drvPath = drv.drvPath,
.output = outputName,
})
.render(),
buildStore->printStorePath(outputPath)
);
}
}
}
if (store != buildStore) {
copyClosure(*buildStore, *store, outputsToCopyAndAllow);
}
if (allowedPaths) {
for (auto & outputPath : outputsToCopyAndAllow) {
/* Add the output of this derivations to the allowed
paths. */
allowPath(outputPath);
}
}
return res;
}
/**
* builtins.addDrvOutputDependencies
*/
static void
prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, Value ** args, Value & v)
static void prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(
pos,
*args[0],
context,
"while evaluating the argument passed to builtins.addDrvOutputDependencies"
);
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardStringContext");
v.mkString(*s);
}
auto contextSize = context.size();
if (contextSize != 1) {
state
.error<EvalError>(
"context of string '%s' must have exactly one element, but has %d", *s, contextSize
)
.atPos(pos)
.debugThrow();
static RegisterPrimOp primop_unsafeDiscardStringContext({
.name = "__unsafeDiscardStringContext",
.arity = 1,
.fun = prim_unsafeDiscardStringContext
});
static void prim_hasContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
state.forceString(*args[0], context, pos, "while evaluating the argument passed to builtins.hasContext");
v.mkBool(!context.empty());
}
static RegisterPrimOp primop_hasContext({
.name = "__hasContext",
.args = {"s"},
.doc = R"(
Return `true` if string *s* has a non-empty context.
The context can be obtained with
[`getContext`](#builtins-getContext).
> **Example**
>
> Many operations require a string context to be empty because they are intended only to work with "regular" strings, and also to help users avoid unintentionally losing track of string context elements.
> `builtins.hasContext` can help create better domain-specific errors in those case.
>
> ```nix
> name: meta:
>
> if builtins.hasContext name
> then throw "package name cannot contain string context"
> else { ${name} = meta; }
> ```
)",
.fun = prim_hasContext
});
static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardOutputDependency");
NixStringContext context2;
for (auto && c : context) {
if (auto * ptr = std::get_if<NixStringContextElem::DrvDeep>(&c.raw)) {
context2.emplace(NixStringContextElem::Opaque {
.path = ptr->drvPath
});
} else {
/* Can reuse original item */
context2.emplace(std::move(c).raw);
}
}
NixStringContext context2{
(NixStringContextElem{std::visit(
overloaded{
[&](const NixStringContextElem::Opaque & c) -> NixStringContextElem::DrvDeep {
if (!c.path.isDerivation()) {
state
.error<EvalError>(
"path '%s' is not a derivation", state.store->printStorePath(c.path)
)
.atPos(pos)
.debugThrow();
}
return NixStringContextElem::DrvDeep{
.drvPath = c.path,
};
},
[&](const NixStringContextElem::Built & c) -> NixStringContextElem::DrvDeep {
state
.error<EvalError>(
"`addDrvOutputDependencies` can only act on derivations, not on a "
"derivation output such as '%1%'",
c.output
)
.atPos(pos)
.debugThrow();
},
[&](const NixStringContextElem::DrvDeep & c) -> NixStringContextElem::DrvDeep {
/* Reuse original item because we want this to be idempotent. */
return std::move(c);
},
v.mkString(*s, context2);
}
static RegisterPrimOp primop_unsafeDiscardOutputDependency({
.name = "__unsafeDiscardOutputDependency",
.args = {"s"},
.doc = R"(
Create a copy of the given string where every "derivation deep" string context element is turned into a constant string context element.
This is the opposite of [`builtins.addDrvOutputDependencies`](#builtins-addDrvOutputDependencies).
This is unsafe because it allows us to "forget" store objects we would have otherwise refered to with the string context,
whereas Nix normally tracks all dependencies consistently.
Safe operations "grow" but never "shrink" string contexts.
[`builtins.addDrvOutputDependencies`] in contrast is safe because "derivation deep" string context element always refers to the underlying derivation (among many more things).
Replacing a constant string context element with a "derivation deep" element is a safe operation that just enlargens the string context without forgetting anything.
[`builtins.addDrvOutputDependencies`]: #builtins-addDrvOutputDependencies
)",
.fun = prim_unsafeDiscardOutputDependency
});
static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.addDrvOutputDependencies");
auto contextSize = context.size();
if (contextSize != 1) {
state.error<EvalError>(
"context of string '%s' must have exactly one element, but has %d",
*s,
contextSize
).atPos(pos).debugThrow();
}
NixStringContext context2 {
(NixStringContextElem { std::visit(overloaded {
[&](const NixStringContextElem::Opaque & c) -> NixStringContextElem::DrvDeep {
if (!c.path.isDerivation()) {
state.error<EvalError>(
"path '%s' is not a derivation",
state.store->printStorePath(c.path)
).atPos(pos).debugThrow();
}
return NixStringContextElem::DrvDeep {
.drvPath = c.path,
};
},
context.begin()->raw
)}),
[&](const NixStringContextElem::Built & c) -> NixStringContextElem::DrvDeep {
state.error<EvalError>(
"`addDrvOutputDependencies` can only act on derivations, not on a derivation output such as '%1%'",
c.output
).atPos(pos).debugThrow();
},
[&](const NixStringContextElem::DrvDeep & c) -> NixStringContextElem::DrvDeep {
/* Reuse original item because we want this to be idempotent. */
return std::move(c);
},
}, context.begin()->raw) }),
};
v.mkString(*s, context2);
}
PrimOp primop_addDrvOutputDependencies(
{.name = "__addDrvOutputDependencies",
.args = {"s"},
.doc = R"(
static RegisterPrimOp primop_addDrvOutputDependencies({
.name = "__addDrvOutputDependencies",
.args = {"s"},
.doc = R"(
Create a copy of the given string where a single constant string context element is turned into a "derivation deep" string context element.
The store path that is the constant string context element should point to a valid derivation, and end in `.drv`.
@@ -194,153 +146,9 @@ PrimOp primop_addDrvOutputDependencies(
This is the opposite of [`builtins.unsafeDiscardOutputDependency`](#builtins-unsafeDiscardOutputDependency).
)",
.fun = prim_addDrvOutputDependencies}
);
/**
* builtins.addErrorContext
*/
static void prim_addErrorContext(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
try {
state.forceValue(*args[1], pos);
v = *args[1];
} catch (Error & e) {
NixStringContext context;
auto message =
state
.coerceToString(
pos,
*args[0],
context,
"while evaluating the error message passed to builtins.addErrorContext",
false,
false
)
.toOwned();
e.addTrace(nullptr, HintFmt(message));
throw;
}
}
PrimOp primop_addErrorContext(PrimOp{
.name = "__addErrorContext",
.arity = 2,
.fun = prim_addErrorContext,
.fun = prim_addDrvOutputDependencies
});
/**
* builtins.appendContext
*/
static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto orig = state.forceString(
*args[0],
context,
noPos,
"while evaluating the first argument passed to builtins.appendContext"
);
state.forceAttrs(
*args[1], pos, "while evaluating the second argument passed to builtins.appendContext"
);
auto sPath = state.symbols.create("path");
auto sAllOutputs = state.symbols.create("allOutputs");
for (auto & i : *args[1]->attrs) {
const auto & name = state.symbols[i.name];
if (!state.store->isStorePath(name)) {
state.error<EvalError>("context key '%s' is not a store path", name)
.atPos(i.pos)
.debugThrow();
}
auto namePath = state.store->parseStorePath(name);
if (!settings.readOnlyMode) {
state.store->ensurePath(namePath);
}
state.forceAttrs(*i.value, i.pos, "while evaluating the value of a string context");
auto iter = i.value->attrs->find(sPath);
if (iter != i.value->attrs->end()) {
if (state.forceBool(
*iter->value,
iter->pos,
"while evaluating the `path` attribute of a string context"
))
{
context.emplace(NixStringContextElem::Opaque{
.path = namePath,
});
}
}
iter = i.value->attrs->find(sAllOutputs);
if (iter != i.value->attrs->end()) {
if (state.forceBool(
*iter->value,
iter->pos,
"while evaluating the `allOutputs` attribute of a string context"
))
{
if (!isDerivation(name)) {
state
.error<EvalError>(
"tried to add all-outputs context of %s, which is not a derivation, to "
"a string",
name
)
.atPos(i.pos)
.debugThrow();
}
context.emplace(NixStringContextElem::DrvDeep{
.drvPath = namePath,
});
}
}
iter = i.value->attrs->find(state.sOutputs);
if (iter != i.value->attrs->end()) {
state.forceList(
*iter->value,
iter->pos,
"while evaluating the `outputs` attribute of a string context"
);
if (iter->value->listSize() && !isDerivation(name)) {
state
.error<EvalError>(
"tried to add derivation output context of %s, which is not a derivation, "
"to a string",
name
)
.atPos(i.pos)
.debugThrow();
}
for (auto elem : iter->value->listItems()) {
auto outputName = state.forceStringNoCtx(
*elem, iter->pos, "while evaluating an output name within a string context"
);
context.emplace(NixStringContextElem::Built{
.drvPath = makeConstantStorePathRef(namePath),
.output = std::string{outputName},
});
}
}
}
v.mkString(orig, context);
}
PrimOp primop_appendContext({
.name = "__appendContext",
.arity = 2,
.fun = prim_appendContext,
});
/**
* builtins.getContext
*/
/* Extract the context of a string as a structured Nix value.
@@ -361,37 +169,31 @@ PrimOp primop_appendContext({
Note that for a given path any combination of the above attributes
may be present.
*/
static void prim_getContext(EvalState & state, const PosIdx pos, Value ** args, Value & v)
static void prim_getContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
struct ContextInfo
{
struct ContextInfo {
bool path = false;
bool allOutputs = false;
Strings outputs;
};
NixStringContext context;
state.forceString(
*args[0], context, pos, "while evaluating the argument passed to builtins.getContext"
);
state.forceString(*args[0], context, pos, "while evaluating the argument passed to builtins.getContext");
auto contextInfos = std::map<StorePath, ContextInfo>();
for (auto && i : context) {
std::visit(
overloaded{
[&](NixStringContextElem::DrvDeep && d) {
contextInfos[std::move(d.drvPath)].allOutputs = true;
},
[&](NixStringContextElem::Built && b) {
// FIXME should eventually show string context as is, no
// resolving here.
auto drvPath = resolveDerivedPath(*state.store, *b.drvPath);
contextInfos[std::move(drvPath)].outputs.emplace_back(std::move(b.output));
},
[&](NixStringContextElem::Opaque && o) {
contextInfos[std::move(o.path)].path = true;
},
std::visit(overloaded {
[&](NixStringContextElem::DrvDeep && d) {
contextInfos[std::move(d.drvPath)].allOutputs = true;
},
((NixStringContextElem &&) i).raw
);
[&](NixStringContextElem::Built && b) {
// FIXME should eventually show string context as is, no
// resolving here.
auto drvPath = resolveDerivedPath(*state.store, *b.drvPath);
contextInfos[std::move(drvPath)].outputs.emplace_back(std::move(b.output));
},
[&](NixStringContextElem::Opaque && o) {
contextInfos[std::move(o.path)].path = true;
},
}, ((NixStringContextElem &&) i).raw);
}
auto attrs = state.buildBindings(contextInfos.size());
@@ -400,18 +202,15 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value ** args,
auto sAllOutputs = state.symbols.create("allOutputs");
for (const auto & info : contextInfos) {
auto infoAttrs = state.buildBindings(3);
if (info.second.path) {
if (info.second.path)
infoAttrs.alloc(sPath).mkBool(true);
}
if (info.second.allOutputs) {
if (info.second.allOutputs)
infoAttrs.alloc(sAllOutputs).mkBool(true);
}
if (!info.second.outputs.empty()) {
auto & outputsVal = infoAttrs.alloc(state.sOutputs);
state.mkList(outputsVal, info.second.outputs.size());
for (const auto & [i, output] : enumerate(info.second.outputs)) {
for (const auto & [i, output] : enumerate(info.second.outputs))
(outputsVal.listElems()[i] = state.allocValue())->mkString(output);
}
}
attrs.alloc(state.store->printStorePath(info.first)).mkAttrs(infoAttrs);
}
@@ -419,7 +218,7 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value ** args,
v.mkAttrs(attrs);
}
PrimOp primop_getContext({
static RegisterPrimOp primop_getContext({
.name = "__getContext",
.args = {"s"},
.doc = R"(
@@ -441,114 +240,84 @@ PrimOp primop_getContext({
{ "/nix/store/arhvjaf6zmlyn8vh8fgn55rpwnxq0n7l-a.drv" = { outputs = [ "out" ]; }; }
```
)",
.fun = prim_getContext,
.fun = prim_getContext
});
/**
* builtins.hasContext
*/
static void prim_hasContext(EvalState & state, const PosIdx pos, Value ** args, Value & v)
/* Append the given context to a given string.
See the commentary above unsafeGetContext for details of the
context representation.
*/
static void prim_appendContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
state.forceString(
*args[0], context, pos, "while evaluating the argument passed to builtins.hasContext"
);
v.mkBool(!context.empty());
}
auto orig = state.forceString(*args[0], context, noPos, "while evaluating the first argument passed to builtins.appendContext");
PrimOp primop_hasContext(
{.name = "__hasContext",
.args = {"s"},
.doc = R"(
Return `true` if string *s* has a non-empty context.
The context can be obtained with
[`getContext`](#builtins-getContext).
state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.appendContext");
> **Example**
>
> Many operations require a string context to be empty because they are intended only to work with "regular" strings, and also to help users avoid unintentionally losing track of string context elements.
> `builtins.hasContext` can help create better domain-specific errors in those case.
>
> ```nix
> name: meta:
>
> if builtins.hasContext name
> then throw "package name cannot contain string context"
> else { ${name} = meta; }
> ```
)",
.fun = prim_hasContext}
);
auto sPath = state.symbols.create("path");
auto sAllOutputs = state.symbols.create("allOutputs");
for (auto & i : *args[1]->attrs) {
const auto & name = state.symbols[i.name];
if (!state.store->isStorePath(name))
state.error<EvalError>(
"context key '%s' is not a store path",
name
).atPos(i.pos).debugThrow();
auto namePath = state.store->parseStorePath(name);
if (!settings.readOnlyMode)
state.store->ensurePath(namePath);
state.forceAttrs(*i.value, i.pos, "while evaluating the value of a string context");
auto iter = i.value->attrs->find(sPath);
if (iter != i.value->attrs->end()) {
if (state.forceBool(*iter->value, iter->pos, "while evaluating the `path` attribute of a string context"))
context.emplace(NixStringContextElem::Opaque {
.path = namePath,
});
}
/**
* builtins.unsafeDiscardOutputDependency
*/
iter = i.value->attrs->find(sAllOutputs);
if (iter != i.value->attrs->end()) {
if (state.forceBool(*iter->value, iter->pos, "while evaluating the `allOutputs` attribute of a string context")) {
if (!isDerivation(name)) {
state.error<EvalError>(
"tried to add all-outputs context of %s, which is not a derivation, to a string",
name
).atPos(i.pos).debugThrow();
}
context.emplace(NixStringContextElem::DrvDeep {
.drvPath = namePath,
});
}
}
static void
prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(
pos,
*args[0],
context,
"while evaluating the argument passed to builtins.unsafeDiscardOutputDependency"
);
NixStringContext context2;
for (auto && c : context) {
if (auto * ptr = std::get_if<NixStringContextElem::DrvDeep>(&c.raw)) {
context2.emplace(NixStringContextElem::Opaque{.path = ptr->drvPath});
} else {
/* Can reuse original item */
context2.emplace(std::move(c).raw);
iter = i.value->attrs->find(state.sOutputs);
if (iter != i.value->attrs->end()) {
state.forceList(*iter->value, iter->pos, "while evaluating the `outputs` attribute of a string context");
if (iter->value->listSize() && !isDerivation(name)) {
state.error<EvalError>(
"tried to add derivation output context of %s, which is not a derivation, to a string",
name
).atPos(i.pos).debugThrow();
}
for (auto elem : iter->value->listItems()) {
auto outputName = state.forceStringNoCtx(*elem, iter->pos, "while evaluating an output name within a string context");
context.emplace(NixStringContextElem::Built {
.drvPath = makeConstantStorePathRef(namePath),
.output = std::string { outputName },
});
}
}
}
v.mkString(*s, context2);
v.mkString(orig, context);
}
PrimOp primop_unsafeDiscardOutputDependency(
{.name = "__unsafeDiscardOutputDependency",
.args = {"s"},
.doc = R"(
Create a copy of the given string where every "derivation deep" string context element is turned into a constant string context element.
This is the opposite of [`builtins.addDrvOutputDependencies`](#builtins-addDrvOutputDependencies).
This is unsafe because it allows us to "forget" store objects we would have otherwise refered to with the string context,
whereas Nix normally tracks all dependencies consistently.
Safe operations "grow" but never "shrink" string contexts.
[`builtins.addDrvOutputDependencies`] in contrast is safe because "derivation deep" string context element always refers to the underlying derivation (among many more things).
Replacing a constant string context element with a "derivation deep" element is a safe operation that just enlargens the string context without forgetting anything.
[`builtins.addDrvOutputDependencies`]: #builtins-addDrvOutputDependencies
)",
.fun = prim_unsafeDiscardOutputDependency}
);
/**
* builtins. unsafeDiscardStringContext
*/
static void
prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(
pos,
*args[0],
context,
"while evaluating the argument passed to builtins.unsafeDiscardStringContext"
);
v.mkString(*s);
}
PrimOp primop_unsafeDiscardStringContext({
.name = "__unsafeDiscardStringContext",
.arity = 1,
.fun = prim_unsafeDiscardStringContext,
static RegisterPrimOp primop_appendContext({
.name = "__appendContext",
.arity = 2,
.fun = prim_appendContext
});
}
-164
View File
@@ -1,164 +0,0 @@
#include <list> // for list
#include <string> // for basic_string
#include "attr-set.hh" // for BindingsBuilder
#include "error.hh" // for Trace, Error, lvlInfo, ErrorInfo
#include "eval-error.hh" // for Abort, ThrownError, EvalErrorBuilder
#include "eval-settings.hh" // for EvalSettings, evalSettings
#include "eval.hh" // for EvalState, PrimOp, DebugTrace, ValMap
#include "fmt.hh" // for HintFmt
#include "pos-idx.hh" // for PosIdx
#include "pos-table.hh" // for PosTable
#include "ref.hh" // for ref
#include "symbol-table.hh" // for Symbol
#include "types.hh" // for BackedStringView, MaintainCount
#include "value.hh" // for Value
#include "value/context.hh" // for NixStringContext
namespace nix {
enum class ReplExitStatus;
/**
* builtins.abort
*/
static void prim_abort(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto s = state
.coerceToString(
pos,
*args[0],
context,
"while evaluating the error message passed to builtins.abort"
)
.toOwned();
state.error<Abort>("evaluation aborted with the following error message: '%1%'", s)
.debugThrow();
}
PrimOp primop_abort({
.name = "abort",
.args = {"s"},
.doc = R"(
Abort Nix expression evaluation and print the error message *s*.
)",
.fun = prim_abort,
});
/**
* builtins.break
*/
static void prim_break(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
if (state.debugRepl && !state.debugTraces.empty()) {
auto error = Error(ErrorInfo{
.level = lvlInfo,
.msg = HintFmt("breakpoint reached"),
.pos = state.positions[pos],
});
auto & dt = state.debugTraces.front();
state.runDebugRepl(&error, dt.env, dt.expr);
}
// Return the value we were passed.
v = *args[0];
}
PrimOp primop_break({
.name = "break",
.args = {"v"},
.doc = R"(
In debug mode (enabled using `--debugger`), pause Nix expression evaluation and enter the REPL.
Otherwise, return the argument `v`.
)",
.fun = prim_break,
});
/**
* builtins.throw
*/
static void prim_throw(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto s =
state
.coerceToString(
pos, *args[0], context, "while evaluating the error message passed to builtin.throw"
)
.toOwned();
state.error<ThrownError>(s).debugThrow();
}
PrimOp primop_throw({
.name = "throw",
.args = {"s"},
.doc = R"(
Throw an error message *s*. This usually aborts Nix expression
evaluation, but in `nix-env -qa` and other commands that try to
evaluate a set of derivations to get information about those
derivations, a derivation that throws an error is silently skipped
(which is not the case for `abort`).
)",
.fun = prim_throw,
});
/**
* builtins.tryEval
*/
static void prim_tryEval(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto attrs = state.buildBindings(2);
/* increment state.trylevel, and decrement it when this function returns. */
MaintainCount trylevel(state.trylevel);
ReplExitStatus (*savedDebugRepl)(ref<EvalState> es, const ValMap & extraEnv) = nullptr;
if (state.debugRepl && evalSettings.ignoreExceptionsDuringTry) {
/* to prevent starting the repl from exceptions withing a tryEval, null it. */
savedDebugRepl = state.debugRepl;
state.debugRepl = nullptr;
}
try {
state.forceValue(*args[0], pos);
attrs.insert(state.sValue, args[0]);
attrs.alloc("success").mkBool(true);
} catch (AssertionError & e) {
attrs.alloc(state.sValue).mkBool(false);
attrs.alloc("success").mkBool(false);
}
// restore the debugRepl pointer if we saved it earlier.
if (savedDebugRepl) {
state.debugRepl = savedDebugRepl;
}
v.mkAttrs(attrs);
}
PrimOp primop_tryEval({
.name = "__tryEval",
.args = {"e"},
.doc = R"(
Try to shallowly evaluate *e*. Return a set containing the
attributes `success` (`true` if *e* evaluated successfully,
`false` if an error was thrown) and `value`, equalling *e* if
successful and `false` otherwise. `tryEval` will only prevent
errors created by `throw` or `assert` from being thrown.
Errors `tryEval` will not catch are for example those created
by `abort` and type errors generated by builtins. Also note that
this doesn't evaluate *e* deeply, so `let e = { x = throw ""; };
in (builtins.tryEval e).success` will be `true`. Using
`builtins.deepSeq` one can get the expected result:
`let e = { x = throw ""; }; in
(builtins.tryEval (builtins.deepSeq e e)).success` will be
`false`.
)",
.fun = prim_tryEval,
});
}
-181
View File
@@ -1,181 +0,0 @@
#include <list> // for list
#include <string> // for basic_string
#include "attr-set.hh" // for Bindings
#include "eval-settings.hh" // for EvalSettings, evalSettings
#include "eval.hh" // for EvalState, PrimOp, DebugTrace
#include "logging.hh" // for Logger, printError
#include "pos-idx.hh" // for PosIdx
#include "pos-table.hh" // for PosTable
#include "position.hh" // for Pos
#include "primops.hh" // for makePositionThunks, prim_second, prim_trace
#include "print.hh" // for ValuePrinter
#include "symbol-table.hh" // for SymbolTable
#include "value.hh" // for Value, nString
namespace nix {
// access to exact position information (ie, line and colum numbers) is deferred
// due to the cost associated with calculating that information and how rarely
// it is used in practice. this is achieved by creating thunks to otherwise
// inaccessible primops that are not exposed as __op or under builtins to turn
// the internal PosIdx back into a line and column number, respectively. exposing
// these primops in any way would at best be not useful and at worst create wildly
// indeterministic eval results depending on parse order of files.
//
// in a simpler world this would instead be implemented as another kind of thunk,
// but each type of thunk has an associated runtime cost in the current evaluator.
// as with black holes this cost is too high to justify another thunk type to check
// for in the very hot path that is forceValue.
static struct LazyPosAcessors
{
PrimOp primop_lineOfPos{
.arity = 1,
.fun = [](EvalState & state, PosIdx pos, Value ** args, Value & v
) { v.mkInt(state.positions[PosIdx(args[0]->integer)].line); }
};
PrimOp primop_columnOfPos{
.arity = 1,
.fun = [](EvalState & state, PosIdx pos, Value ** args, Value & v
) { v.mkInt(state.positions[PosIdx(args[0]->integer)].column); }
};
Value lineOfPos, columnOfPos;
LazyPosAcessors()
{
lineOfPos.mkPrimOp(&primop_lineOfPos);
columnOfPos.mkPrimOp(&primop_columnOfPos);
}
void operator()(EvalState & state, const PosIdx pos, Value & line, Value & column)
{
Value * posV = state.allocValue();
posV->mkInt(pos.id);
line.mkApp(&lineOfPos, posV);
column.mkApp(&columnOfPos, posV);
}
} makeLazyPosAccessors;
void makePositionThunks(EvalState & state, const PosIdx pos, Value & line, Value & column)
{
makeLazyPosAccessors(state, pos, line, column);
}
/**
* Takes two arguments and evaluates to the second one. Used as the
* builtins.traceVerbose implementation when --trace-verbose is not enabled
*/
void prim_second(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[1], pos);
v = *args[1];
}
/**
* builtins.deepSeq
*/
static void prim_deepSeq(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValueDeep(*args[0]);
state.forceValue(*args[1], pos);
v = *args[1];
}
PrimOp primop_deepSeq({
.name = "__deepSeq",
.args = {"e1", "e2"},
.doc = R"(
This is like `seq e1 e2`, except that *e1* is evaluated *deeply*:
if its a list or set, its elements or attributes are also
evaluated recursively.
)",
.fun = prim_deepSeq,
});
/**
* builtins.seq
*/
static void prim_seq(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
state.forceValue(*args[1], pos);
v = *args[1];
}
PrimOp primop_seq({
.name = "__seq",
.args = {"e1", "e2"},
.doc = R"(
Evaluate *e1*, then evaluate and return *e2*. This ensures that a
computation is strict in the value of *e1*.
)",
.fun = prim_seq,
});
/**
* builtins.trace
*/
void prim_trace(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
if (args[0]->type() == nString) {
printError("trace: %1%", args[0]->string.s);
} else {
printError("trace: %1%", ValuePrinter(state, *args[0]));
}
if (evalSettings.builtinsTraceDebugger && state.debugRepl && !state.debugTraces.empty()) {
const DebugTrace & last = state.debugTraces.front();
state.runDebugRepl(nullptr, last.env, last.expr);
}
state.forceValue(*args[1], pos);
v = *args[1];
}
PrimOp primop_trace({
.name = "__trace",
.args = {"e1", "e2"},
.doc = R"(
Evaluate *e1* and print its abstract syntax representation on
standard error. Then return *e2*. This function is useful for
debugging.
If the
[`debugger-on-trace`](@docroot@/command-ref/conf-file.md#conf-debugger-on-trace)
option is set to `true` and the `--debugger` flag is given, the
interactive debugger will be started when `trace` is called (like
[`break`](@docroot@/language/builtins.md#builtins-break)).
)",
.fun = prim_trace,
});
/**
* builtins.unsafeGetAttrPos
*/
static void prim_unsafeGetAttrPos(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto attr = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.unsafeGetAttrPos"
);
state.forceAttrs(
*args[1], pos, "while evaluating the second argument passed to builtins.unsafeGetAttrPos"
);
Bindings::iterator i = args[1]->attrs->find(state.symbols.create(attr));
if (i == args[1]->attrs->end()) {
v.mkNull();
} else {
state.mkPos(v, i->pos);
}
}
PrimOp primop_unsafeGetAttrPos(PrimOp{
.name = "__unsafeGetAttrPos",
.arity = 2,
.fun = prim_unsafeGetAttrPos,
});
}
-429
View File
@@ -1,429 +0,0 @@
#include <map> // for _Rb_tree_const_ite...
#include <memory> // for allocator, shared_ptr
#include <nlohmann/json.hpp> // for basic_json
#include <nlohmann/json_fwd.hpp> // for json
#include <optional> // for optional, operator==
#include <set> // for set
#include <string> // for char_traits, basic...
#include <string_view> // for operator==, basic_...
#include <utility> // for move, pair
#include <variant> // for visit
#include "attr-set.hh" // for Attr, Bindings
#include "config.hh" // for ExperimentalFeatur...
#include "content-address.hh" // for ContentAddressMethod
#include "derivations.hh" // for Derivation, Deriva...
#include "derived-path-map.hh" // for DerivedPathMap
#include "error.hh" // for Error, lvlChatty
#include "eval-error.hh" // for EvalError, EvalErr...
#include "eval.hh" // for EvalState, PrimOp
#include "experimental-features.hh" // for ExperimentalFeature
#include "fmt.hh" // for HintFmt
#include "hash.hh" // for parseHashTypeOpt
#include "logging.hh" // for Logger, printMsg
#include "path.hh" // for StorePath, StorePa...
#include "pos-idx.hh" // for PosIdx, noPos
#include "pos-table.hh" // for PosTable
#include "position.hh" // for Pos, operator<<
#include "primops.hh" // for getAttr, mkOutputS...
#include "ref.hh" // for ref
#include "store-api.hh" // for Store
#include "strings.hh" // for tokenizeString
#include "symbol-table.hh" // for Symbol, SymbolStr
#include "sync.hh" // for Sync
#include "types.hh" // for Strings, BackedStr...
#include "value-to-json.hh" // for printValueAsJSON
#include "value.hh" // for Value, nNull
#include "value/context.hh" // for NixStringContextElem
namespace nix {
static void derivationStrictInternal(EvalState & state, const std::string &
drvName, Bindings * attrs, Value & v)
{
/* Check whether attributes should be passed as a JSON file. */
using nlohmann::json;
std::optional<json> jsonObject;
auto pos = v.determinePos(noPos);
auto attr = attrs->find(state.sStructuredAttrs);
if (attr != attrs->end() &&
state.forceBool(*attr->value, pos,
"while evaluating the `__structuredAttrs` "
"attribute passed to builtins.derivationStrict"))
jsonObject = json::object();
/* Check whether null attributes should be ignored. */
bool ignoreNulls = false;
attr = attrs->find(state.sIgnoreNulls);
if (attr != attrs->end())
ignoreNulls = state.forceBool(*attr->value, pos, "while evaluating the `__ignoreNulls` attribute " "passed to builtins.derivationStrict");
/* Build the derivation expression by processing the attributes. */
Derivation drv;
drv.name = drvName;
NixStringContext context;
bool contentAddressed = false;
bool isImpure = false;
std::optional<std::string> outputHash;
std::string outputHashAlgo;
std::optional<ContentAddressMethod> ingestionMethod;
StringSet outputs;
outputs.insert("out");
for (auto & i : attrs->lexicographicOrder(state.symbols)) {
if (i->name == state.sIgnoreNulls) continue;
const std::string & key = state.symbols[i->name];
vomit("processing attribute '%1%'", key);
auto handleHashMode = [&](const std::string_view s) {
if (s == "recursive") ingestionMethod = FileIngestionMethod::Recursive;
else if (s == "flat") ingestionMethod = FileIngestionMethod::Flat;
else if (s == "text") {
experimentalFeatureSettings.require(Xp::DynamicDerivations);
ingestionMethod = TextIngestionMethod {};
} else
state.error<EvalError>(
"invalid value '%s' for 'outputHashMode' attribute", s
).atPos(v).debugThrow();
};
auto handleOutputs = [&](const Strings & ss) {
outputs.clear();
for (auto & j : ss) {
if (outputs.find(j) != outputs.end())
state.error<EvalError>("duplicate derivation output '%1%'", j)
.atPos(v)
.debugThrow();
/* !!! Check whether j is a valid attribute
name. */
/* Derivations cannot be named drv, because
then we'd have an attribute drvPath in
the resulting set. */
if (j == "drv")
state.error<EvalError>("invalid derivation output name 'drv'")
.atPos(v)
.debugThrow();
outputs.insert(j);
}
if (outputs.empty())
state.error<EvalError>("derivation cannot have an empty set of outputs")
.atPos(v)
.debugThrow();
};
try {
// This try-catch block adds context for most errors.
// Use this empty error context to signify that we defer to it.
const std::string_view context_below("");
if (ignoreNulls) {
state.forceValue(*i->value, pos);
if (i->value->type() == nNull) continue;
}
if (i->name == state.sContentAddressed && state.forceBool(*i->value, pos, context_below)) {
contentAddressed = true;
experimentalFeatureSettings.require(Xp::CaDerivations);
}
else if (i->name == state.sImpure && state.forceBool(*i->value, pos, context_below)) {
isImpure = true;
experimentalFeatureSettings.require(Xp::ImpureDerivations);
}
/* The `args' attribute is special: it supplies the
command-line arguments to the builder. */
else if (i->name == state.sArgs) {
state.forceList(*i->value, pos, context_below);
for (auto elem : i->value->listItems()) {
auto s = state.coerceToString(pos, *elem, context,
"while evaluating an element of the argument list",
true).toOwned();
drv.args.push_back(s);
}
}
/* All other attributes are passed to the builder through
the environment. */
else {
if (jsonObject) {
if (i->name == state.sStructuredAttrs) continue;
(*jsonObject)[key] = printValueAsJSON(state, true, *i->value, pos, context);
if (i->name == state.sBuilder)
drv.builder = state.forceString(*i->value, context, pos, context_below);
else if (i->name == state.sSystem)
drv.platform = state.forceStringNoCtx(*i->value, pos, context_below);
else if (i->name == state.sOutputHash)
outputHash = state.forceStringNoCtx(*i->value, pos, context_below);
else if (i->name == state.sOutputHashAlgo)
outputHashAlgo = state.forceStringNoCtx(*i->value, pos, context_below);
else if (i->name == state.sOutputHashMode)
handleHashMode(state.forceStringNoCtx(*i->value, pos, context_below));
else if (i->name == state.sOutputs) {
/* Require outputs to be a list of strings. */
state.forceList(*i->value, pos, context_below);
Strings ss;
for (auto elem : i->value->listItems())
ss.emplace_back(state.forceStringNoCtx(*elem, pos, context_below));
handleOutputs(ss);
}
} else {
auto s = state.coerceToString(pos, *i->value, context, context_below, true).toOwned();
drv.env.emplace(key, s);
if (i->name == state.sBuilder) drv.builder = std::move(s);
else if (i->name == state.sSystem) drv.platform = std::move(s);
else if (i->name == state.sOutputHash) outputHash = std::move(s);
else if (i->name == state.sOutputHashAlgo) outputHashAlgo = std::move(s);
else if (i->name == state.sOutputHashMode) handleHashMode(s);
else if (i->name == state.sOutputs)
handleOutputs(tokenizeString<Strings>(s));
}
}
} catch (Error & e) {
e.addTrace(state.positions[i->pos],
HintFmt("while evaluating attribute '%1%' of derivation '%2%'", key, drvName));
throw;
}
}
if (jsonObject) {
drv.env.emplace("__json", jsonObject->dump());
jsonObject.reset();
}
/* Everything in the context of the strings in the derivation
attributes should be added as dependencies of the resulting
derivation. */
for (auto & c : context) {
std::visit(overloaded {
/* Since this allows the builder to gain access to every
path in the dependency graph of the derivation (including
all outputs), all paths in the graph must be added to
this derivation's list of inputs to ensure that they are
available when the builder runs. */
[&](const NixStringContextElem::DrvDeep & d) {
/* !!! This doesn't work if readOnlyMode is set. */
StorePathSet refs;
state.store->computeFSClosure(d.drvPath, refs);
for (auto & j : refs) {
drv.inputSrcs.insert(j);
if (j.isDerivation()) {
drv.inputDrvs.map[j].value = state.store->readDerivation(j).outputNames();
}
}
},
[&](const NixStringContextElem::Built & b) {
drv.inputDrvs.ensureSlot(*b.drvPath).value.insert(b.output);
},
[&](const NixStringContextElem::Opaque & o) {
drv.inputSrcs.insert(o.path);
},
}, c.raw);
}
/* Do we have all required attributes? */
if (drv.builder == "")
state.error<EvalError>("required attribute 'builder' missing")
.atPos(v)
.debugThrow();
if (drv.platform == "")
state.error<EvalError>("required attribute 'system' missing")
.atPos(v)
.debugThrow();
/* Check whether the derivation name is valid. */
if (isDerivation(drvName) &&
!(ingestionMethod == ContentAddressMethod { TextIngestionMethod { } } &&
outputs.size() == 1 &&
*(outputs.begin()) == "out"))
{
state.error<EvalError>(
"derivation names are allowed to end in '%s' only if they produce a single derivation file",
drvExtension
).atPos(v).debugThrow();
}
if (outputHash) {
/* Handle fixed-output derivations.
Ignore `__contentAddressed` because fixed output derivations are
already content addressed. */
if (outputs.size() != 1 || *(outputs.begin()) != "out")
state.error<EvalError>(
"multiple outputs are not supported in fixed-output derivations"
).atPos(v).debugThrow();
auto h = newHashAllowEmpty(*outputHash, parseHashTypeOpt(outputHashAlgo));
auto method = ingestionMethod.value_or(FileIngestionMethod::Flat);
DerivationOutput::CAFixed dof {
.ca = ContentAddress {
.method = std::move(method),
.hash = std::move(h),
},
};
drv.env["out"] = state.store->printStorePath(dof.path(*state.store, drvName, "out"));
drv.outputs.insert_or_assign("out", std::move(dof));
}
else if (contentAddressed || isImpure) {
if (contentAddressed && isImpure)
state.error<EvalError>("derivation cannot be both content-addressed and impure")
.atPos(v).debugThrow();
auto ht = parseHashTypeOpt(outputHashAlgo).value_or(htSHA256);
auto method = ingestionMethod.value_or(FileIngestionMethod::Recursive);
for (auto & i : outputs) {
drv.env[i] = hashPlaceholder(i);
if (isImpure)
drv.outputs.insert_or_assign(i,
DerivationOutput::Impure {
.method = method,
.hashType = ht,
});
else
drv.outputs.insert_or_assign(i,
DerivationOutput::CAFloating {
.method = method,
.hashType = ht,
});
}
}
else {
/* Compute a hash over the "masked" store derivation, which is
the final one except that in the list of outputs, the
output paths are empty strings, and the corresponding
environment variables have an empty value. This ensures
that changes in the set of output names do get reflected in
the hash. */
for (auto & i : outputs) {
drv.env[i] = "";
drv.outputs.insert_or_assign(i,
DerivationOutput::Deferred { });
}
auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true);
switch (hashModulo.kind) {
case DrvHash::Kind::Regular:
for (auto & i : outputs) {
auto h = get(hashModulo.hashes, i);
if (!h)
state.error<AssertionError>(
"derivation produced no hash for output '%s'",
i
).atPos(v).debugThrow();
auto outPath = state.store->makeOutputPath(i, *h, drvName);
drv.env[i] = state.store->printStorePath(outPath);
drv.outputs.insert_or_assign(
i,
DerivationOutput::InputAddressed {
.path = std::move(outPath),
});
}
break;
;
case DrvHash::Kind::Deferred:
for (auto & i : outputs) {
drv.outputs.insert_or_assign(i, DerivationOutput::Deferred {});
}
}
}
/* Write the resulting term into the Nix store directory. */
auto drvPath = writeDerivation(*state.store, drv, state.repair);
auto drvPathS = state.store->printStorePath(drvPath);
printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
/* Optimisation, but required in read-only mode! because in that
case we don't actually write store derivations, so we can't
read them later. */
{
auto h = hashDerivationModulo(*state.store, drv, false);
drvHashes.lock()->insert_or_assign(drvPath, h);
}
auto result = state.buildBindings(1 + drv.outputs.size());
result.alloc(state.sDrvPath).mkString(drvPathS, {
NixStringContextElem::DrvDeep { .drvPath = drvPath },
});
for (auto & i : drv.outputs)
mkOutputString(state, result, drvPath, i);
v.mkAttrs(result);
}
/* Construct (as a unobservable side effect) a Nix derivation
expression that performs the derivation described by the argument
set. Returns the original set extended with the following
attributes: `outPath' containing the primary output path of the
derivation; `drvPath' containing the path of the Nix expression;
and `type' set to `derivation' to indicate that this is a
derivation. */
static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
state.forceAttrs(*args[0], pos, "while evaluating the argument passed to builtins.derivationStrict");
Bindings * attrs = args[0]->attrs;
/* Figure out the name first (for stack backtraces). */
Bindings::iterator nameAttr = getAttr(state, state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
std::string drvName;
try {
drvName = state.forceStringNoCtx(*nameAttr->value, pos, "while evaluating the `name` attribute passed to builtins.derivationStrict");
} catch (Error & e) {
e.addTrace(state.positions[nameAttr->pos], "while evaluating the derivation attribute 'name'");
throw;
}
try {
derivationStrictInternal(state, drvName, attrs, v);
} catch (Error & e) {
Pos pos = state.positions[nameAttr->pos];
/*
* Here we make two abuses of the error system
*
* 1. We print the location as a string to avoid a code snippet being
* printed. While the location of the name attribute is a good hint, the
* exact code there is irrelevant.
*
* 2. We mark this trace as a frame trace, meaning that we stop printing
* less important traces from now on. In particular, this prevents the
* display of the automatic "while calling builtins.derivationStrict"
* trace, which is of little use for the public we target here.
*
* Please keep in mind that error reporting is done on a best-effort
* basis in nix. There is no accurate location for a derivation, as it
* often results from the composition of several functions
* (derivationStrict, derivation, mkDerivation, mkPythonModule, etc.)
*/
e.addTrace(nullptr, HintFmt(
"while evaluating derivation '%s'\n"
" whose name attribute is located at %s",
drvName, pos));
throw;
}
}
PrimOp primop_derivationStrict(PrimOp {
.name = "derivationStrict",
.arity = 1,
.fun = prim_derivationStrict,
});
}
+7 -26
View File
@@ -1,26 +1,7 @@
#include <boost/optional/optional.hpp> // for get_pointer
#include <map> // for map
#include <optional> // for optional
#include <string> // for allocator, char_tr...
#include "attr-set.hh" // for Attr, Bindings
#include "environment-variables.hh" // for getEnv
#include "error.hh" // for Error, Trace
#include "eval.hh" // for EvalState, PrimOp
#include "experimental-features.hh" // for ExperimentalFeature
#include "fmt.hh" // for HintFmt
#include "make-content-addressed.hh" // for makeContentAddressed
#include "path-info.hh" // for ValidPathInfo
#include "path.hh" // for StorePath
#include "pos-idx.hh" // for PosIdx
#include "pos-table.hh" // for PosTable
#include "realisation.hh" // for RealisedPath
#include "ref.hh" // for ref
#include "store-api.hh" // for Store, copyClosure
#include "strings.hh" // for operator+
#include "symbol-table.hh" // for SymbolStr, operator<<
#include "url.hh" // for ParsedURL, parseURL
#include "value.hh" // for Value, nString
#include "value/context.hh" // for NixStringContext
#include "primops.hh"
#include "store-api.hh"
#include "make-content-addressed.hh"
#include "url.hh"
namespace nix {
@@ -70,7 +51,7 @@ static void runFetchClosureWithRewrite(EvalState & state, const PosIdx pos, Stor
throw Error({
.msg = HintFmt(
"The 'toPath' value '%s' is input-addressed, so it can't possibly be the result of rewriting to a content-addressed path.\n\n"
"Set 'toPath' to an empty string to make Nix report the correct content-addressed path.",
"Set 'toPath' to an empty string to make Lix report the correct content-addressed path.",
state.store->printStorePath(toPath)),
.pos = state.positions[pos]
});
@@ -226,7 +207,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value * * arg
runFetchClosureWithContentAddressedPath(state, pos, *fromStore, *fromPath, v);
}
PrimOp primop_fetchClosure({
static RegisterPrimOp primop_fetchClosure({
.name = "__fetchClosure",
.args = {"args"},
.doc = R"(
@@ -290,7 +271,7 @@ PrimOp primop_fetchClosure({
```
It is possible to fetch an [input-addressed store path](@docroot@/glossary.md#gloss-input-addressed-store-object) and return it as is.
However, this is the least preferred way of invoking `fetchClosure`, because it requires that the input-addressed paths are trusted by the Nix configuration.
However, this is the least preferred way of invoking `fetchClosure`, because it requires that the input-addressed paths are trusted by the Lix configuration.
**`builtins.storePath`**
+8 -21
View File
@@ -1,23 +1,10 @@
#include <optional> // for optional
#include <regex> // for regex_match
#include <string> // for allocator, char_traits, basic_string
#include <string_view> // for operator==, basic_string_view, string_view
#include <utility> // for move
#include "attr-set.hh" // for Attr, BindingsBuilder, Bindings
#include "attrs.hh" // for Attrs
#include "error.hh" // for Error
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval-settings.hh" // for EvalSettings, evalSettings
#include "eval.hh" // for EvalState, PrimOp
#include "fetchers.hh" // for Input, Tree
#include "hash.hh" // for Hash, HashType
#include "pos-idx.hh" // for PosIdx
#include "ref.hh" // for ref
#include "symbol-table.hh" // for Symbol, SymbolStr, SymbolTable
#include "types.hh" // for BackedStringView
#include "url-parts.hh" // for revRegex
#include "value.hh" // for Value, nAttrs
#include "value/context.hh" // for NixStringContext
#include "primops.hh"
#include "eval-inline.hh"
#include "eval-settings.hh"
#include "store-api.hh"
#include "fetchers.hh"
#include "url.hh"
#include "url-parts.hh"
namespace nix {
@@ -96,7 +83,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value * * a
state.allowPath(tree.storePath);
}
PrimOp primop_fetchMercurial({
static RegisterPrimOp r_fetchMercurial({
.name = "fetchMercurial",
.arity = 1,
.fun = prim_fetchMercurial
+16 -37
View File
@@ -1,36 +1,15 @@
#include <assert.h> // for assert
#include <stdint.h> // for uint64_t
#include <ctime> // for gmtime
#include <iomanip> // for put_time, operator<<
#include <map> // for operator==, map
#include <optional> // for optional, operator==
#include <regex> // for regex_match, regex...
#include <string> // for char_traits, basic...
#include <string_view> // for operator==, basic_...
#include <utility> // for move, pair
#include "attr-set.hh" // for Attr, BindingsBuilder
#include "attrs.hh" // for maybeGetStrAttr
#include "config.hh" // for Setting, Experimen...
#include "content-address.hh" // for FileIngestionMethod
#include "eval-error.hh" // for EvalError, EvalErr...
#include "eval-settings.hh" // for EvalSettings, eval...
#include "eval.hh" // for EvalState, PrimOp
#include "experimental-features.hh" // for ExperimentalFeature
#include "fetchers.hh" // for Input, Tree, downl...
#include "file-system.hh" // for baseNameOf
#include "fmt.hh" // for fmt
#include "hash.hh" // for Hash, HashType, Base
#include "path-info.hh" // for ValidPathInfo
#include "path.hh" // for StorePath
#include "pos-idx.hh" // for PosIdx
#include "ref.hh" // for ref
#include "registry.hh" // for lookupInRegistries
#include "store-api.hh" // for Store
#include "symbol-table.hh" // for Symbol, SymbolTable
#include "types.hh" // for BackedStringView
#include "url.hh" // for ParsedURL
#include "value.hh" // for Value, nAttrs, nBool
#include "value/context.hh" // for NixStringContext
#include "primops.hh"
#include "eval-inline.hh"
#include "eval-settings.hh"
#include "store-api.hh"
#include "fetchers.hh"
#include "filetransfer.hh"
#include "registry.hh"
#include "url.hh"
#include <ctime>
#include <iomanip>
#include <regex>
namespace nix {
@@ -219,7 +198,7 @@ static void prim_fetchTree(EvalState & state, const PosIdx pos, Value * * args,
}
// FIXME: document
PrimOp primop_fetchTree({
static RegisterPrimOp primop_fetchTree({
.name = "fetchTree",
.arity = 1,
.fun = prim_fetchTree
@@ -311,7 +290,7 @@ static void prim_fetchurl(EvalState & state, const PosIdx pos, Value * * args, V
fetch(state, pos, args, v, "fetchurl", false, "");
}
PrimOp primop_fetchurl({
static RegisterPrimOp primop_fetchurl({
.name = "__fetchurl",
.args = {"url"},
.doc = R"(
@@ -327,7 +306,7 @@ static void prim_fetchTarball(EvalState & state, const PosIdx pos, Value * * arg
fetch(state, pos, args, v, "fetchTarball", true, "source");
}
PrimOp primop_fetchTarball({
static RegisterPrimOp primop_fetchTarball({
.name = "fetchTarball",
.args = {"args"},
.doc = R"(
@@ -377,7 +356,7 @@ static void prim_fetchGit(EvalState & state, const PosIdx pos, Value * * args, V
fetchTree(state, pos, args, v, "git", FetchTreeParams { .emptyRevFallback = true, .allowNameArgument = true });
}
PrimOp primop_fetchGit({
static RegisterPrimOp primop_fetchGit({
.name = "fetchGit",
.args = {"args"},
.doc = R"(
+6 -25
View File
@@ -1,27 +1,8 @@
#include <stddef.h> // for size_t
#include <stdint.h> // for int64_t
#include <exception> // for exception
#include <functional> // for function
#include <sstream> // for char_traits, basic_ostringstream
#include <stdexcept> // for runtime_error
#include <string> // for basic_string, operator<<, operat...
#include <toml/comments.hpp> // for operator<<
#include <toml/datetime.hpp> // for operator<<
#include <toml/get.hpp> // for get
#include <toml/parser.hpp> // for parse
#include <toml/serializer.hpp> // for operator<<
#include <toml/types.hpp> // for value_t
#include <toml/value.hpp> // for value, table
#include <unordered_map> // for unordered_map, _Node_iterator
#include <utility> // for pair
#include <vector> // for vector
#include "attr-set.hh" // for BindingsBuilder
#include "config.hh" // for ExperimentalFeatureSettings, exp...
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp
#include "experimental-features.hh" // for ExperimentalFeature, Xp
#include "pos-idx.hh" // for PosIdx
#include "value.hh" // for Value, NixFloat
#include "primops.hh"
#include "eval-inline.hh"
#include <sstream>
#include <toml.hpp>
namespace nix {
@@ -105,7 +86,7 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value * * args, V
}
}
PrimOp primop_fromTOML({
static RegisterPrimOp primop_fromTOML({
.name = "fromTOML",
.args = {"e"},
.doc = R"(
-75
View File
@@ -1,75 +0,0 @@
#include "hash.hh"
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp
#include "pos-idx.hh" // for PosIdx
#include "primops.hh" // for realisePath
#include "source-path.hh" // for SourcePath
#include "value.hh" // for Value
#include "value/context.hh" // for NixStringContext
namespace nix {
/**
* builtins.hashFile
*/
static void prim_hashFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto type = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.hashFile"
);
std::optional<HashType> ht = parseHashType(type);
if (!ht) {
state.error<EvalError>("unknown hash type '%1%'", type).atPos(pos).debugThrow();
}
auto path = realisePath(state, pos, *args[1]);
v.mkString(hashString(*ht, path.readFile()).to_string(Base16, false));
}
PrimOp primop_hashFile({
.name = "__hashFile",
.args = {"type", "p"},
.doc = R"(
Return a base-16 representation of the cryptographic hash of the
file at path *p*. The hash algorithm specified by *type* must be one
of `"md5"`, `"sha1"`, `"sha256"` or `"sha512"`.
)",
.fun = prim_hashFile,
});
/**
* builtins.hashString
*/
static void prim_hashString(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto type = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.hashString"
);
std::optional<HashType> ht = parseHashType(type);
if (!ht) {
state.error<EvalError>("unknown hash algorithm '%1%'", type).atPos(pos).debugThrow();
}
NixStringContext context; // discarded
auto s = state.forceString(
*args[1], context, pos, "while evaluating the second argument passed to builtins.hashString"
);
v.mkString(hashString(*ht, s).to_string(Base16, false));
}
PrimOp primop_hashString({
.name = "__hashString",
.args = {"type", "s"},
.doc = R"(
Return a base-16 representation of the cryptographic hash of string
*s*. The hash algorithm specified by *type* must be one of `"md5"`,
`"sha1"`, `"sha256"` or `"sha512"`.
)",
.fun = prim_hashString,
});
}
-290
View File
@@ -1,290 +0,0 @@
#include <dlfcn.h> // for dlerror, dlopen, dlsym, RTLD_LAZY, RTLD_...
#include <memory> // for allocator, __shared_ptr_access, shared_ptr
#include <optional> // for optional, nullopt, nullopt_t
#include <string> // for basic_string, char_traits, string, opera...
#include <utility> // for pair
#include "attr-set.hh" // for BindingsBuilder, Bindings, Attr
#include "canon-path.hh" // for CanonPath
#include "derivations.hh" // for Derivation, DerivationOutput, isDerivation
#include "derived-path.hh" // for makeConstantStorePathRef, SingleDerivedPath
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp, Env, resolveExprPath
#include "logging.hh" // for Logger, debug
#include "nixexpr.hh" // for StaticEnv, Expr
#include "path.hh" // for StorePath
#include "pos-idx.hh" // for PosIdx
#include "primops.hh" // for realisePath, prim_importNative, mkOutput...
#include "ref.hh" // for ref
#include "source-path.hh" // for SourcePath
#include "store-api.hh" // for Store
#include "symbol-table.hh" // for Symbol
#include "types.hh" // for enumerate
#include "value.hh" // for Value, allocRootValue
#include "value/context.hh" // for NixStringContextElem
namespace nix {
/* Want reasonable symbol names, so extern C */
/* !!! Should we pass the Pos or the file name too? */
extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v);
/**
* Add and attribute to the given attribute map from the output name to
* the output path, or a placeholder.
*
* Where possible the path is used, but for floating CA derivations we
* may not know it. For sake of determinism we always assume we don't
* and instead put in a place holder. In either case, however, the
* string context will contain the drv path and output name, so
* downstream derivations will have the proper dependency, and in
* addition, before building, the placeholder will be rewritten to be
* the actual path.
*
* The 'drv' and 'drvPath' outputs must correspond.
*/
void mkOutputString(
EvalState & state,
BindingsBuilder & attrs,
const StorePath & drvPath,
const std::pair<std::string, DerivationOutput> & o
)
{
state.mkOutputString(
attrs.alloc(o.first),
SingleDerivedPath::Built{
.drvPath = makeConstantStorePathRef(drvPath),
.output = o.first,
},
o.second.path(*state.store, Derivation::nameFromPath(drvPath), o.first)
);
}
/* Load and evaluate an expression from path specified by the
argument. */
static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * vScope, Value & v)
{
auto path = realisePath(state, pos, vPath);
auto path2 = path.path.abs();
// FIXME
auto isValidDerivationInStore = [&]() -> std::optional<StorePath> {
if (!state.store->isStorePath(path2)) {
return std::nullopt;
}
auto storePath = state.store->parseStorePath(path2);
if (!(state.store->isValidPath(storePath) && isDerivation(path2))) {
return std::nullopt;
}
return storePath;
};
if (auto storePath = isValidDerivationInStore()) {
Derivation drv = state.store->readDerivation(*storePath);
auto attrs = state.buildBindings(3 + drv.outputs.size());
attrs.alloc(state.sDrvPath)
.mkString(
path2,
{
NixStringContextElem::DrvDeep{.drvPath = *storePath},
}
);
attrs.alloc(state.sName).mkString(drv.env["name"]);
auto & outputsVal = attrs.alloc(state.sOutputs);
state.mkList(outputsVal, drv.outputs.size());
for (const auto & [i, o] : enumerate(drv.outputs)) {
mkOutputString(state, attrs, *storePath, o);
(outputsVal.listElems()[i] = state.allocValue())->mkString(o.first);
}
auto w = state.allocValue();
w->mkAttrs(attrs);
if (!state.vImportedDrvToDerivation) {
state.vImportedDrvToDerivation = allocRootValue(state.allocValue());
state.eval(
state.parseExprFromString(
#include "imported-drv-to-derivation.nix.gen.hh"
, CanonPath::root
),
**state.vImportedDrvToDerivation
);
}
state.forceFunction(
**state.vImportedDrvToDerivation,
pos,
"while evaluating imported-drv-to-derivation.nix.gen.hh"
);
v.mkApp(*state.vImportedDrvToDerivation, w);
state.forceAttrs(v, pos, "while calling imported-drv-to-derivation.nix.gen.hh");
}
else if (path2 == corepkgsPrefix + "fetchurl.nix")
{
state.eval(
state.parseExprFromString(
#include "fetchurl.nix.gen.hh"
, CanonPath::root
),
v
);
}
else
{
if (!vScope) {
state.evalFile(path, v);
} else {
state.forceAttrs(
*vScope, pos, "while evaluating the first argument passed to builtins.scopedImport"
);
Env * env = &state.allocEnv(vScope->attrs->size());
env->up = &state.baseEnv;
auto staticEnv = std::make_shared<StaticEnv>(
nullptr, state.staticBaseEnv.get(), vScope->attrs->size()
);
unsigned int displ = 0;
for (auto & attr : *vScope->attrs) {
staticEnv->vars.emplace_back(attr.name, displ);
env->values[displ++] = attr.value;
}
// No need to call staticEnv.sort(), because
// args[0]->attrs is already sorted.
debug("evaluating file '%1%'", path);
Expr * e = state.parseExprFromFile(resolveExprPath(path), staticEnv);
e->eval(state, *env, v);
}
}
}
static void prim_import(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
import(state, pos, *args[0], nullptr, v);
}
PrimOp primop_import({
.name = "import",
.args = {"path"},
// TODO turn "normal path values" into link below
.doc = R"(
Load, parse and return the Nix expression in the file *path*.
The value *path* can be a path, a string, or an attribute set with an
`__toString` attribute or a `outPath` attribute (as derivations or flake
inputs typically have).
If *path* is a directory, the file `default.nix` in that directory
is loaded.
Evaluation aborts if the file doesnt exist or contains
an incorrect Nix expression. `import` implements Nixs module
system: you can put any Nix expression (such as a set or a
function) in a separate file, and use it from Nix expressions in
other files.
> **Note**
>
> Unlike some languages, `import` is a regular function in Nix.
> Paths using the angle bracket syntax (e.g., `import` *\<foo\>*)
> are normal [path values](@docroot@/language/values.md#type-path).
A Nix expression loaded by `import` must not contain any *free
variables* (identifiers that are not defined in the Nix expression
itself and are not built-in). Therefore, it cannot refer to
variables that are in scope at the call site. For instance, if you
have a calling expression
```nix
rec {
x = 123;
y = import ./foo.nix;
}
```
then the following `foo.nix` will give an error:
```nix
x + 456
```
since `x` is not in scope in `foo.nix`. If you want `x` to be
available in `foo.nix`, you should pass it as a function argument:
```nix
rec {
x = 123;
y = import ./foo.nix x;
}
```
and
```nix
x: x + 456
```
(The function argument doesnt have to be called `x` in `foo.nix`;
any name would work.)
)",
.fun = prim_import,
});
/* Load a ValueInitializer from a DSO and return whatever it initializes */
void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
std::string sym(state.forceStringNoCtx(
*args[1], pos, "while evaluating the second argument passed to builtins.importNative"
));
void * handle = dlopen(path.path.c_str(), RTLD_LAZY | RTLD_LOCAL);
if (!handle) {
state.error<EvalError>("could not open '%1%': %2%", path, dlerror()).debugThrow();
}
dlerror();
ValueInitializer func = (ValueInitializer) dlsym(handle, sym.c_str());
if (!func) {
char * message = dlerror();
if (message) {
state
.error<EvalError>("could not load symbol '%1%' from '%2%': %3%", sym, path, message)
.debugThrow();
} else {
state
.error<EvalError>(
"symbol '%1%' from '%2%' resolved to NULL when a function pointer was expected",
sym,
path
)
.debugThrow();
}
}
(func)(state, v);
/* We don't dlclose because v may be a primop referencing a function in the shared object file
*/
}
static void prim_scopedImport(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
import(state, pos, *args[1], args[0], v);
}
PrimOp primop_scopedImport(PrimOp{
.name = "scopedImport",
.arity = 2,
.fun = prim_scopedImport,
});
}
-70
View File
@@ -1,70 +0,0 @@
#include <sstream> // for basic_ostringstream, char_traits, ostrin...
#include "eval.hh" // for PrimOp, EvalState
#include "json-to-value.hh" // for JSONParseError, parseJSON
#include "pos-idx.hh" // for PosIdx
#include "pos-table.hh" // for PosTable
#include "value-to-json.hh" // for printValueAsJSON
#include "value.hh" // for Value
#include "value/context.hh" // for NixStringContext
namespace nix {
/**
* builtins.fromJSON
*/
static void prim_fromJSON(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto s = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.fromJSON"
);
try {
parseJSON(state, s, v);
} catch (JSONParseError & e) {
e.addTrace(state.positions[pos], "while decoding a JSON string");
throw;
}
}
PrimOp primop_fromJSON({
.name = "__fromJSON",
.args = {"e"},
.doc = R"(
Convert a JSON string to a Nix value. For example,
```nix
builtins.fromJSON ''{"x": [1, 2, 3], "y": null}''
```
returns the value `{ x = [ 1 2 3 ]; y = null; }`.
)",
.fun = prim_fromJSON,
});
/**
* builtins.toJSON
*/
static void prim_toJSON(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
std::ostringstream out;
NixStringContext context;
printValueAsJSON(state, true, *args[0], pos, out, context);
v.mkString(out.str(), context);
}
PrimOp primop_toJSON({
.name = "__toJSON",
.args = {"e"},
.doc = R"(
Return a string containing a JSON representation of *e*. Strings,
integers, floats, booleans, nulls and lists are mapped to their JSON
equivalents. Sets (except derivations) are represented as objects.
Derivations are translated to a JSON string containing the
derivations output path. Paths are copied to the store and
represented as a JSON string of the resulting store path.
)",
.fun = prim_toJSON,
});
}
-694
View File
@@ -1,694 +0,0 @@
#include <string.h> // for memcpy, size_t
#include <algorithm> // for stable_sort
#include <boost/container/vector.hpp> // for vector
#include <functional> // for function
#include <set> // for set
#include <string> // for basic_string, allocator, opera...
#include <utility> // for pair
#include "attr-set.hh" // for BindingsBuilder, Bindings
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp
#include "gc-small-vector.hh" // for SmallTemporaryValueVector, Sma...
#include "pos-idx.hh" // for PosIdx, noPos
#include "primops.hh" // for getAttr, CompareValues, prim_l...
#include "symbol-table.hh" // for Symbol, SymbolTable
#include "types.hh" // for enumerate
#include "value.hh" // for Value, ValueVector
namespace nix {
static void anyOrAll(bool any, EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
const std::string fun = any ? "builtins.any" : "builtins.all";
state.forceFunction(
*args[0], pos, std::string("while evaluating the first argument passed to " + fun)
);
state.forceList(
*args[1], pos, std::string("while evaluating the second argument passed to " + fun)
);
Value vTmp;
for (auto elem : args[1]->listItems()) {
state.callFunction(*args[0], *elem, vTmp, pos);
bool res = state.forceBool(
vTmp, pos, "while evaluating the return value of the function passed to " + fun
);
if (res == any) {
v.mkBool(any);
return;
}
}
v.mkBool(!any);
}
static void elemAt(EvalState & state, const PosIdx pos, Value & list, int n, Value & v)
{
state.forceList(list, pos, "while evaluating the first argument passed to builtins.elemAt");
if (n < 0 || (unsigned int) n >= list.listSize()) {
state.error<EvalError>("list index %1% is out of bounds", n).atPos(pos).debugThrow();
}
state.forceValue(*list.listElems()[n], pos);
v = *list.listElems()[n];
}
/**
* builtins.all
*/
static void prim_all(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
anyOrAll(false, state, pos, args, v);
}
PrimOp primop_all({
.name = "__all",
.args = {"pred", "list"},
.doc = R"(
Return `true` if the function *pred* returns `true` for all elements
of *list*, and `false` otherwise.
)",
.fun = prim_all,
});
/**
* builtins.any
*/
static void prim_any(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
anyOrAll(true, state, pos, args, v);
}
PrimOp primop_any({
.name = "__any",
.args = {"pred", "list"},
.doc = R"(
Return `true` if the function *pred* returns `true` for at least one
element of *list*, and `false` otherwise.
)",
.fun = prim_any,
});
/**
* builtins.concatLists
*/
static void prim_concatLists(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(
*args[0], pos, "while evaluating the first argument passed to builtins.concatLists"
);
state.concatLists(
v,
args[0]->listSize(),
args[0]->listElems(),
pos,
"while evaluating a value of the list passed to builtins.concatLists"
);
}
PrimOp primop_concatLists({
.name = "__concatLists",
.args = {"lists"},
.doc = R"(
Concatenate a list of lists into a single list.
)",
.fun = prim_concatLists,
});
/**
* builtins.concatMap
*/
static void prim_concatMap(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.concatMap"
);
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.concatMap"
);
auto nrLists = args[1]->listSize();
// List of returned lists before concatenation. References to these Values must NOT be
// persisted.
SmallTemporaryValueVector<conservativeStackReservation> lists(nrLists);
size_t len = 0;
for (unsigned int n = 0; n < nrLists; ++n) {
Value * vElem = args[1]->listElems()[n];
state.callFunction(*args[0], *vElem, lists[n], pos);
state.forceList(
lists[n],
lists[n].determinePos(args[0]->determinePos(pos)),
"while evaluating the return value of the function passed to builtins.concatMap"
);
len += lists[n].listSize();
}
state.mkList(v, len);
auto out = v.listElems();
for (unsigned int n = 0, pos = 0; n < nrLists; ++n) {
auto l = lists[n].listSize();
if (l) {
memcpy(out + pos, lists[n].listElems(), l * sizeof(Value *));
}
pos += l;
}
}
PrimOp primop_concatMap({
.name = "__concatMap",
.args = {"f", "list"},
.doc = R"(
This function is equivalent to `builtins.concatLists (map f list)`
but is more efficient.
)",
.fun = prim_concatMap,
});
/**
* builtins.elem
*/
static void prim_elem(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
bool res = false;
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.elem");
for (auto elem : args[1]->listItems()) {
if (state.eqValues(
*args[0],
*elem,
pos,
"while searching for the presence of the given element in the list"
))
{
res = true;
break;
}
}
v.mkBool(res);
}
PrimOp primop_elem({
.name = "__elem",
.args = {"x", "xs"},
.doc = R"(
Return `true` if a value equal to *x* occurs in the list *xs*, and
`false` otherwise.
)",
.fun = prim_elem,
});
/**
* builtins.elemAt
*/
static void prim_elemAt(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
elemAt(
state,
pos,
*args[0],
state.forceInt(
*args[1], pos, "while evaluating the second argument passed to builtins.elemAt"
),
v
);
}
PrimOp primop_elemAt({
.name = "__elemAt",
.args = {"xs", "n"},
.doc = R"(
Return element *n* from the list *xs*. Elements are counted starting
from 0. A fatal error occurs if the index is out of bounds.
)",
.fun = prim_elemAt,
});
/**
* builtins.filter
*/
static void prim_filter(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.filter"
);
if (args[1]->listSize() == 0) {
v = *args[1];
return;
}
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.filter"
);
SmallValueVector<nonRecursiveStackReservation> vs(args[1]->listSize());
size_t k = 0;
bool same = true;
for (unsigned int n = 0; n < args[1]->listSize(); ++n) {
Value res;
state.callFunction(*args[0], *args[1]->listElems()[n], res, noPos);
if (state.forceBool(
res,
pos,
"while evaluating the return value of the filtering function passed to "
"builtins.filter"
))
{
vs[k++] = args[1]->listElems()[n];
} else {
same = false;
}
}
if (same) {
v = *args[1];
} else {
state.mkList(v, k);
for (unsigned int n = 0; n < k; ++n) {
v.listElems()[n] = vs[n];
}
}
}
/**
* builtins.foldl'
*/
static void prim_foldlStrict(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.foldlStrict"
);
state.forceList(
*args[2], pos, "while evaluating the third argument passed to builtins.foldlStrict"
);
if (args[2]->listSize()) {
Value * vCur = args[1];
for (auto [n, elem] : enumerate(args[2]->listItems())) {
Value * vs[]{vCur, elem};
vCur = n == args[2]->listSize() - 1 ? &v : state.allocValue();
state.callFunction(*args[0], 2, vs, *vCur, pos);
}
state.forceValue(v, pos);
} else {
state.forceValue(*args[1], pos);
v = *args[1];
}
}
PrimOp primop_foldlStrict({
.name = "__foldl'",
.args = {"op", "nul", "list"},
.doc = R"(
Reduce a list by applying a binary operator, from left to right,
e.g. `foldl' op nul [x0 x1 x2 ...] = op (op (op nul x0) x1) x2)
...`. For example, `foldl' (x: y: x + y) 0 [1 2 3]` evaluates to 6.
The return value of each application of `op` is evaluated immediately,
even for intermediate values.
)",
.fun = prim_foldlStrict,
});
/**
* builtins.genList
*/
static void prim_genList(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto len = state.forceInt(
*args[1], pos, "while evaluating the second argument passed to builtins.genList"
);
if (len < 0) {
state.error<EvalError>("cannot create list of size %1%", len).atPos(pos).debugThrow();
}
// More strict than striclty (!) necessary, but acceptable
// as evaluating map without accessing any values makes little sense.
state.forceFunction(
*args[0], noPos, "while evaluating the first argument passed to builtins.genList"
);
state.mkList(v, len);
for (unsigned int n = 0; n < (unsigned int) len; ++n) {
auto arg = state.allocValue();
arg->mkInt(n);
(v.listElems()[n] = state.allocValue())->mkApp(args[0], arg);
}
}
PrimOp primop_genList({
.name = "__genList",
.args = {"generator", "length"},
.doc = R"(
Generate list of size *length*, with each element *i* equal to the
value returned by *generator* `i`. For example,
```nix
builtins.genList (x: x * x) 5
```
returns the list `[ 0 1 4 9 16 ]`.
)",
.fun = prim_genList,
});
/**
* builtins.head
*/
static void prim_head(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
elemAt(state, pos, *args[0], 0, v);
}
PrimOp primop_head({
.name = "__head",
.args = {"list"},
.doc = R"(
Return the first element of a list; abort evaluation if the argument
isnt a list or is an empty list. You can test whether a list is
empty by comparing it with `[]`.
)",
.fun = prim_head,
});
/**
* builtins.length
*/
static void prim_length(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.length");
v.mkInt(args[0]->listSize());
}
PrimOp primop_length({
.name = "__length",
.args = {"e"},
.doc = R"(
Return the length of the list *e*.
)",
.fun = prim_length,
});
PrimOp primop_filter({
.name = "__filter",
.args = {"f", "list"},
.doc = R"(
Return a list consisting of the elements of *list* for which the
function *f* returns `true`.
)",
.fun = prim_filter,
});
/**
* builtins.listToAttrs
*/
static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the argument passed to builtins.listToAttrs");
auto attrs = state.buildBindings(args[0]->listSize());
std::set<Symbol> seen;
for (auto v2 : args[0]->listItems()) {
state.forceAttrs(
*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs"
);
Bindings::iterator j =
getAttr(state, state.sName, v2->attrs, "in a {name=...; value=...;} pair");
auto name = state.forceStringNoCtx(
*j->value,
j->pos,
"while evaluating the `name` attribute of an element of the list passed to "
"builtins.listToAttrs"
);
auto sym = state.symbols.create(name);
if (seen.insert(sym).second) {
Bindings::iterator j2 =
getAttr(state, state.sValue, v2->attrs, "in a {name=...; value=...;} pair");
attrs.insert(sym, j2->value, j2->pos);
}
}
v.mkAttrs(attrs);
}
PrimOp primop_listToAttrs({
.name = "__listToAttrs",
.args = {"e"},
.doc = R"(
Construct a set from a list specifying the names and values of each
attribute. Each element of the list should be a set consisting of a
string-valued attribute `name` specifying the name of the attribute,
and an attribute `value` specifying its value.
In case of duplicate occurrences of the same name, the first
takes precedence.
Example:
```nix
builtins.listToAttrs
[ { name = "foo"; value = 123; }
{ name = "bar"; value = 456; }
{ name = "bar"; value = 420; }
]
```
evaluates to
```nix
{ foo = 123; bar = 456; }
```
)",
.fun = prim_listToAttrs,
});
/**
* builtins.map
*/
static void prim_map(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.map");
if (args[1]->listSize() == 0) {
v = *args[1];
return;
}
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.map"
);
state.mkList(v, args[1]->listSize());
for (unsigned int n = 0; n < v.listSize(); ++n) {
(v.listElems()[n] = state.allocValue())->mkApp(args[0], args[1]->listElems()[n]);
}
}
PrimOp primop_map({
.name = "map",
.args = {"f", "list"},
.doc = R"(
Apply the function *f* to each element in the list *list*. For
example,
```nix
map (x: "foo" + x) [ "bar" "bla" "abc" ]
```
evaluates to `[ "foobar" "foobla" "fooabc" ]`.
)",
.fun = prim_map,
});
/**
* builtins.partition
*/
static void prim_partition(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.partition"
);
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.partition"
);
auto len = args[1]->listSize();
ValueVector right, wrong;
for (unsigned int n = 0; n < len; ++n) {
auto vElem = args[1]->listElems()[n];
state.forceValue(*vElem, pos);
Value res;
state.callFunction(*args[0], *vElem, res, pos);
if (state.forceBool(
res,
pos,
"while evaluating the return value of the partition function passed to "
"builtins.partition"
))
{
right.push_back(vElem);
} else {
wrong.push_back(vElem);
}
}
auto attrs = state.buildBindings(2);
auto & vRight = attrs.alloc(state.sRight);
auto rsize = right.size();
state.mkList(vRight, rsize);
if (rsize) {
memcpy(vRight.listElems(), right.data(), sizeof(Value *) * rsize);
}
auto & vWrong = attrs.alloc(state.sWrong);
auto wsize = wrong.size();
state.mkList(vWrong, wsize);
if (wsize) {
memcpy(vWrong.listElems(), wrong.data(), sizeof(Value *) * wsize);
}
v.mkAttrs(attrs);
}
PrimOp primop_partition({
.name = "__partition",
.args = {"pred", "list"},
.doc = R"(
Given a predicate function *pred*, this function returns an
attrset containing a list named `right`, containing the elements
in *list* for which *pred* returned `true`, and a list named
`wrong`, containing the elements for which it returned
`false`. For example,
```nix
builtins.partition (x: x > 10) [1 23 9 3 42]
```
evaluates to
```nix
{ right = [ 23 42 ]; wrong = [ 1 9 3 ]; }
```
)",
.fun = prim_partition,
});
/**
* builtins.sort
*/
static void prim_sort(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.sort");
auto len = args[1]->listSize();
if (len == 0) {
v = *args[1];
return;
}
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.sort"
);
state.mkList(v, len);
for (unsigned int n = 0; n < len; ++n) {
state.forceValue(*args[1]->listElems()[n], pos);
v.listElems()[n] = args[1]->listElems()[n];
}
auto comparator = [&](Value * a, Value * b) {
/* Optimization: if the comparator is lessThan, bypass
callFunction. */
/* TODO: (layus) this is absurd. An optimisation like this
should be outside the lambda creation */
if (args[0]->isPrimOp()) {
auto ptr = args[0]->primOp->fun.target<decltype(&prim_lessThan)>();
if (ptr && *ptr == prim_lessThan) {
return CompareValues(state, noPos, "while evaluating the ordering function passed to builtins.sort")(a, b);
}
}
Value * vs[] = {a, b};
Value vBool;
state.callFunction(*args[0], 2, vs, vBool, noPos);
return state.forceBool(
vBool,
pos,
"while evaluating the return value of the sorting function passed to builtins.sort"
);
};
/* FIXME: std::sort can segfault if the comparator is not a strict
weak ordering. What to do? std::stable_sort() seems more
resilient, but no guarantees... */
std::stable_sort(v.listElems(), v.listElems() + len, comparator);
}
PrimOp primop_sort({
.name = "__sort",
.args = {"comparator", "list"},
.doc = R"(
Return *list* in sorted order. It repeatedly calls the function
*comparator* with two elements. The comparator should return `true`
if the first element is less than the second, and `false` otherwise.
For example,
```nix
builtins.sort builtins.lessThan [ 483 249 526 147 42 77 ]
```
produces the list `[ 42 77 147 249 483 526 ]`.
This is a stable sort: it preserves the relative order of elements
deemed equal by the comparator.
)",
.fun = prim_sort,
});
/**
* builtins.tail
*/
static void prim_tail(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.tail");
if (args[0]->listSize() == 0) {
state.error<EvalError>("'tail' called on an empty list").atPos(pos).debugThrow();
}
state.mkList(v, args[0]->listSize() - 1);
for (unsigned int n = 0; n < v.listSize(); ++n) {
v.listElems()[n] = args[0]->listElems()[n + 1];
}
}
PrimOp primop_tail({
.name = "__tail",
.args = {"list"},
.doc = R"(
Return the second to last elements of a list; abort evaluation if
the argument isnt a list or is an empty list.
> **Warning**
>
> This function should generally be avoided since it's inefficient:
> unlike Haskell's `tail`, it takes O(n) time, so recursing over a
> list by repeatedly calling `tail` takes O(n^2) time.
)",
.fun = prim_tail,
});
}
-967
View File
@@ -1,967 +0,0 @@
#include <sys/stat.h> // for stat, S_ISDIR, S_I...
#include <list> // for list
#include <optional> // for optional, operator!=
#include <set> // for _Rb_tree_const_ite...
#include <string> // for basic_string, char...
#include <string_view> // for string_view
#include <utility> // for move, pair
#include <variant> // for get_if
#include "attr-set.hh" // for Attr, Bindings
#include "canon-path.hh" // for CanonPath
#include "content-address.hh" // for FileIngestionMethod
#include "derivations.hh" // for hashPlaceholder
#include "derived-path.hh" // for SingleDerivedPath
#include "error.hh" // for Error, SysError
#include "eval-error.hh" // for EvalError, EvalErr...
#include "eval-settings.hh" // for EvalSettings, eval...
#include "eval.hh" // for EvalState, PrimOp
#include "experimental-features.hh" // for ExperimentalFeature
#include "fetch-to-store.hh" // for fetchToStore
#include "file-system.hh" // for canonPath, dirOf
#include "globals.hh" // for Settings, settings
#include "hash.hh" // for Hash, newHashAllow...
#include "input-accessor.hh" // for InputAccessor
#include "outputs-spec.hh" // for OutputNameView
#include "path-info.hh" // for ValidPathInfo
#include "path-references.hh" // for PathRefScanSink
#include "path.hh" // for StorePath, StorePa...
#include "pos-idx.hh" // for PosIdx, noPos
#include "pos-table.hh" // for PosTable
#include "primops.hh" // for RealisePathFlags
#include "ref.hh" // for ref, make_ref
#include "search-path.hh" // for SearchPath
#include "serialise.hh" // for operator<<, Sink
#include "source-path.hh" // for SourcePath, Symlin...
#include "store-api.hh" // for Store
#include "strings.hh" // for rewriteStrings
#include "symbol-table.hh" // for Symbol, SymbolStr
#include "types.hh" // for BackedStringView
#include "value.hh" // for Value, nPath, nString
#include "value/context.hh" // for NixStringContext
namespace nix {
// TODO: Use a const map instead ?
static std::string_view fileTypeToString(InputAccessor::Type type)
{
return type == InputAccessor::Type::tRegular ? "regular"
: type == InputAccessor::Type::tDirectory ? "directory"
: type == InputAccessor::Type::tSymlink ? "symlink"
: "unknown";
}
static void addPath(
EvalState & state,
const PosIdx pos,
std::string_view name,
Path path,
Value * filterFun,
FileIngestionMethod method,
const std::optional<Hash> expectedHash,
Value & v,
const NixStringContext & context
)
{
try {
// FIXME: handle CA derivation outputs (where path needs to
// be rewritten to the actual output).
auto rewrites = state.realiseContext(context);
path = state.toRealPath(rewriteStrings(path, rewrites), context);
StorePathSet refs;
if (state.store->isInStore(path)) {
try {
auto [storePath, subPath] = state.store->toStorePath(path);
// FIXME: we should scanForReferences on the path before adding it
refs = state.store->queryPathInfo(storePath)->references;
path = state.store->toRealPath(storePath) + subPath;
} catch (Error &) { // FIXME: should be InvalidPathError
}
}
path = evalSettings.pureEval && expectedHash
? path
: state.checkSourcePath(CanonPath(path)).path.abs();
PathFilter filter = filterFun ? ([&](const Path & path) {
auto st = lstat(path);
/* Call the filter function. The first argument is the path,
the second is a string indicating the type of the file. */
Value arg1;
arg1.mkString(path);
Value arg2;
arg2.mkString(
S_ISREG(st.st_mode) ? "regular"
: S_ISDIR(st.st_mode) ? "directory"
: S_ISLNK(st.st_mode) ? "symlink"
: "unknown" /* not supported, will fail! */
);
Value * args[]{&arg1, &arg2};
Value res;
state.callFunction(*filterFun, 2, args, res, pos);
return state.forceBool(
res, pos, "while evaluating the return value of the path filter function"
);
})
: defaultPathFilter;
std::optional<StorePath> expectedStorePath;
if (expectedHash) {
expectedStorePath = state.store->makeFixedOutputPath(
name,
FixedOutputInfo{
.method = method,
.hash = *expectedHash,
.references = {},
}
);
}
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
auto dstPath = fetchToStore(
*state.store, state.rootPath(CanonPath(path)), name, method, &filter, state.repair
);
if (expectedHash && expectedStorePath != dstPath) {
state
.error<EvalError>(
"store path mismatch in (possibly filtered) path added from '%s'", path
)
.atPos(pos)
.debugThrow();
}
state.allowAndSetStorePathString(dstPath, v);
} else {
state.allowAndSetStorePathString(*expectedStorePath, v);
}
} catch (Error & e) {
e.addTrace(state.positions[pos], "while adding path '%s'", path);
throw;
}
}
SourcePath realisePath(EvalState & state, const PosIdx pos, Value & v, const RealisePathFlags flags)
{
NixStringContext context;
auto path = state.coerceToPath(noPos, v, context, "while realising the context of a path");
try {
StringMap rewrites = state.realiseContext(context);
auto realPath = state.rootPath(CanonPath(state.toRealPath(rewriteStrings(path.path.abs(), rewrites), context)));
return flags.checkForPureEval
? state.checkSourcePath(realPath)
: realPath;
} catch (Error & e) {
e.addTrace(state.positions[pos], "while realising the context of path '%s'", path);
throw;
}
}
/**
* builtins.dirOf
*/
static void prim_dirOf(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
if (args[0]->type() == nPath) {
auto path = args[0]->path();
v.mkPath(path.path.isRoot() ? path : path.parent());
} else {
NixStringContext context;
auto path = state.coerceToString(
pos,
*args[0],
context,
"while evaluating the first argument passed to 'builtins.dirOf'",
false,
false
);
auto dir = dirOf(*path);
v.mkString(dir, context);
}
}
PrimOp primop_dirOf({
.name = "dirOf",
.args = {"s"},
.doc = R"(
Return the directory part of the string *s*, that is, everything
before the final slash in the string. This is similar to the GNU
`dirname` command.
)",
.fun = prim_dirOf,
});
/**
* builtins.filterSource
*/
static void prim_filterSource(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto path = state.coerceToPath(
pos,
*args[1],
context,
"while evaluating the second argument (the path to filter) passed to builtins.filterSource"
);
state.forceFunction(
*args[0], pos, "while evaluating the first argument passed to builtins.filterSource"
);
addPath(
state,
pos,
path.baseName(),
path.path.abs(),
args[0],
FileIngestionMethod::Recursive,
std::nullopt,
v,
context
);
}
PrimOp primop_filterSource({
.name = "__filterSource",
.args = {"e1", "e2"},
.doc = R"(
> **Warning**
>
> `filterSource` should not be used to filter store paths. Since
> `filterSource` uses the name of the input directory while naming
> the output directory, doing so will produce a directory name in
> the form of `<hash2>-<hash>-<name>`, where `<hash>-<name>` is
> the name of the input directory. Since `<hash>` depends on the
> unfiltered directory, the name of the output directory will
> indirectly depend on files that are filtered out by the
> function. This will trigger a rebuild even when a filtered out
> file is changed. Use `builtins.path` instead, which allows
> specifying the name of the output directory.
This function allows you to copy sources into the Nix store while
filtering certain files. For instance, suppose that you want to use
the directory `source-dir` as an input to a Nix expression, e.g.
```nix
stdenv.mkDerivation {
...
src = ./source-dir;
}
```
However, if `source-dir` is a Subversion working copy, then all
those annoying `.svn` subdirectories will also be copied to the
store. Worse, the contents of those directories may change a lot,
causing lots of spurious rebuilds. With `filterSource` you can
filter out the `.svn` directories:
```nix
src = builtins.filterSource
(path: type: type != "directory" || baseNameOf path != ".svn")
./source-dir;
```
Thus, the first argument *e1* must be a predicate function that is
called for each regular file, directory or symlink in the source
tree *e2*. If the function returns `true`, the file is copied to the
Nix store, otherwise it is omitted. The function is called with two
arguments. The first is the full path of the file. The second is a
string that identifies the type of the file, which is either
`"regular"`, `"directory"`, `"symlink"` or `"unknown"` (for other
kinds of files such as device nodes or fifos but note that those
cannot be copied to the Nix store, so if the predicate returns
`true` for them, the copy will fail). If you exclude a directory,
the entire corresponding subtree of *e2* will be excluded.
)",
.fun = prim_filterSource,
});
/**
* builtins.findFile
*/
static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(
*args[0], pos, "while evaluating the first argument passed to builtins.findFile"
);
SearchPath searchPath;
for (auto v2 : args[0]->listItems()) {
state.forceAttrs(
*v2, pos, "while evaluating an element of the list passed to builtins.findFile"
);
std::string prefix;
Bindings::iterator i = v2->attrs->find(state.sPrefix);
if (i != v2->attrs->end()) {
prefix = state.forceStringNoCtx(
*i->value,
pos,
"while evaluating the `prefix` attribute of an element of the list passed to "
"builtins.findFile"
);
}
i = getAttr(state, state.sPath, v2->attrs, "in an element of the __nixPath");
NixStringContext context;
auto path = state
.coerceToString(
pos,
*i->value,
context,
"while evaluating the `path` attribute of an element of the list "
"passed to builtins.findFile",
false,
false
)
.toOwned();
try {
auto rewrites = state.realiseContext(context);
path = rewriteStrings(path, rewrites);
} catch (InvalidPathError & e) {
state.error<EvalError>("cannot find '%1%', since path '%2%' is not valid", path, e.path)
.atPos(pos)
.debugThrow();
}
searchPath.elements.emplace_back(SearchPath::Elem{
.prefix = SearchPath::Prefix{.s = prefix},
.path = SearchPath::Path{.s = path},
});
}
auto path = state.forceStringNoCtx(
*args[1], pos, "while evaluating the second argument passed to builtins.findFile"
);
v.mkPath(state.checkSourcePath(state.findFile(searchPath, path, pos)));
}
PrimOp primop_findFile(PrimOp{
.name = "__findFile",
.args = {"search path", "lookup path"},
.doc = R"(
Look up the given path with the given search path.
A search path is represented list of [attribute sets](./values.md#attribute-set) with two attributes, `prefix`, and `path`.
`prefix` is a relative path.
`path` denotes a file system location; the exact syntax depends on the command line interface.
Examples of search path attribute sets:
- ```
{
prefix = "nixos-config";
path = "/etc/nixos/configuration.nix";
}
```
- ```
{
prefix = "";
path = "/nix/var/nix/profiles/per-user/root/channels";
}
```
The lookup algorithm checks each entry until a match is found, returning a [path value](@docroot@/language/values.html#type-path) of the match.
This is the process for each entry:
If the lookup path matches `prefix`, then the remainder of the lookup path (the "suffix") is searched for within the directory denoted by `patch`.
Note that the `path` may need to be downloaded at this point to look inside.
If the suffix is found inside that directory, then the entry is a match;
the combined absolute path of the directory (now downloaded if need be) and the suffix is returned.
The syntax
```nix
<nixpkgs>
```
is equivalent to:
```nix
builtins.findFile builtins.nixPath "nixpkgs"
```
)",
.fun = prim_findFile,
});
/**
* builtins.outputOf
*/
static void prim_outputOf(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
SingleDerivedPath drvPath = state.coerceToSingleDerivedPath(
pos, *args[0], "while evaluating the first argument to builtins.outputOf"
);
OutputNameView outputName = state.forceStringNoCtx(
*args[1], pos, "while evaluating the second argument to builtins.outputOf"
);
state.mkSingleDerivedPathString(
SingleDerivedPath::Built{
.drvPath = make_ref<SingleDerivedPath>(drvPath),
.output = std::string{outputName},
},
v
);
}
PrimOp primop_outputOf({
.name = "__outputOf",
.args = {"derivation-reference", "output-name"},
.doc = R"(
Return the output path of a derivation, literally or using a placeholder if needed.
If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addresed), the output path will just be returned.
But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), a placeholder will be returned instead.
*`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be a placeholder reference. If the derivation is produced by a derivation, you must explicitly select `drv.outPath`.
This primop can be chained arbitrarily deeply.
For instance,
```nix
builtins.outputOf
(builtins.outputOf myDrv "out)
"out"
```
will return a placeholder for the output of the output of `myDrv`.
This primop corresponds to the `^` sigil for derivable paths, e.g. as part of installable syntax on the command line.
)",
.fun = prim_outputOf,
.experimentalFeature = Xp::DynamicDerivations,
});
/**
* builtins.path
*/
static void prim_path(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
std::optional<SourcePath> path;
std::string name;
Value * filterFun = nullptr;
auto method = FileIngestionMethod::Recursive;
std::optional<Hash> expectedHash;
NixStringContext context;
state.forceAttrs(*args[0], pos, "while evaluating the argument passed to 'builtins.path'");
for (auto & attr : *args[0]->attrs) {
auto n = state.symbols[attr.name];
if (n == "path") {
path.emplace(state.coerceToPath(
attr.pos,
*attr.value,
context,
"while evaluating the 'path' attribute passed to 'builtins.path'"
));
} else if (attr.name == state.sName) {
name = state.forceStringNoCtx(
*attr.value,
attr.pos,
"while evaluating the `name` attribute passed to builtins.path"
);
} else if (n == "filter") {
state.forceFunction(
*(filterFun = attr.value),
attr.pos,
"while evaluating the `filter` parameter passed to builtins.path"
);
} else if (n == "recursive") {
method = FileIngestionMethod{state.forceBool(
*attr.value,
attr.pos,
"while evaluating the `recursive` attribute passed to builtins.path"
)};
} else if (n == "sha256") {
expectedHash = newHashAllowEmpty(
state.forceStringNoCtx(
*attr.value,
attr.pos,
"while evaluating the `sha256` attribute passed to builtins.path"
),
htSHA256
);
} else {
state
.error<EvalError>(
"unsupported argument '%1%' to 'addPath'", state.symbols[attr.name]
)
.atPos(attr.pos)
.debugThrow();
}
}
if (!path) {
state
.error<EvalError>(
"missing required 'path' attribute in the first argument to builtins.path"
)
.atPos(pos)
.debugThrow();
}
if (name.empty()) {
name = path->baseName();
}
addPath(state, pos, name, path->path.abs(), filterFun, method, expectedHash, v, context);
}
PrimOp primop_path({
.name = "__path",
.args = {"args"},
.doc = R"(
An enrichment of the built-in path type, based on the attributes
present in *args*. All are optional except `path`:
- path\
The underlying path.
- name\
The name of the path when added to the store. This can used to
reference paths that have nix-illegal characters in their names,
like `@`.
- filter\
A function of the type expected by `builtins.filterSource`,
with the same semantics.
- recursive\
When `false`, when `path` is added to the store it is with a
flat hash, rather than a hash of the NAR serialization of the
file. Thus, `path` must refer to a regular file, not a
directory. This allows similar behavior to `fetchurl`. Defaults
to `true`.
- sha256\
When provided, this is the expected hash of the file at the
path. Evaluation will fail if the hash is incorrect, and
providing a hash allows `builtins.path` to be used even when the
`pure-eval` nix config option is on.
)",
.fun = prim_path,
});
/**
* builtins.pathExists
*/
static void prim_pathExists(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto & arg = *args[0];
/* We dont check the path right now, because we dont want to
throw if the path isnt allowed, but just return false (and we
cant just catch the exception here because we still want to
throw if something in the evaluation of `arg` tries to
access an unauthorized path). */
auto path = realisePath(state, pos, arg, {.checkForPureEval = false});
/* SourcePath doesn't know about trailing slash. */
auto mustBeDir =
arg.type() == nString && (arg.str().ends_with("/") || arg.str().ends_with("/."));
try {
auto checked = state.checkSourcePath(path).resolveSymlinks(
mustBeDir ? SymlinkResolution::Full : SymlinkResolution::Ancestors
);
auto st = checked.maybeLstat();
auto exists = st && (!mustBeDir || st->type == InputAccessor::tDirectory);
v.mkBool(exists);
} catch (SysError & e) {
/* Don't give away info from errors while canonicalising
path in restricted mode. */
v.mkBool(false);
} catch (RestrictedPathError & e) {
v.mkBool(false);
}
}
PrimOp primop_pathExists({
.name = "__pathExists",
.args = {"path"},
.doc = R"(
Return `true` if the path *path* exists at evaluation time, and
`false` otherwise.
)",
.fun = prim_pathExists,
});
/**
* builtins.palceholder
*/
/* Return a placeholder string for the specified output that will be
substituted by the corresponding output path at build time. For
example, 'placeholder "out"' returns the string
/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9. At build
time, any occurrence of this string in an derivation attribute will
be replaced with the concrete path in the Nix store of the output
out. */
static void prim_placeholder(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
v.mkString(hashPlaceholder(state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.placeholder"
)));
}
PrimOp primop_placeholder({
.name = "placeholder",
.args = {"output"},
.doc = R"(
Return a placeholder string for the specified *output* that will be
substituted by the corresponding output path at build time. Typical
outputs would be `"out"`, `"bin"` or `"dev"`.
)",
.fun = prim_placeholder,
});
/**
* builtins.toPath
* WARNING: deprecated
*/
static void prim_toPath(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto path = state.coerceToPath(
pos, *args[0], context, "while evaluating the first argument passed to builtins.toPath"
);
v.mkString(path.path.abs(), context);
}
PrimOp primop_toPath({
.name = "__toPath",
.args = {"s"},
.doc = R"(
**DEPRECATED.** Use `/. + "/path"` to convert a string into an absolute
path. For relative paths, use `./. + "/path"`.
)",
.fun = prim_toPath,
});
/**
* builtins.readDir
*/
static void prim_readDir(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
// Retrieve directory entries for all nodes in a directory.
// This is similar to `getFileType` but is optimized to reduce system calls
// on many systems.
auto entries = path.readDirectory();
auto attrs = state.buildBindings(entries.size());
// If we hit unknown directory entry types we may need to fallback to
// using `getFileType` on some systems.
// In order to reduce system calls we make each lookup lazy by using
// `builtins.readFileType` application.
Value * readFileType = nullptr;
for (auto & [name, type] : entries) {
auto & attr = attrs.alloc(name);
if (!type) {
// Some filesystems or operating systems may not be able to return
// detailed node info quickly in this case we produce a thunk to
// query the file type lazily.
auto epath = state.allocValue();
epath->mkPath(path + name);
if (!readFileType) {
readFileType = &state.getBuiltin("readFileType");
}
attr.mkApp(readFileType, epath);
} else {
// This branch of the conditional is much more likely.
// Here we just stringize the directory entry type.
attr.mkString(fileTypeToString(*type));
}
}
v.mkAttrs(attrs);
}
PrimOp primop_readDir({
.name = "__readDir",
.args = {"path"},
.doc = R"(
Return the contents of the directory *path* as a set mapping
directory entries to the corresponding file type. For instance, if
directory `A` contains a regular file `B` and another directory
`C`, then `builtins.readDir ./A` will return the set
```nix
{ B = "regular"; C = "directory"; }
```
The possible values for the file type are `"regular"`,
`"directory"`, `"symlink"` and `"unknown"`.
)",
.fun = prim_readDir,
});
/**
* builtins.readFile
*/
static void prim_readFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto s = path.readFile();
if (s.find((char) 0) != std::string::npos) {
state
.error<EvalError>(
"the contents of the file '%1%' cannot be represented as a Nix string", path
)
.atPos(pos)
.debugThrow();
}
StorePathSet refs;
if (state.store->isInStore(path.path.abs())) {
try {
refs = state.store->queryPathInfo(state.store->toStorePath(path.path.abs()).first)
->references;
} catch (Error &) { // FIXME: should be InvalidPathError
}
// Re-scan references to filter down to just the ones that actually occur in the file.
auto refsSink = PathRefScanSink::fromPaths(refs);
refsSink << s;
refs = refsSink.getResultPaths();
}
NixStringContext context;
for (auto && p : std::move(refs)) {
context.insert(NixStringContextElem::Opaque{
.path = std::move((StorePath &&) p),
});
}
v.mkString(s, context);
}
PrimOp primop_readFile({
.name = "__readFile",
.args = {"path"},
.doc = R"(
Return the contents of the file *path* as a string.
)",
.fun = prim_readFile,
});
/**
* builtins.readFileType
*/
static void prim_readFileType(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
/* Retrieve the directory entry type and stringize it. */
v.mkString(fileTypeToString(path.lstat().type));
}
PrimOp primop_readFileType({
.name = "__readFileType",
.args = {"p"},
.doc = R"(
Determine the directory entry type of a filesystem node, being
one of "directory", "regular", "symlink", or "unknown".
)",
.fun = prim_readFileType,
});
/**
* builtins.storePath
*/
static void prim_storePath(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
if (evalSettings.pureEval) {
state.error<EvalError>("'%s' is not allowed in pure evaluation mode", "builtins.storePath")
.atPos(pos)
.debugThrow();
}
NixStringContext context;
auto path = state
.checkSourcePath(state.coerceToPath(
pos,
*args[0],
context,
"while evaluating the first argument passed to builtins.storePath"
))
.path;
/* Resolve symlinks in path, unless path itself is a symlink
directly in the store. The latter condition is necessary so
e.g. nix-push does the right thing. */
if (!state.store->isStorePath(path.abs())) {
path = CanonPath(canonPath(path.abs(), true));
}
if (!state.store->isInStore(path.abs())) {
state.error<EvalError>("path '%1%' is not in the Nix store", path).atPos(pos).debugThrow();
}
auto path2 = state.store->toStorePath(path.abs()).first;
if (!settings.readOnlyMode) {
state.store->ensurePath(path2);
}
context.insert(NixStringContextElem::Opaque{.path = path2});
v.mkString(path.abs(), context);
}
PrimOp primop_storePath({
.name = "__storePath",
.args = {"path"},
.doc = R"(
This function allows you to define a dependency on an already
existing store path. For example, the derivation attribute `src
= builtins.storePath /nix/store/f1d18v1y-source` causes the
derivation to depend on the specified path, which must exist or
be substitutable. Note that this differs from a plain path
(e.g. `src = /nix/store/f1d18v1y-source`) in that the latter
causes the path to be *copied* again to the Nix store, resulting
in a new path (e.g. `/nix/store/ld01dnzc-source-source`).
Not available in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval).
See also [`builtins.fetchClosure`](#builtins-fetchClosure).
)",
.fun = prim_storePath,
});
/**
* builtins.toFile
*/
static void prim_toFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
std::string name(state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.toFile"
));
std::string contents(state.forceString(
*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile"
));
StorePathSet refs;
for (auto c : context) {
if (auto p = std::get_if<NixStringContextElem::Opaque>(&c.raw)) {
refs.insert(p->path);
} else {
state
.error<EvalError>(
"files created by %1% may not reference derivations, but %2% references %3%",
"builtins.toFile",
name,
c.to_string()
)
.atPos(pos)
.debugThrow();
}
}
auto storePath = settings.readOnlyMode
? state.store->computeStorePathForText(name, contents, refs)
: state.store->addTextToStore(name, contents, refs, state.repair);
/* Note: we don't need to add `context' to the context of the
result, since `storePath' itself has references to the paths
used in args[1]. */
/* Add the output of this to the allowed paths. */
state.allowAndSetStorePathString(storePath, v);
}
PrimOp primop_toFile({
.name = "__toFile",
.args = {"name", "s"},
.doc = R"(
Store the string *s* in a file in the Nix store and return its
path. The file has suffix *name*. This file can be used as an
input to derivations. One application is to write builders
inline. For instance, the following Nix expression combines the
Nix expression for GNU Hello and its build script into one file:
```nix
{ stdenv, fetchurl, perl }:
stdenv.mkDerivation {
name = "hello-2.1.1";
builder = builtins.toFile "builder.sh" "
source $stdenv/setup
PATH=$perl/bin:$PATH
tar xvfz $src
cd hello-*
./configure --prefix=$out
make
make install
";
src = fetchurl {
url = "http://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
};
inherit perl;
}
```
It is even possible for one file to refer to another, e.g.,
```nix
builder = let
configFile = builtins.toFile "foo.conf" "
# This is some dummy configuration file.
...
";
in builtins.toFile "builder.sh" "
source $stdenv/setup
...
cp ${configFile} $out/etc/foo.conf
";
```
Note that `${configFile}` is a
[string interpolation](@docroot@/language/values.md#type-string), so the result of the
expression `configFile`
(i.e., a path like `/nix/store/m7p7jfny445k...-foo.conf`) will be
spliced into the resulting string.
It is however *not* allowed to have files mutually referring to each
other, like so:
```nix
let
foo = builtins.toFile "foo" "...${bar}...";
bar = builtins.toFile "bar" "...${foo}...";
in foo
```
This is not allowed because it would cause a cyclic dependency in
the computation of the cryptographic hashes for `foo` and `bar`.
It is also not possible to reference the result of a derivation. If
you are using Nixpkgs, the `writeTextFile` function is able to do
that.
)",
.fun = prim_toFile,
});
}
-641
View File
@@ -1,641 +0,0 @@
#include <assert.h> // for assert
#include <stddef.h> // for size_t
#include <iterator> // for distance
#include <memory> // for shared_ptr, __shared_ptr_access, make_sh...
#include <regex> // for regex_error, match_results, cregex_iterator
#include <string> // for basic_string, char_traits, string
#include <string_view> // for basic_string_view
#include <unordered_map> // for unordered_map, operator==, _Node_iterator
#include <utility> // for move, pair
#include <vector> // for vector
#include "attr-set.hh" // for BindingsBuilder
#include "eval-error.hh" // for EvalError, EvalErrorBuilder
#include "eval.hh" // for EvalState, PrimOp, makeRegexCache, prim_...
#include "file-system.hh" // for baseNameOf
#include "names.hh" // for DrvName, compareVersions, nextComponent
#include "pos-idx.hh" // for PosIdx
#include "primops.hh" // for RegexCache
#include "symbol-table.hh" // for Symbol
#include "types.hh" // for BackedStringView, enumerate, Strings
#include "value.hh" // for Value, nString
#include "value/context.hh" // for NixStringContext
namespace nix {
std::shared_ptr<RegexCache> makeRegexCache()
{
return std::make_shared<RegexCache>();
}
/**
* builtins.baseNameOf
*/
static void prim_baseNameOf(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
v.mkString(
baseNameOf(*state.coerceToString(
pos,
*args[0],
context,
"while evaluating the first argument passed to builtins.baseNameOf",
false,
false
)),
context
);
}
PrimOp primop_baseNameOf({
.name = "baseNameOf",
.args = {"s"},
.doc = R"(
Return the *base name* of the string *s*, that is, everything
following the final slash in the string. This is similar to the GNU
`basename` command.
)",
.fun = prim_baseNameOf,
});
/**
* builtins.compareVersions
*/
static void prim_compareVersions(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto version1 = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.compareVersions"
);
auto version2 = state.forceStringNoCtx(
*args[1], pos, "while evaluating the second argument passed to builtins.compareVersions"
);
v.mkInt(compareVersions(version1, version2));
}
PrimOp primop_compareVersions({
.name = "__compareVersions",
.args = {"s1", "s2"},
.doc = R"(
Compare two strings representing versions and return `-1` if
version *s1* is older than version *s2*, `0` if they are the same,
and `1` if *s1* is newer than *s2*. The version comparison
algorithm is the same as the one used by [`nix-env
-u`](../command-ref/nix-env.md#operation---upgrade).
)",
.fun = prim_compareVersions,
});
/**
* builtins.concatStringsSep
*/
static void prim_concatStringsSep(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto sep = state.forceString(
*args[0],
context,
pos,
"while evaluating the first argument (the separator string) passed to "
"builtins.concatStringsSep"
);
state.forceList(
*args[1],
pos,
"while evaluating the second argument (the list of strings to concat) passed to "
"builtins.concatStringsSep"
);
std::string res;
res.reserve((args[1]->listSize() + 32) * sep.size());
bool first = true;
for (auto elem : args[1]->listItems()) {
if (first) {
first = false;
} else {
res += sep;
}
res += *state.coerceToString(
pos,
*elem,
context,
"while evaluating one element of the list of strings to concat passed to "
"builtins.concatStringsSep"
);
}
v.mkString(res, context);
}
PrimOp primop_concatStringsSep(
{.name = "__concatStringsSep",
.args = {"separator", "list"},
.doc = R"(
Concatenate a list of strings with a separator between each
element, e.g. `concatStringsSep "/" ["usr" "local" "bin"] ==
"usr/local/bin"`.
)",
.fun = prim_concatStringsSep}
);
/**
* builtins.match
*/
void prim_match(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto re = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.match"
);
try {
auto regex = state.regexCache->get(re);
NixStringContext context;
const auto str = state.forceString(
*args[1], context, pos, "while evaluating the second argument passed to builtins.match"
);
std::cmatch match;
if (!std::regex_match(str.begin(), str.end(), match, regex)) {
v.mkNull();
return;
}
// the first match is the whole string
const size_t len = match.size() - 1;
state.mkList(v, len);
for (size_t i = 0; i < len; ++i) {
if (!match[i + 1].matched) {
(v.listElems()[i] = state.allocValue())->mkNull();
} else {
(v.listElems()[i] = state.allocValue())->mkString(match[i + 1].str());
}
}
} catch (std::regex_error & e) {
if (e.code() == std::regex_constants::error_space) {
// limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
state.error<EvalError>("memory limit exceeded by regular expression '%s'", re)
.atPos(pos)
.debugThrow();
} else {
state.error<EvalError>("invalid regular expression '%s'", re).atPos(pos).debugThrow();
}
}
}
PrimOp primop_match({
.name = "__match",
.args = {"regex", "str"},
.doc = R"s(
Returns a list if the [extended POSIX regular
expression](http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04)
*regex* matches *str* precisely, otherwise returns `null`. Each item
in the list is a regex group.
```nix
builtins.match "ab" "abc"
```
Evaluates to `null`.
```nix
builtins.match "abc" "abc"
```
Evaluates to `[ ]`.
```nix
builtins.match "a(b)(c)" "abc"
```
Evaluates to `[ "b" "c" ]`.
```nix
builtins.match "[[:space:]]+([[:upper:]]+)[[:space:]]+" " FOO "
```
Evaluates to `[ "FOO" ]`.
)s",
.fun = prim_match,
});
/**
* builtins.parseDrvName
*/
static void prim_parseDrvName(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto name = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.parseDrvName"
);
DrvName parsed(name);
auto attrs = state.buildBindings(2);
attrs.alloc(state.sName).mkString(parsed.name);
attrs.alloc("version").mkString(parsed.version);
v.mkAttrs(attrs);
}
PrimOp primop_parseDrvName({
.name = "__parseDrvName",
.args = {"s"},
.doc = R"(
Split the string *s* into a package name and version. The package
name is everything up to but not including the first dash not followed
by a letter, and the version is everything following that dash. The
result is returned in a set `{ name, version }`. Thus,
`builtins.parseDrvName "nix-0.12pre12876"` returns `{ name =
"nix"; version = "0.12pre12876"; }`.
)",
.fun = prim_parseDrvName,
});
/**
* builtins.replaceStrings
*/
static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(
*args[0], pos, "while evaluating the first argument passed to builtins.replaceStrings"
);
state.forceList(
*args[1], pos, "while evaluating the second argument passed to builtins.replaceStrings"
);
if (args[0]->listSize() != args[1]->listSize()) {
state
.error<EvalError>(
"'from' and 'to' arguments passed to builtins.replaceStrings have different lengths"
)
.atPos(pos)
.debugThrow();
}
std::vector<std::string> from;
from.reserve(args[0]->listSize());
for (auto elem : args[0]->listItems()) {
from.emplace_back(state.forceString(
*elem,
pos,
"while evaluating one of the strings to replace passed to builtins.replaceStrings"
));
}
std::unordered_map<size_t, std::string> cache;
auto to = args[1]->listItems();
NixStringContext context;
auto s = state.forceString(
*args[2],
context,
pos,
"while evaluating the third argument passed to builtins.replaceStrings"
);
std::string res;
// Loops one past last character to handle the case where 'from' contains an empty string.
for (size_t p = 0; p <= s.size();) {
bool found = false;
auto i = from.begin();
auto j = to.begin();
size_t j_index = 0;
for (; i != from.end(); ++i, ++j, ++j_index) {
if (s.compare(p, i->size(), *i) == 0) {
found = true;
auto v = cache.find(j_index);
if (v == cache.end()) {
NixStringContext ctx;
auto ts = state.forceString(
**j,
ctx,
pos,
"while evaluating one of the replacement strings passed to "
"builtins.replaceStrings"
);
v = (cache.emplace(j_index, ts)).first;
for (auto & path : ctx) {
context.insert(path);
}
}
res += v->second;
if (i->empty()) {
if (p < s.size()) {
res += s[p];
}
p++;
} else {
p += i->size();
}
break;
}
}
if (!found) {
if (p < s.size()) {
res += s[p];
}
p++;
}
}
v.mkString(res, context);
}
PrimOp primop_replaceStrings({
.name = "__replaceStrings",
.args = {"from", "to", "s"},
.doc = R"(
Given string *s*, replace every occurrence of the strings in *from*
with the corresponding string in *to*.
The argument *to* is lazy, that is, it is only evaluated when its corresponding pattern in *from* is matched in the string *s*
Example:
```nix
builtins.replaceStrings ["oo" "a"] ["a" "i"] "foobar"
```
evaluates to `"fabir"`.
)",
.fun = prim_replaceStrings,
});
/**
* builtins.split
*/
void prim_split(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto re = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.split"
);
try {
auto regex = state.regexCache->get(re);
NixStringContext context;
const auto str = state.forceString(
*args[1], context, pos, "while evaluating the second argument passed to builtins.split"
);
auto begin = std::cregex_iterator(str.begin(), str.end(), regex);
auto end = std::cregex_iterator();
// Any matches results are surrounded by non-matching results.
const size_t len = std::distance(begin, end);
state.mkList(v, 2 * len + 1);
size_t idx = 0;
if (len == 0) {
v.listElems()[idx++] = args[1];
return;
}
for (auto i = begin; i != end; ++i) {
assert(idx <= 2 * len + 1 - 3);
auto match = *i;
// Add a string for non-matched characters.
(v.listElems()[idx++] = state.allocValue())->mkString(match.prefix().str());
// Add a list for matched substrings.
const size_t slen = match.size() - 1;
auto elem = v.listElems()[idx++] = state.allocValue();
// Start at 1, beacause the first match is the whole string.
state.mkList(*elem, slen);
for (size_t si = 0; si < slen; ++si) {
if (!match[si + 1].matched) {
(elem->listElems()[si] = state.allocValue())->mkNull();
} else {
(elem->listElems()[si] = state.allocValue())->mkString(match[si + 1].str());
}
}
// Add a string for non-matched suffix characters.
if (idx == 2 * len) {
(v.listElems()[idx++] = state.allocValue())->mkString(match.suffix().str());
}
}
assert(idx == 2 * len + 1);
} catch (std::regex_error & e) {
if (e.code() == std::regex_constants::error_space) {
// limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
state.error<EvalError>("memory limit exceeded by regular expression '%s'", re)
.atPos(pos)
.debugThrow();
} else {
state.error<EvalError>("invalid regular expression '%s'", re).atPos(pos).debugThrow();
}
}
}
PrimOp primop_split({
.name = "__split",
.args = {"regex", "str"},
.doc = R"s(
Returns a list composed of non matched strings interleaved with the
lists of the [extended POSIX regular
expression](http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04)
*regex* matches of *str*. Each item in the lists of matched
sequences is a regex group.
```nix
builtins.split "(a)b" "abc"
```
Evaluates to `[ "" [ "a" ] "c" ]`.
```nix
builtins.split "([ac])" "abc"
```
Evaluates to `[ "" [ "a" ] "b" [ "c" ] "" ]`.
```nix
builtins.split "(a)|(c)" "abc"
```
Evaluates to `[ "" [ "a" null ] "b" [ null "c" ] "" ]`.
```nix
builtins.split "([[:upper:]]+)" " FOO "
```
Evaluates to `[ " " [ "FOO" ] " " ]`.
)s",
.fun = prim_split,
});
/**
* builtins.splitVersion
*/
static void prim_splitVersion(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto version = state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.splitVersion"
);
auto iter = version.cbegin();
Strings components;
while (iter != version.cend()) {
auto component = nextComponent(iter, version.cend());
if (component.empty()) {
break;
}
components.emplace_back(component);
}
state.mkList(v, components.size());
for (const auto & [n, component] : enumerate(components)) {
(v.listElems()[n] = state.allocValue())->mkString(std::move(component));
}
}
PrimOp primop_splitVersion({
.name = "__splitVersion",
.args = {"s"},
.doc = R"(
Split a string representing a version into its components, by the
same version splitting logic underlying the version comparison in
[`nix-env -u`](../command-ref/nix-env.md#operation---upgrade).
)",
.fun = prim_splitVersion,
});
/**
* builtins.stringLength
*/
static void prim_stringLength(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(
pos, *args[0], context, "while evaluating the argument passed to builtins.stringLength"
);
v.mkInt(s->size());
}
PrimOp primop_stringLength({
.name = "__stringLength",
.args = {"e"},
.doc = R"(
Return the length of the string *e*. If *e* is not a string,
evaluation is aborted.
)",
.fun = prim_stringLength,
});
/**
* builtins.substring
*/
static void prim_substring(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
int start = state.forceInt(
*args[0],
pos,
"while evaluating the first argument (the start offset) passed to builtins.substring"
);
if (start < 0) {
state.error<EvalError>("negative start position in 'substring'").atPos(pos).debugThrow();
}
int len = state.forceInt(
*args[1],
pos,
"while evaluating the second argument (the substring length) passed to builtins.substring"
);
// Special-case on empty substring to avoid O(n) strlen
// This allows for the use of empty substrings to efficently capture string context
if (len == 0) {
state.forceValue(*args[2], pos);
if (args[2]->type() == nString) {
v.mkString("", args[2]->string.context);
return;
}
}
NixStringContext context;
auto s = state.coerceToString(
pos,
*args[2],
context,
"while evaluating the third argument (the string) passed to builtins.substring"
);
v.mkString((unsigned int) start >= s->size() ? "" : s->substr(start, len), context);
}
PrimOp primop_substring({
.name = "__substring",
.args = {"start", "len", "s"},
.doc = R"(
Return the substring of *s* from character position *start*
(zero-based) up to but not including *start + len*. If *start* is
greater than the length of the string, an empty string is returned,
and if *start + len* lies beyond the end of the string, only the
substring up to the end of the string is returned. *start* must be
non-negative. For example,
```nix
builtins.substring 0 3 "nixos"
```
evaluates to `"nix"`.
)",
.fun = prim_substring,
});
/**
* builtins.toString
*/
static void prim_toString(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
auto s = state.coerceToString(
pos,
*args[0],
context,
"while evaluating the first argument passed to builtins.toString",
true,
false
);
v.mkString(*s, context);
}
PrimOp primop_toString({
.name = "toString",
.args = {"e"},
.doc = R"(
Convert the expression *e* to a string. *e* can be:
- A string (in which case the string is returned unmodified).
- A path (e.g., `toString /foo/bar` yields `"/foo/bar"`.
- A set containing `{ __toString = self: ...; }` or `{ outPath = ...; }`.
- An integer.
- A list, in which case the string representations of its elements
are joined with spaces.
- A Boolean (`false` yields `""`, `true` yields `"1"`).
- `null`, which yields the empty string.
)",
.fun = prim_toString,
});
}
-116
View File
@@ -1,116 +0,0 @@
#include <string> // for basic_string, char...
#include <utility> // for move
#include "canon-path.hh" // for CanonPath
#include "environment-variables.hh"
#include "error.hh" // for Error
#include "eval-error.hh" // for EvalError, EvalErr...
#include "eval-settings.hh" // for EvalSettings, eval...
#include "eval.hh" // for EvalState, PrimOp
#include "pos-idx.hh" // for PosIdx
#include "pos-table.hh" // for PosTable
#include "primops.hh" // for prim_exec
#include "processes.hh"
#include "types.hh" // for BackedStringView
#include "value.hh" // for Value
#include "value/context.hh" // for NixStringContext
namespace nix {
struct Expr;
/**
* builtins.exec
*/
void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.exec");
auto elems = args[0]->listElems();
auto count = args[0]->listSize();
if (count == 0) {
state.error<EvalError>("at least one argument to 'exec' required").atPos(pos).debugThrow();
}
NixStringContext context;
auto program =
state
.coerceToString(
pos,
*elems[0],
context,
"while evaluating the first element of the argument passed to builtins.exec",
false,
false
)
.toOwned();
Strings commandArgs;
for (unsigned int i = 1; i < args[0]->listSize(); ++i) {
commandArgs.push_back(
state
.coerceToString(
pos,
*elems[i],
context,
"while evaluating an element of the argument passed to builtins.exec",
false,
false
)
.toOwned()
);
}
try {
auto _ = state.realiseContext(context); // FIXME: Handle CA derivations
} catch (InvalidPathError & e) {
state
.error<EvalError>(
"cannot execute '%1%', since path '%2%' is not valid", program, e.path
)
.atPos(pos)
.debugThrow();
}
auto output = runProgram(program, true, commandArgs);
Expr * parsed;
try {
parsed = state.parseExprFromString(std::move(output), state.rootPath(CanonPath::root));
} catch (Error & e) {
e.addTrace(state.positions[pos], "while parsing the output from '%1%'", program);
throw;
}
try {
state.eval(parsed, v);
} catch (Error & e) {
e.addTrace(state.positions[pos], "while evaluating the output from '%1%'", program);
throw;
}
}
/**
* builtins.getEnv
*/
static void prim_getEnv(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
std::string name(state.forceStringNoCtx(
*args[0], pos, "while evaluating the first argument passed to builtins.getEnv"
));
v.mkString(evalSettings.restrictEval || evalSettings.pureEval ? "" : getEnv(name).value_or(""));
}
PrimOp primop_getEnv({
.name = "__getEnv",
.args = {"s"},
.doc = R"(
`getEnv` returns the value of the environment variable *s*, or an
empty string if the variable doesnt exist. This function should be
used with care, as it can introduce all sorts of nasty environment
dependencies in your Nix expression.
`getEnv` is used in Nix Packages to locate the file
`~/.nixpkgs/config.nix`, which contains user-local settings for Nix
Packages. (That is, it does a `getEnv "HOME"` to locate the users
home directory.)
)",
.fun = prim_getEnv,
});
}
-119
View File
@@ -1,119 +0,0 @@
#include <sstream> // for basic_ostringstream, basic_ios, basic_os...
#include "eval.hh" // for PrimOp, EvalState (ptr only)
#include "pos-idx.hh" // for PosIdx
#include "value.hh" // for Value
#include "value-to-xml.hh" // for printValueAsXML
#include "value/context.hh" // for NixStringContext
namespace nix {
/**
* builtins.toXML
*/
static void prim_toXML(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
std::ostringstream out;
NixStringContext context;
printValueAsXML(state, true, false, *args[0], out, context, pos);
v.mkString(out.str(), context);
}
PrimOp primop_toXML({
.name = "__toXML",
.args = {"e"},
.doc = R"(
Return a string containing an XML representation of *e*. The main
application for `toXML` is to communicate information with the
builder in a more structured format than plain environment
variables.
Here is an example where this is the case:
```nix
{ stdenv, fetchurl, libxslt, jira, uberwiki }:
stdenv.mkDerivation (rec {
name = "web-server";
buildInputs = [ libxslt ];
builder = builtins.toFile "builder.sh" "
source $stdenv/setup
mkdir $out
echo "$servlets" | xsltproc ${stylesheet} - > $out/server-conf.xml
";
stylesheet = builtins.toFile "stylesheet.xsl"
"<?xml version='1.0' encoding='UTF-8'?>
<xsl:stylesheet xmlns:xsl='http://www.w3.org/1999/XSL/Transform' version='1.0'>
<xsl:template match='/'>
<Configure>
<xsl:for-each select='/expr/list/attrs'>
<Call name='addWebApplication'>
<Arg><xsl:value-of select=\"attr[@name = 'path']/string/@value\" /></Arg>
<Arg><xsl:value-of select=\"attr[@name = 'war']/path/@value\" /></Arg>
</Call>
</xsl:for-each>
</Configure>
</xsl:template>
</xsl:stylesheet>
";
servlets = builtins.toXML [
{ path = "/bugtracker"; war = jira + "/lib/atlassian-jira.war"; }
{ path = "/wiki"; war = uberwiki + "/uberwiki.war"; }
];
})
```
The builder is supposed to generate the configuration file for a
[Jetty servlet container](http://jetty.mortbay.org/). A servlet
container contains a number of servlets (`*.war` files) each
exported under a specific URI prefix. So the servlet configuration
is a list of sets containing the `path` and `war` of the servlet
(). This kind of information is difficult to communicate with the
normal method of passing information through an environment
variable, which just concatenates everything together into a
string (which might just work in this case, but wouldnt work if
fields are optional or contain lists themselves). Instead the Nix
expression is converted to an XML representation with `toXML`,
which is unambiguous and can easily be processed with the
appropriate tools. For instance, in the example an XSLT stylesheet
(at point ) is applied to it (at point ) to generate the XML
configuration file for the Jetty server. The XML representation
produced at point by `toXML` is as follows:
```xml
<?xml version='1.0' encoding='utf-8'?>
<expr>
<list>
<attrs>
<attr name="path">
<string value="/bugtracker" />
</attr>
<attr name="war">
<path value="/nix/store/d1jh9pasa7k2...-jira/lib/atlassian-jira.war" />
</attr>
</attrs>
<attrs>
<attr name="path">
<string value="/wiki" />
</attr>
<attr name="war">
<path value="/nix/store/y6423b1yi4sx...-uberwiki/uberwiki.war" />
</attr>
</attrs>
</list>
</expr>
```
Note that we used the `toFile` built-in to write the builder and
the stylesheet inline in the Nix expression. The path of the
stylesheet is spliced into the builder using the syntax `xsltproc
${stylesheet}`.
)",
.fun = prim_toXML,
});
}
-245
View File
@@ -1,245 +0,0 @@
#include <stdlib.h> // for abort
#include <string> // for basic_string, char_traits, string
#include "attr-set.hh" // for BindingsBuilder
#include "eval-error.hh" // for TypeError, EvalErrorBuilder
#include "eval.hh" // for PrimOp, EvalState
#include "nixexpr.hh" // for ExprLambda, Formal, Formals
#include "pos-idx.hh" // for PosIdx
#include "symbol-table.hh" // for Symbol
#include "value.hh" // for Value, nAttrs, nBool, nFloat, nFunction
namespace nix {
/**
* Generic isType function
*/
static inline auto prim_isType(auto n)
{
return [n](EvalState & state, const PosIdx pos, Value ** args, Value & v) {
state.forceValue(*args[0], pos);
v.mkBool(args[0]->type() == n);
};
}
/**
* builtins.functionArgs
*/
static void prim_functionArgs(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
if (args[0]->isPrimOpApp() || args[0]->isPrimOp()) {
v.mkAttrs(&state.emptyBindings);
return;
}
if (!args[0]->isLambda()) {
state.error<TypeError>("'functionArgs' requires a function").atPos(pos).debugThrow();
}
if (!args[0]->lambda.fun->hasFormals()) {
v.mkAttrs(&state.emptyBindings);
return;
}
auto attrs = state.buildBindings(args[0]->lambda.fun->formals->formals.size());
for (auto & i : args[0]->lambda.fun->formals->formals) {
// !!! should optimise booleans (allocate only once)
attrs.alloc(i.name, i.pos).mkBool(i.def);
}
v.mkAttrs(attrs);
}
PrimOp primop_functionArgs({
.name = "__functionArgs",
.args = {"f"},
.doc = R"(
Return a set containing the names of the formal arguments expected
by the function *f*. The value of each attribute is a Boolean
denoting whether the corresponding argument has a default value. For
instance, `functionArgs ({ x, y ? 123}: ...) = { x = false; y =
true; }`.
"Formal argument" here refers to the attributes pattern-matched by
the function. Plain lambdas are not included, e.g. `functionArgs (x:
...) = { }`.
)",
.fun = prim_functionArgs,
});
/**
* builtins.isAttrs
*/
PrimOp primop_isAttrs({
.name = "__isAttrs",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a set, and `false` otherwise.
)",
.fun = prim_isType(nAttrs),
});
/**
* builtins.isBool
*/
PrimOp primop_isBool({
.name = "__isBool",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a bool, and `false` otherwise.
)",
.fun = prim_isType(nBool),
});
/**
* builtins.Float
*/
PrimOp primop_isFloat({
.name = "__isFloat",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a float, and `false` otherwise.
)",
.fun = prim_isType(nFloat),
});
/**
* builtins.isFunction
*/
PrimOp primop_isFunction({
.name = "__isFunction",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a function, and `false` otherwise.
)",
.fun = prim_isType(nFunction),
});
/**
* builtins.isInt
*/
PrimOp primop_isInt({
.name = "__isInt",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to an integer, and `false` otherwise.
)",
.fun = prim_isType(nInt),
});
/**
* builtins.isList
*/
PrimOp primop_isList({
.name = "__isList",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a list, and `false` otherwise.
)",
.fun = prim_isType(nList),
});
/**
* builtins.isNull
*/
PrimOp primop_isNull({
.name = "isNull",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to `null`, and `false` otherwise.
This is equivalent to `e == null`.
)",
.fun = prim_isType(nNull),
});
/**
* builtins.isPath
*/
PrimOp primop_isPath({
.name = "__isPath",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a path, and `false` otherwise.
)",
.fun = prim_isType(nPath),
});
/**
* builtins.isString
*/
PrimOp primop_isString({
.name = "__isString",
.args = {"e"},
.doc = R"(
Return `true` if *e* evaluates to a string, and `false` otherwise.
)",
.fun = prim_isType(nString),
});
/**
* builtins.typeOf
*/
static void prim_typeOf(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceValue(*args[0], pos);
std::string t;
switch (args[0]->type()) {
case nInt:
t = "int";
break;
case nBool:
t = "bool";
break;
case nString:
t = "string";
break;
case nPath:
t = "path";
break;
case nNull:
t = "null";
break;
case nAttrs:
t = "set";
break;
case nList:
t = "list";
break;
case nFunction:
t = "lambda";
break;
case nExternal:
t = args[0]->external->typeOf();
break;
case nFloat:
t = "float";
break;
case nThunk:
abort();
}
v.mkString(t);
}
PrimOp primop_typeOf({
.name = "__typeOf",
.args = {"e"},
.doc = R"(
Return a string representing the type of the value *e*, namely
`"int"`, `"bool"`, `"string"`, `"path"`, `"null"`, `"set"`,
`"list"`, `"lambda"` or `"float"`.
)",
.fun = prim_typeOf,
});
}
+1
View File
@@ -1,6 +1,7 @@
#include <limits>
#include <span>
#include <unordered_set>
#include <sstream>
#include "escape-string.hh"
#include "print.hh"
+11 -1
View File
@@ -87,7 +87,17 @@ struct FetchSettings : public Config
{}, true, Xp::Flakes};
Setting<bool> acceptFlakeConfig{this, false, "accept-flake-config",
"Whether to accept nix configuration from a flake without prompting.",
R"(
Whether to accept Lix configuration from the `nixConfig` attribute of
a flake without prompting. This is almost always a very bad idea.
Setting this setting as a trusted user allows Nix flakes to gain root
access on your machine if they set one of the several
trusted-user-only settings that execute commands as root.
See [multi-user installations](@docroot@/installation/multi-user.md)
for more details on the Lix security model.
)",
{}, true, Xp::Flakes};
Setting<std::string> commitLockFileSummary{
+9 -3
View File
@@ -322,9 +322,15 @@ struct GitLabInputScheme : GitArchiveInputScheme
readFile(
store->toRealPath(
downloadFile(store, url, "source", false, headers).storePath)));
auto rev = Hash::parseAny(std::string(json[0]["id"]), htSHA1);
debug("HEAD revision for '%s' is %s", url, rev.gitRev());
return rev;
if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) {
auto rev = Hash::parseAny(std::string(json[0]["id"]), htSHA1);
debug("HEAD revision for '%s' is %s", url, rev.gitRev());
return rev;
} else if (json.is_array() && json.size() == 0) {
throw Error("No commits returned by GitLab API -- does the ref really exist?");
} else {
throw Error("Didn't know what to do with response from GitLab: %s", json);
}
}
DownloadUrl getDownloadUrl(const Input & input) const override
+11
View File
@@ -30,6 +30,7 @@ libfetchers = library(
liblixutil,
nlohmann_json,
],
cpp_pch : cpp_pch,
install : true,
# FIXME(Qyriad): is this right?
install_rpath : libdir,
@@ -55,3 +56,13 @@ liblixfetchers = declare_dependency(
include_directories : include_directories('.'),
link_with : libfetchers,
)
# FIXME: remove when https://git.lix.systems/lix-project/lix/issues/359 is fixed.
if is_static
liblixfetchers_mstatic = declare_dependency(
include_directories : include_directories('.'),
link_whole : libfetchers,
)
else
liblixfetchers_mstatic = liblixfetchers
endif
+1 -1
View File
@@ -33,7 +33,7 @@ MixCommonArgs::MixCommonArgs(const std::string & programName)
addFlag({
.longName = "option",
.description = "Set the Nix configuration setting *name* to *value* (overriding `nix.conf`).",
.description = "Set the Lix configuration setting *name* to *value* (overriding `nix.conf`).",
.category = miscCategory,
.labels = {"name", "value"},
.handler = {[this](std::string name, std::string value) {
+1
View File
@@ -20,6 +20,7 @@ libmain = library(
liblixutil,
liblixstore,
],
cpp_pch : cpp_pch,
install : true,
# FIXME(Qyriad): is this right?
install_rpath : libdir,
+1
View File
@@ -7,6 +7,7 @@
#include <atomic>
#include <map>
#include <thread>
#include <sstream>
#include <iostream>
#include <chrono>
+1 -1
View File
@@ -124,7 +124,7 @@ void detectStackOverflow();
*
* This is called by the handler installed by detectStackOverflow().
*
* This gives Nix library consumers a limit opportunity to report the error
* This gives Lix library consumers a limit opportunity to report the error
* condition. The handler should exit the process.
* See defaultStackOverflowHandler() for a reference implementation.
*
+1
View File
@@ -16,6 +16,7 @@
#include <future>
#include <regex>
#include <fstream>
#include <sstream>
#include <nlohmann/json.hpp>
+10 -3
View File
@@ -64,10 +64,13 @@ extern "C" int sandbox_init_with_parameters(const char *profile, uint64_t flags,
namespace nix {
namespace {
/**
* The system for which Nix is compiled.
*/
constexpr std::string_view nativeSystem = SYSTEM;
[[gnu::unused]]
constexpr const std::string_view nativeSystem = SYSTEM;
}
void handleDiffHook(
uid_t uid, uid_t gid,
@@ -1844,8 +1847,12 @@ void LocalDerivationGoal::runChild()
copyFile(path, chrootRootDir + path, { .followSymlinks = true });
}
if (settings.caFile != "")
pathsInChroot.try_emplace("/etc/ssl/certs/ca-certificates.crt", settings.caFile, true);
if (settings.caFile != "" && pathExists(settings.caFile)) {
// For the same reasons as above, copy the CA certificates file too.
// It should be even less likely to change during the build than resolv.conf.
createDirs(chrootRootDir + "/etc/ssl/certs");
copyFile(settings.caFile, chrootRootDir + "/etc/ssl/certs/ca-certificates.crt", { .followSymlinks = true });
}
}
for (auto & i : ss) pathsInChroot.emplace(i, i);
+3 -1
View File
@@ -14,6 +14,8 @@
#include "derivations.hh"
#include "args.hh"
#include <sstream>
namespace nix::daemon {
Sink & operator << (Sink & sink, const Logger::Fields & fields)
@@ -241,7 +243,7 @@ struct ClientSettings
} else if (name == settings.pluginFiles.name) {
if (tokenizeString<Paths>(value) != settings.pluginFiles.get())
warn("Ignoring the client-specified plugin-files.\n"
"The client specifying plugins to the daemon never made sense, and was removed in Nix >=2.14.");
"The client specifying plugins to the daemon never made sense, and was removed in Nix.");
}
else if (trusted
|| name == settings.buildTimeout.name
+1 -1
View File
@@ -208,7 +208,7 @@ struct DerivationType {
/**
* Impure derivation type
*
* This is similar at buil-time to the content addressed, not standboxed, not fixed
* This is similar at build-time to the content addressed, not sandboxed, not fixed
* type, but has some restrictions on its usage.
*/
struct Impure {
+1 -2
View File
@@ -196,7 +196,6 @@ struct curlFileTransfer : public FileTransfer
result.immutableUrl = match.str(1);
else
debug("got invalid link header '%s'", value);
warn("foo %s", value);
}
}
}
@@ -657,7 +656,7 @@ struct curlFileTransfer : public FileTransfer
/* Ugly hack to support s3:// URIs. */
if (request.uri.starts_with("s3://")) {
// FIXME: do this on a worker thread
return std::async(std::launch::deferred, [uri{request.uri}] {
return std::async(std::launch::deferred, [uri{request.uri}]() -> FileTransferResult {
#if ENABLE_S3
auto [bucketName, key, params] = parseS3Uri(uri);
+2 -2
View File
@@ -40,12 +40,12 @@ struct FileTransferSettings : Config
this, 300, "stalled-download-timeout",
R"(
The timeout (in seconds) for receiving data from servers
during download. Nix cancels idle downloads after this
during download. Lix cancels idle downloads after this
timeout's duration.
)"};
Setting<unsigned int> tries{this, 5, "download-attempts",
"How often Nix will attempt to download a file before giving up."};
"How often Lix will attempt to download a file before giving up."};
};
extern FileTransferSettings fileTransferSettings;
+2 -2
View File
@@ -87,7 +87,7 @@ Settings::Settings()
/* Set the build hook location
For builds we perform a self-invocation, so Nix has to be self-aware.
For builds we perform a self-invocation, so Lix has to be self-aware.
That is, it has to know where it is installed. We don't think it's sentient.
Normally, nix is installed according to `nixBinDir`, which is set at compile time,
@@ -101,7 +101,7 @@ Settings::Settings()
current executable, using `getSelfExe()`.
As a last resort, we resort to `PATH`. Hopefully we find a `nix` there that's compatible.
If you're porting Nix to a new platform, that might be good enough for a while, but
If you're porting Lix to a new platform, that might be good enough for a while, but
you'll want to improve `getSelfExe()` to work on your platform.
*/
std::string nixExePath = nixBinDir + "/nix";
+55 -48
View File
@@ -131,7 +131,7 @@ public:
Setting<bool> tryFallback{
this, false, "fallback",
R"(
If set to `true`, Nix will fall back to building from source if a
If set to `true`, Lix will fall back to building from source if a
binary substitute fails. This is equivalent to the `--fallback`
flag. The default is `false`.
)",
@@ -149,9 +149,9 @@ public:
MaxBuildJobsSetting maxBuildJobs{
this, 1, "max-jobs",
R"(
This option defines the maximum number of jobs that Nix will try to
This option defines the maximum number of jobs that Lix will try to
build in parallel. The default is `1`. The special value `auto`
causes Nix to use the number of CPUs in your system. `0` is useful
causes Lix to use the number of CPUs in your system. `0` is useful
when using remote builders to prevent any local builds (except for
`preferLocalBuild` derivation attribute which executes locally
regardless). It can be overridden using the `--max-jobs` (`-j`)
@@ -195,11 +195,11 @@ public:
Setting<std::string> thisSystem{
this, SYSTEM, "system",
R"(
The system type of the current Nix installation.
Nix will only build a given [derivation](@docroot@/language/derivations.md) locally when its `system` attribute equals any of the values specified here or in [`extra-platforms`](#conf-extra-platforms).
The system type of the current Lix installation.
Lix will only build a given [derivation](@docroot@/language/derivations.md) locally when its `system` attribute equals any of the values specified here or in [`extra-platforms`](#conf-extra-platforms).
The default value is set when Nix itself is compiled for the system it will run on.
The following system types are widely used, as [Nix is actively supported on these platforms](@docroot@/contributing/hacking.md#platforms):
The default value is set when Lix itself is compiled for the system it will run on.
The following system types are widely used, as [Lix is actively supported on these platforms](@docroot@/contributing/hacking.md#platforms):
- `x86_64-linux`
- `x86_64-darwin`
@@ -210,7 +210,7 @@ public:
- `armv7l-linux`
In general, you do not have to modify this setting.
While you can force Nix to run a Darwin-specific `builder` executable on a Linux machine, the result would obviously be wrong.
While you can force Lix to run a Darwin-specific `builder` executable on a Linux machine, the result would obviously be wrong.
This value is available in the Nix language as
[`builtins.currentSystem`](@docroot@/language/builtin-constants.md#builtins-currentSystem)
@@ -252,8 +252,8 @@ public:
R"(
The path to the helper program that executes remote builds.
Nix communicates with the build hook over `stdio` using a custom protocol to request builds that cannot be performed directly by the Nix daemon.
The default value is the internal Nix binary that implements remote building.
Lix communicates with the build hook over `stdio` using a custom protocol to request builds that cannot be performed directly by the Nix daemon.
The default value is the internal Lix binary that implements remote building.
> **Important**
>
@@ -270,7 +270,7 @@ public:
Setting<bool> alwaysAllowSubstitutes{
this, false, "always-allow-substitutes",
R"(
If set to `true`, Nix will ignore the `allowSubstitutes` attribute in
If set to `true`, Lix will ignore the `allowSubstitutes` attribute in
derivations and always attempt to use available substituters.
For more information on `allowSubstitutes`, see [the manual chapter on advanced attributes](../language/advanced-attributes.md).
)"};
@@ -278,7 +278,7 @@ public:
Setting<bool> buildersUseSubstitutes{
this, false, "builders-use-substitutes",
R"(
If set to `true`, Nix will instruct remote build machines to use
If set to `true`, Lix will instruct remote build machines to use
their own binary substitutes if available. In practical terms, this
means that remote hosts will fetch as many build dependencies as
possible from their own substitutes (e.g, from `cache.nixos.org`),
@@ -317,7 +317,7 @@ public:
Setting<bool> useSubstitutes{
this, true, "substitute",
R"(
If set to `true` (default), Nix will use binary substitutes if
If set to `true` (default), Lix will use binary substitutes if
available. This option can be disabled to force building from
source.
)",
@@ -326,23 +326,23 @@ public:
Setting<std::string> buildUsersGroup{
this, "", "build-users-group",
R"(
This options specifies the Unix group containing the Nix build user
accounts. In multi-user Nix installations, builds should not be
performed by the Nix account since that would allow users to
This options specifies the Unix group containing the Lix build user
accounts. In multi-user Lix installations, builds should not be
performed by the Lix account since that would allow users to
arbitrarily modify the Nix store and database by supplying specially
crafted builders; and they cannot be performed by the calling user
since that would allow him/her to influence the build result.
since that would allow them to influence the build result.
Therefore, if this option is non-empty and specifies a valid group,
builds will be performed under the user accounts that are a member
of the group specified here (as listed in `/etc/group`). Those user
accounts should not be used for any other purpose\!
Nix will never run two builds under the same user account at the
Lix will never run two builds under the same user account at the
same time. This is to prevent an obvious security hole: a malicious
user writing a Nix expression that modifies the build result of a
legitimate Nix expression being built by another user. Therefore it
is good to have as many Nix build user accounts as you can spare.
is good to have as many Lix build user accounts as you can spare.
(Remember: uids are cheap.)
The build users should have permission to create files in the Nix
@@ -351,11 +351,18 @@ public:
and its mode should be `1775`.
If the build users group is empty, builds will be performed under
the uid of the Nix process (that is, the uid of the caller if
`NIX_REMOTE` is empty, the uid under which the Nix daemon runs if
`NIX_REMOTE` is `daemon`). Obviously, this should not be used
the uid of the Lix process (that is, the uid of the caller if
both `NIX_REMOTE` is either empty or `auto` and the Nix store is
owned by that user, or, alternatively, the uid under which the Nix
daemon runs if `NIX_REMOTE` is `daemon` or if it is `auto` and the
store is not owned by the caller). Obviously, this should not be used
with a nix daemon accessible to untrusted clients.
For the avoidance of doubt, explicitly setting this to *empty* with a
Lix daemon running as root means that builds will be executed as root
with respect to the rest of the system.
We intend to fix this: https://git.lix.systems/lix-project/lix/issues/242
Defaults to `nixbld` when running as root, *empty* otherwise.
)",
{}, false};
@@ -405,7 +412,7 @@ public:
Setting<bool> keepLog{
this, true, "keep-build-log",
R"(
If set to `true` (the default), Nix will write the build log of a
If set to `true` (the default), Lix will write the build log of a
derivation (i.e. the standard output and error of its builder) to
the directory `/nix/var/log/nix/drvs`. The build log can be
retrieved using the command `nix-store -l path`.
@@ -468,7 +475,7 @@ public:
Setting<bool> autoOptimiseStore{
this, false, "auto-optimise-store",
R"(
If set to `true`, Nix automatically detects files in the store
If set to `true`, Lix automatically detects files in the store
that have identical contents, and replaces them with hard links to
a single copy. This saves disk space. If set to `false` (the
default), you can still run `nix-store --optimise` to get rid of
@@ -519,7 +526,7 @@ public:
network).
Currently, sandboxing only work on Linux and macOS. The use of a
sandbox requires that Nix is run as root (so you should use the
sandbox requires that Lix is run as root (so you should use the
build users feature to perform the actual builds under different
users than root).
@@ -545,7 +552,7 @@ public:
If the source is in the Nix store, then its closure will be added to
the sandbox as well.
Depending on how Nix was built, the default value for this option
Depending on how Lix was built, the default value for this option
may be empty or provide `/bin/sh` as a bind-mount of `bash`.
)",
{"build-chroot-dirs", "build-sandbox-paths"}};
@@ -556,7 +563,7 @@ public:
Setting<bool> requireDropSupplementaryGroups{this, getuid() == 0, "require-drop-supplementary-groups",
R"(
Following the principle of least privilege,
Nix will attempt to drop supplementary groups when building with sandboxing.
Lix will attempt to drop supplementary groups when building with sandboxing.
However this can fail under some circumstances.
For example, if the user lacks the `CAP_SETGID` capability.
@@ -642,7 +649,7 @@ public:
A whitespace-separated list of public keys.
At least one of the following condition must be met
for Nix to accept copying a store object from another
for Lix to accept copying a store object from another
Nix store (such as a substituter):
- the store object has been signed using a key in the trusted keys list
@@ -665,14 +672,14 @@ public:
this, 60 * 60, "tarball-ttl",
R"(
The number of seconds a downloaded tarball is considered fresh. If
the cached tarball is stale, Nix will check whether it is still up
to date using the ETag header. Nix will download a new version if
the cached tarball is stale, Lix will check whether it is still up
to date using the ETag header. Lix will download a new version if
the ETag header is unsupported, or the cached ETag doesn't match.
Setting the TTL to `0` forces Nix to always check if the tarball is
Setting the TTL to `0` forces Lix to always check if the tarball is
up to date.
Nix caches tarballs in `$XDG_CACHE_HOME/nix/tarballs`.
Lix caches tarballs in `$XDG_CACHE_HOME/nix/tarballs`.
Files fetched via `NIX_PATH`, `fetchGit`, `fetchMercurial`,
`fetchTarball`, and `fetchurl` respect this TTL.
@@ -701,7 +708,7 @@ public:
R"(
System types of executables that can be run on this machine.
Nix will only build a given [derivation](@docroot@/language/derivations.md) locally when its `system` attribute equals any of the values specified here or in the [`system` option](#conf-system).
Lix will only build a given [derivation](@docroot@/language/derivations.md) locally when its `system` attribute equals any of the values specified here or in the [`system` option](#conf-system).
Setting this can be useful to build derivations locally on compatible machines:
- `i686-linux` executables can be run on `x86_64-linux` machines (set by default)
@@ -748,16 +755,16 @@ public:
"substituters",
R"(
A list of [URLs of Nix stores](@docroot@/command-ref/new-cli/nix3-help-stores.md#store-url-format) to be used as substituters, separated by whitespace.
A substituter is an additional [store](@docroot@/glossary.md#gloss-store) from which Nix can obtain [store objects](@docroot@/glossary.md#gloss-store-object) instead of building them.
A substituter is an additional [store](@docroot@/glossary.md#gloss-store) from which Lix can obtain [store objects](@docroot@/glossary.md#gloss-store-object) instead of building them.
Substituters are tried based on their priority value, which each substituter can set independently.
Lower value means higher priority.
The default is `https://cache.nixos.org`, which has a priority of 40.
At least one of the following conditions must be met for Nix to use a substituter:
At least one of the following conditions must be met for Lix to use a substituter:
- The substituter is in the [`trusted-substituters`](#conf-trusted-substituters) list
- The user calling Nix is in the [`trusted-users`](#conf-trusted-users) list
- The user calling Lix is in the [`trusted-users`](#conf-trusted-users) list
In addition, each store path should be trusted as described in [`trusted-public-keys`](#conf-trusted-public-keys)
)",
@@ -866,13 +873,13 @@ public:
this, 0, "download-speed",
R"(
Specify the maximum transfer rate in kilobytes per second you want
Nix to use for downloads.
Lix to use for downloads.
)"};
Setting<std::string> netrcFile{
this, fmt("%s/%s", nixConfDir, "netrc"), "netrc-file",
R"(
If set to an absolute path to a `netrc` file, Nix will use the HTTP
If set to an absolute path to a `netrc` file, Lix will use the HTTP
authentication credentials in this file when trying to download from
a remote host through HTTP or HTTPS. Defaults to
`$NIX_CONF_DIR/netrc`.
@@ -898,7 +905,7 @@ public:
this, getDefaultSSLCertFile(), "ssl-cert-file",
R"(
The path of a file containing CA certificates used to
authenticate `https://` downloads. Nix by default will use
authenticate `https://` downloads. Lix by default will use
the first of the following files that exists:
1. `/etc/ssl/certs/ca-certificates.crt`
@@ -915,7 +922,7 @@ public:
Setting<StringSet> ignoredAcls{
this, {"security.selinux", "system.nfs4_acl", "security.csm"}, "ignored-acls",
R"(
A list of ACLs that should be ignored, normally Nix attempts to
A list of ACLs that should be ignored, normally Lix attempts to
remove all ACLs from files and directories in the Nix store, but
some ACLs like `security.selinux` or `system.nfs4_acl` can't be
removed even by root. Therefore it's best to just ignore them.
@@ -926,7 +933,7 @@ public:
this, {}, "hashed-mirrors",
R"(
A list of web servers used by `builtins.fetchurl` to obtain files by
hash. Given a hash type *ht* and a base-16 hash *h*, Nix will try to
hash. Given a hash type *ht* and a base-16 hash *h*, Lix will try to
download the file from *hashed-mirror*/*ht*/*h*. This allows files to
be downloaded even if they have disappeared from their original URI.
For example, given an example mirror `http://tarballs.nixos.org/`,
@@ -939,7 +946,7 @@ public:
}
```
Nix will attempt to download this file from
Lix will attempt to download this file from
`http://tarballs.nixos.org/sha256/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae`
first. If it is not available there, if will try the original URI.
)"};
@@ -948,7 +955,7 @@ public:
this, 0, "min-free",
R"(
When free disk space in `/nix/store` drops below `min-free` during a
build, Nix performs a garbage-collection until `max-free` bytes are
build, Lix performs a garbage-collection until `max-free` bytes are
available or there is no more garbage. A value of `0` (the default)
disables this feature.
)"};
@@ -983,7 +990,7 @@ public:
itself, they must be DSOs compatible with the instance of Nix
running at the time (i.e. compiled against the same headers, not
linked to any incompatible libraries). They should not be linked to
any Nix libs directly, as those will be available already at load
any Lix libs directly, as those will be available already at load
time.
If an entry in the list is a directory, all files in the directory
@@ -996,7 +1003,7 @@ public:
Setting<bool> allowSymlinkedStore{
this, false, "allow-symlinked-store",
R"(
If set to `true`, Nix will stop complaining if the store directory
If set to `true`, Lix will stop complaining if the store directory
(typically /nix/store) contains symlink components.
This risks making some builds "impure" because builders sometimes
@@ -1009,13 +1016,13 @@ public:
Setting<bool> useXDGBaseDirectories{
this, false, "use-xdg-base-directories",
R"(
If set to `true`, Nix will conform to the [XDG Base Directory Specification] for files in `$HOME`.
If set to `true`, Lix will conform to the [XDG Base Directory Specification] for files in `$HOME`.
The environment variables used to implement this are documented in the [Environment Variables section](@docroot@/command-ref/env-common.md).
[XDG Base Directory Specification]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
> **Warning**
> This changes the location of some well-known symlinks that Nix creates, which might break tools that rely on the old, non-XDG-conformant locations.
> This changes the location of some well-known symlinks that Lix creates, which might break tools that rely on the old, non-XDG-conformant locations.
In particular, the following locations change:
@@ -1025,7 +1032,7 @@ public:
| `~/.nix-defexpr` | `$XDG_STATE_HOME/nix/defexpr` |
| `~/.nix-channels` | `$XDG_STATE_HOME/nix/channels` |
If you already have Nix installed and are using [profiles](@docroot@/package-management/profiles.md) or [channels](@docroot@/command-ref/nix-channel.md), you should migrate manually when you enable this option.
If you already have Lix installed and are using [profiles](@docroot@/package-management/profiles.md) or [channels](@docroot@/command-ref/nix-channel.md), you should migrate manually when you enable this option.
If `$XDG_STATE_HOME` is not set, use `$HOME/.local/state/nix` instead of `$XDG_STATE_HOME/nix`.
This can be achieved with the following shell commands:
+1 -1
View File
@@ -5,7 +5,7 @@
* a glibc git checkout via `rg --pcre2 'define __NR_fchmodat2 (?!452)'`.
*
* The problem is that glibc 2.39 and libseccomp 2.5.5 are needed to
* get the syscall number. However, a Nix built against nixpkgs 23.11
* get the syscall number. However, a Lix built against nixpkgs 23.11
* (glibc 2.38) should still have the issue fixed without depending
* on the build environment.
*
+2 -2
View File
@@ -18,12 +18,12 @@ struct LocalFSStoreConfig : virtual StoreConfig
const PathSetting stateDir{this,
rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir,
"state",
"Directory where Nix will store state."};
"Directory where Lix will store state."};
const PathSetting logDir{this,
rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir,
"log",
"directory where Nix will store log files."};
"directory where Lix will store log files."};
const PathSetting realStoreDir{this,
rootDir.get() ? *rootDir.get() + "/nix/store" : storeDir, "real",
+8 -3
View File
@@ -313,13 +313,13 @@ LocalStore::LocalStore(const Params & params)
throw Error(
"Your Nix store has a database in Berkeley DB format,\n"
"which is no longer supported. To convert to the new format,\n"
"please upgrade Nix to version 0.12 first.");
"please use the original Nix version 0.12 first.");
if (curSchema < 6)
throw Error(
"Your Nix store has a database in flat file format,\n"
"which is no longer supported. To convert to the new format,\n"
"please upgrade Nix to version 1.11 first.");
"please use the original Nix version 1.11 first.");
if (!lockFile(globalLock.get(), ltWrite, false)) {
printInfo("waiting for exclusive access to the Nix store...");
@@ -550,7 +550,12 @@ void LocalStore::openDB(State & state, bool create)
if (mode == "wal" ) {
/* persist the WAL files when the DB connection is closed.
* This allows for read-only connections without any write permissions
* on the state directory to succeed on a closed database. */
* on the state directory to succeed on a closed database. Setting the
* journal_size_limit to 2^40 bytes results in the WAL files getting
* truncated to 0 on exit and limits the on disk size of the WAL files
* to 2^40 bytes following a checkpoint */
if (sqlite3_exec(db, "pragma main.journal_size_limit = 1099511627776;", 0, 0, 0) != SQLITE_OK)
SQLiteError::throw_(db, "setting journal_size_limit");
int enable = 1;
if (sqlite3_file_control(db, NULL, SQLITE_FCNTL_PERSIST_WAL, &enable) != SQLITE_OK)
SQLiteError::throw_(db, "setting persistent WAL mode");
+4 -1
View File
@@ -23,6 +23,9 @@ namespace nix {
* 0.7. Version 2 was Nix 0.8 and 0.9. Version 3 is Nix 0.10.
* Version 4 is Nix 0.11. Version 5 is Nix 0.12-0.16. Version 6 is
* Nix 1.0. Version 7 is Nix 1.3. Version 10 is 2.0.
*
* Lix started at 2.90, it cannot ever go past version 10 (Nix 2.18),
* since doing so will break compatibility with future CppNix versions.
*/
const int nixSchemaVersion = 10;
@@ -49,7 +52,7 @@ struct LocalStoreConfig : virtual LocalFSStoreConfig
R"(
Allow this store to be opened when its [database](@docroot@/glossary.md#gloss-nix-database) is on a read-only filesystem.
Normally Nix will attempt to open the store database in read-write mode, even for querying (when write access is not needed), causing it to fail if the database is on a read-only filesystem.
Normally Lix will attempt to open the store database in read-write mode, even for querying (when write access is not needed), causing it to fail if the database is on a read-only filesystem.
Enable read-only mode to disable locking and open the SQLite database with the [`immutable` parameter](https://www.sqlite.org/c3ref/open.html) set.
+1 -1
View File
@@ -140,7 +140,7 @@ static Machine parseBuilderLine(const std::string & line)
};
auto parseFloatField = [&](size_t fieldIndex) {
const auto result = string2Int<float>(tokens[fieldIndex]);
const auto result = string2Float<float>(tokens[fieldIndex]);
if (!result) {
throw FormatError("bad machine specification: failed to convert column #%lu in a row: '%s' to 'float'", fieldIndex, line);
}

Some files were not shown because too many files have changed in this diff Show More