From ab994bfcde0317faae78c88002f602075f90c398 Mon Sep 17 00:00:00 2001 From: tobtoht Date: Wed, 9 Apr 2025 21:13:40 +0200 Subject: [PATCH] macos: update gen-sdk --- contrib/macdeploy/README.md | 85 ++++++++++++------------------------- contrib/macdeploy/gen-sdk | 26 +----------- 2 files changed, 28 insertions(+), 83 deletions(-) diff --git a/contrib/macdeploy/README.md b/contrib/macdeploy/README.md index ff491b70..a6efa91a 100644 --- a/contrib/macdeploy/README.md +++ b/contrib/macdeploy/README.md @@ -6,95 +6,64 @@ A free Apple Developer Account is required to proceed. -Our current macOS SDK -(`Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz`) -can be extracted from -[Xcode_12.2.xip](https://download.developer.apple.com/Developer_Tools/Xcode_12.2/Xcode_12.2.xip). +Our macOS SDK can be extracted from +[Xcode_15.xip](https://download.developer.apple.com/Developer_Tools/Xcode_15/Xcode_15.xip). Alternatively, after logging in to your account go to 'Downloads', then 'More' -and search for [`Xcode 12.2`](https://developer.apple.com/download/all/?q=Xcode%2012.2). +and search for [`Xcode 15`](https://developer.apple.com/download/all/?q=Xcode%2015). An Apple ID and cookies enabled for the hostname are needed to download this. -The `sha256sum` of the downloaded XIP archive should be `28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0`. +The `sha256sum` of the downloaded XIP archive should be `4daaed2ef2253c9661779fa40bfff50655dc7ec45801aba5a39653e7bcdde48e`. -After Xcode version 7.x, Apple started shipping the `Xcode.app` in a `.xip` -archive. This makes the SDK less-trivial to extract on non-macOS machines. One -approach (tested on Debian Buster) is outlined below: +To extract the `.xip` on Linux: ```bash # Install/clone tools needed for extracting Xcode.app apt install cpio git clone https://github.com/bitcoin-core/apple-sdk-tools.git -# Unpack Xcode_12.2.xip and place the resulting Xcode.app in your current +# Unpack the .xip and place the resulting Xcode.app in your current # working directory -python3 apple-sdk-tools/extract_xcode.py -f Xcode_12.2.xip | cpio -d -i +python3 apple-sdk-tools/extract_xcode.py -f Xcode_15.xip | cpio -d -i ``` -On macOS the process is more straightforward: +On macOS: ```bash -xip -x Xcode_12.2.xip +xip -x Xcode_15.xip ``` -### Step 2: Generating `Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app` +### Step 2: Generating the SDK tarball from `Xcode.app` -To generate `Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz`, run -the script [`gen-sdk`](./gen-sdk) with the path to `Xcode.app` (extracted in the -previous stage) as the first argument. +To generate the SDK, run the script [`gen-sdk`](./gen-sdk) with the +path to `Xcode.app` (extracted in the previous stage) as the first argument. ```bash -# Generate a Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz from -# the supplied Xcode.app ./contrib/macdeploy/gen-sdk '/path/to/Xcode.app' ``` -The `sha256sum` of the generated TAR.GZ archive should be `332477876917786b26dd7c3fc1665d2c5cdca81c72755e6a9754f308de77d33b`. +The generated archive should be: `Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers.tar.gz`. +The `sha256sum` should be `c0c2e7bb92c1fee0c4e9f3a485e4530786732d6c6dd9e9f418c282aa6892f55d`. ## Deterministic macOS App Notes -macOS Applications are created in Linux by combining a recent `clang` and the Apple -`binutils` (`ld`, `ar`, etc). +macOS Applications are created on Linux using a recent LLVM. -Apple uses `clang` extensively for development and has upstreamed the necessary -functionality so that a vanilla clang can take advantage. It supports the use of `-F`, -`-target`, `-mmacosx-version-min`, and `-isysroot`, which are all necessary when -building for macOS. +All builds must target an Apple SDK. These SDKs are free to download, but not redistributable. +See the SDK Extraction notes above for how to obtain it. -Apple's version of `binutils` (called `cctools`) contains lots of functionality missing in the -FSF's `binutils`. In addition to extra linker options for frameworks and sysroots, several -other tools are needed as well such as `install_name_tool`, `lipo`, and `nmedit`. These -do not build under Linux, so they have been patched to do so. The work here was used as -a starting point: [mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4). +The Guix build process has been designed to avoid including the SDK's files in Guix's outputs. +All interim tarballs are fully deterministic and may be freely redistributed. -In order to build a working toolchain, the following source packages are needed from -Apple: `cctools`, `dyld`, and `ld64`. - -These tools inject timestamps by default, which produce non-deterministic binaries. The -`ZERO_AR_DATE` environment variable is used to disable that. - -This version of `cctools` has been patched to use the current version of `clang`'s headers -and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done in `toolchain4`. - -To complicate things further, all builds must target an Apple SDK. These SDKs are free to -download, but not redistributable. See the SDK Extraction notes above for how to obtain it. - -The Guix process builds 2 sets of files: Linux tools, then Apple binaries which are -created using these tools. The build process has been designed to avoid including the -SDK's files in Guix's outputs. All interim tarballs are fully deterministic and may be freely -redistributed. - -As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in -order to satisfy the new Gatekeeper requirements. Because this private key cannot be -shared, we'll have to be a bit creative in order for the build process to remain somewhat -deterministic. Here's how it works: +Using an Apple-blessed key to sign binaries is a requirement to produce (distributable) macOS +binaries. Because this private key cannot be shared, we'll have to be a bit creative in order +for the build process to remain somewhat deterministic. Here's how it works: - Builders use Guix to create an unsigned release. This outputs an unsigned ZIP which - users may choose to bless and run. It also outputs an unsigned app structure in the form - of a tarball, which also contains all of the tools that have been previously (deterministically) - built in order to create a final DMG. + users may choose to bless, self-codesign, and run. It also outputs an unsigned app structure + in the form of a tarball. - The Apple keyholder uses this unsigned app to create a detached signature, using the - script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). -- Builders feed the unsigned app + detached signature back into Guix. It uses the - pre-built tools to recombine the pieces into a deterministic ZIP. + included script. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs). +- Builders feed the unsigned app + detached signature back into Guix, which combines the + pieces into a deterministic ZIP. diff --git a/contrib/macdeploy/gen-sdk b/contrib/macdeploy/gen-sdk index e93a3eab..86a6262b 100755 --- a/contrib/macdeploy/gen-sdk +++ b/contrib/macdeploy/gen-sdk @@ -8,21 +8,6 @@ import gzip import os import contextlib -# monkey-patch Python 3.8 and older to fix wrong TAR header handling -# see https://github.com/bitcoin/bitcoin/pull/24534 -# and https://github.com/python/cpython/pull/18080 for more info -if sys.version_info < (3, 9): - _old_create_header = tarfile.TarInfo._create_header - def _create_header(info, format, encoding, errors): - buf = _old_create_header(info, format, encoding, errors) - # replace devmajor/devminor with binary zeroes - buf = buf[:329] + bytes(16) + buf[345:] - # recompute checksum - chksum = tarfile.calc_chksums(buf)[0] - buf = buf[:-364] + bytes("%06o\0" % chksum, "ascii") + buf[-357:] - return buf - tarfile.TarInfo._create_header = staticmethod(_create_header) - @contextlib.contextmanager def cd(path): """Context manager that restores PWD even if an exception was raised.""" @@ -60,13 +45,8 @@ def run(): sdk_build_id = pl['ProductBuildVersion'] print("Found MacOSX SDK (version: {sdk_version}, build id: {sdk_build_id})".format(sdk_version=sdk_version, sdk_build_id=sdk_build_id)) - clang_rt = xcode_app.joinpath("Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin") - out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id) - xcode_libcxx_dir = xcode_app.joinpath("Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1") - assert xcode_libcxx_dir.is_dir() - if args.out_sdktgz: out_sdktgz_path = pathlib.Path(args.out_sdktgz_path) else: @@ -74,7 +54,7 @@ def run(): out_sdktgz_path = pathlib.Path("./{}.tar.gz".format(out_name)) def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir): - """Add all files in dir_to_add to tarfp, but prepent MEMBERPREFIX to the files' + """Add all files in dir_to_add to tarfp, but prepent alt_base_dir to the files' names e.g. if the only file under /root/bazdir is /root/bazdir/qux, invoking: @@ -109,10 +89,6 @@ def run(): with tarfile.open(mode="w", fileobj=gzf, format=tarfile.GNU_FORMAT) as tarfp: print("Adding MacOSX SDK {} files...".format(sdk_version)) tarfp_add_with_base_change(tarfp, sdk_dir, out_name) - print("Adding libc++ headers...") - tarfp_add_with_base_change(tarfp, xcode_libcxx_dir, "{}/usr/include/c++/v1".format(out_name)) - print("Adding clang_rt libraries") - tarfp_add_with_base_change(tarfp, clang_rt, "{}/lib/darwin".format(out_name)) print("Done! Find the resulting gzipped tarball at:") print(out_sdktgz_path.resolve()) -- 2.52.0